Arize AI Community Icon

Customizing Model Names in Phoenix Traces for Llama2CPP Usage | Arize AI Community

Phoenix Support

Customizing Model Names in Phoenix Traces for Llama2CPP Usage

·Feb 06, 2024 08:53 PM

Hi – We’re using our own LLM (llama2cpp) within an OpenAI wrapper, and we can successfully get Phoenix to provide traces for it. However, we’ve run into a couple of issues: Question 1: Our LLM is always erroneously called “gpt-3.5-turbo” as the model name in the Phoenix trace. Is there a parameter we can override to change it to an arbitrary name? It seems like using “model_name = “ in OpenAIModel might actual force an OpenAI model to be used, but I could be wrong here. We define our LLM as follows: llm=OpenAI( api_base=http://xxx.yyy.zzz.aaa/, ) service_context= ServiceContext.from_defaults(llm=llm) Then we generate answers to the questions in our vector store, using the LLM: vector_index = VectorStoreIndex(train_nodes, service_context=service_context) query_engine = vector_index.as_query_engine(verbose=True) We have verified that the answers are generated using llama2cpp, but the Phoenix trace reads: { "id": "chatcmpl-c2a035ab-488e-4e21-a6ac-dbc51a5397d3", "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": " <block of content printed here>", "role": "assistant" } } ], "created": 1707240657, "model": "gpt-3.5-turbo", "object": "chat.completion", "usage": { "completion_tokens": 236, "prompt_tokens": 833, "total_tokens": 1069 } } How can we change the “model” line highlighted above to reflect a user-defined name for our local LLM? Question 2: As a second issue, as noted in the id field above, our local LLM is somehow defaulting to ChatCompletion, instead of the Completion trace that we get when running, say, GPT3.5. This might be something we need to address through LlamaIndex, but there’s still an issue on the Phoenix side: Whereas Completion creates a Query/Retrieval span where all retrieved documents can be recovered through Phoenix, ChatCompletion doesn’t allow access to any retrieved documents, as far as I can tell. In other words, get_retrieved_documents(px.active_session()) returns an empty dataframe (with only column headers showing) when using ChatCompletion. Is this expected behavior? Is there a different way to access the retrieved documents when using ChatCompletion? Thanks and regards, David

Mikyo
·
Hey David K., thanks for flagging these issues! Just double checking that you are using the LlamaIndex set_global_handler as the method for instrumenting your app in a notebook? I'll get to filing some github issues - it might be related to how LlamaIndex is sending info to us via their callback system. For 2, just double checking that you are using a llamaIndex retriever? I wouldn't think swapping out the model would impact the traces for retrieval but I can't be for sure.
Mikyo
·
Here are the tickets: https://github.com/Arize-ai/phoenix/issues/2217 https://github.com/Arize-ai/phoenix/issues/2218 David K. if there is any more details you can provide us (like maybe some code blocks of how you have it setup) - we'll try to repro on our end. Let us know if you feel blocked. Happy to schedule a quick chat to get the details of your app
David K.
·
Thanks, Mikyo. For Issue #2, I think the problem was probably on my end: CODE BLOCK A: phoenix_host="127.0.0.1" os.environ["PHOENIX_HOST"]=phoenix_host phoenix_collector_endpoint="http://127.0.0.1:6006" os.environ["PHOENIX_COLLECTOR_ENDPOINT"]=phoenix_collector_endpoint CODE BLOCK B: px.close_app() px.launch_app() CODE BLOCK C: import llama_index set_global_handler("arize_phoenix") llama_index.set_global_handler("arize_phoenix") CODE BLOCK D : (called inside my Q&A generation routine) from openinference.instrumentation.openai import OpenAIInstrumentor from opentelemetry import trace as trace_api from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk import trace as trace_sdk from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace.export import SimpleSpanProcessor resource = Resource(attributes={}) tracer_provider = trace_sdk.TracerProvider(resource=resource) span_exporter = OTLPSpanExporter(endpoint="http://127.0.0.1:6006/v1/traces") span_processor = SimpleSpanProcessor(span_exporter=span_exporter) tracer_provider.add_span_processor(span_processor=span_processor) trace_api.set_tracer_provider(tracer_provider=tracer_provider) OpenAIInstrumentor().instrument() A few observations:
1.
GPT-3.5-turbo always works properly (from a Phoenix trace standpoint), whether or not I include A and/or C. I get the "query chain" traces and subsequently the get_retrieved_documents function works fine.
2.
Local LLM does not return query chains if I omit A or C (but include B&D). I still do get LLM spans, but get_retrieved_documents returns an empty dataframe.
3.
Local LLM does return query chains if I include A, B, C, D. Hooray!
So I think this is probably expected behavior, although I'm excited for the day when you can simplify the tracing code needed for non-OpenAI LLMs! Thanks and regards, David
Roger Y.
·
Local LLM does not return query chains if I omit A or C (but include B&D).
This is because Llama-index is being instrumented by C. Without that the retrieval won’t be recorded. So any time C is included, the get_retrieved_documents should work because that’s provided by the Llama-Index callback system
1
Roger Y.
·
Question 1: Our LLM is always erroneously called “gpt-3.5-turbo” as the model name in the Phoenix trace.
Is this observed only when D is not included? that would mean it comes fromC, i.e. Llama-Index is not picking up the right names. If we include D but not C are we getting the right name in the span?
Mikyo
·
Thanks Roger. David, if you can remove some of the duplicate instrumentation. _________ As an aside I ran using llama_cpp. I do see that we fail to get the model name. However I do get full traces. I've parked my notebook here: https://github.com/Arize-ai/phoenix/pull/2221
David K.
·
Hi Roger --
Local LLM does not return query chains if I omit A or C (but include B&D).
"So any time C is included, the get_retrieved_documents should work because that’s provided by the Llama-Index callback system" I find this is only true if both A and C are included. If A is omitted, C will only result in my previous issue of missing query traces, and get_retrieved_documents returns an empty frame. That said, I think A is needed because I'm using a local LLM, but your statement may hold true for OpenAI LLMs. Thanks and regards, David
David K.
·
Hi Mikyo -- "David, if you can remove some of the duplicate instrumentation." I'm not sure which instrumentation is duplicated. for a local LLM, am I able to delete anything from A,C,D and still have tracing work properly? Perhaps you're referring to the following lines, one of which could likely be deleted? set_global_handler("arize_phoenix") llama_index.set_global_handler("arize_phoenix") Thanks and regards, David
Mikyo
·
Hey David K. - I quickly looked over your instrumentation and you are right, there's no duplicates. You may have caught us mid-migration so it might be that you are saying your OpenAI spans are detached from your LlamaIndex Spans? Might make the most sense to just stick with OpenInference https://github.com/Arize-ai/openinference/blob/main/python/instrumentation/openinference-instrumentation-llama-index/README.rst As that will be the direction everything will be going (including Phoenix). These sets of instrumentation should compose together. I may be mis-reading your message however. Would it be possible for you to share a screenshot of the symptoms?