Running Phoenix CLI with Inference Sets for LLM Summarization

Neeraja S. · 2024-06-13T06:17:26.625Z

Hi All, I am working on implementing the LLM summarization example "https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llm_summarization_tutorial.ipynb". The example is executed perfectly fine in google colab notebook, so I wanted to test the same example by running the Phoenix from command terminal. But in this example the Phoenix is launched using Primary and Reference Inference Sets i.e, "session = px.launch_app(primary=recent_ds, reference=baseline_ds)". The command to run Phoenix from CLI is "python3 -m phoenix.server.main serve". So, when I run the Phoenix and run the LLM summarization python code in CLI, I am not able to get the analysis result on Phoenix UI, may be because the inference sets are not used as an argument in the CLI command. How can I run Phoenix in CLI while defining the inference sets? Thanks in advance for the help.

Mikyo
·
Good morning Neeraja S. - Phoenix's embedding analysis tools are designed to run in Jupyter Notebooks since they require configuration of your inference schemas before launching the app. Phoenix can collect your embeddings and persist them but this requires tracing your embedding calls. We will be working on dense embedding projection (https://github.com/Arize-ai/phoenix/issues/2018) of traces later this year. If you want to stand up phoenix from the command-line, you can certainly write a small python script that calls launch_app and keeps the process running via while(true) sleep loop. Hope that helps! Would love to learn your use-case so please file us a github issue if you have any feedback.
Neeraja S.
·
Hi Mikyo, thank you for clarifying my doubt. Can I integrate any database such as the clickhouse with Phoenix to store the traces or dataframes or embeddings once the analysis is done?
Mikyo
·
For traces yes, you can deploy and host Phoenix with persistence https://docs.arize.com/phoenix/deployment
Neeraja S.
·
Suppose if I am trying to build a different UI by using the API's and SDK's of Phoenix, how can I display the embeddings in personalised UI somewhat similar to Phoenix UI?
Neeraja S.
·
Hi parser word-break-and-wrap hover:underline slack_blocks_to_jsx__rich_text_section_element_link" href="/members/31b52ca1-6f92-486f-815c-92bc4bd5987b">Mikyo, Can I make changes to the Phoenix server main.py file to make use of options like "primary and reference"? ix/blob/main/src/phoenix/server/main.py#L28" class="slack_blocks_to_jsx__rich_text_section_element_link text-blue-primary hover:underline underline-offset-4">main.py = ArgumentParser() parser.add_argument("--database-url", required=False) parser.add_argument("--export_path") parser.add_argument("--host", type=str, required=False) parser.add_argument("--port", type=int, required=False) parser.add_argument("--read-only", type=bool, default=False) parser.add_argument("--no-internet", action="store_true") parser.add_argument("--umap_params", type=str, required=False, default=DEFAULT_UMAP_PARAMS_STR) parser.add_argument("--debug", action="store_false") subparsers = parser.add_subparsers(dest="command", required=True) serve_parser = subparsers.add_parser("serve") datasets_parser = subparsers.add_parser("datasets") datasets_parser.add_argument("--primary", type=str, required=True) datasets_parser.add_argument("--reference", type=str, required=False) datasets_parser.add_argument("--corpus", type=str, required=False) datasets_parser.add_argument("--trace", type=str, required=False) fixture_parser = subparsers.add_parser("fixture") fixture_parser.add_argument("fixture", type=str, choices=[fixture.name for fixture in FIXTURES]) fixture_parser.add_argument("--primary-only", type=bool) trace_fixture_parser = subparsers.add_parser("trace-fixture") trace_fixture_parser.add_argument( "fixture", type=str, choices=[fixture.name for fixture in TRACES_FIXTURES]
Mikyo
·
Neeraja S Jawali [10:31 PM] Suppose if I am trying to build a different UI by using the API's and SDK's of Phoenix, how can I display the embeddings in personalised UI somewhat similar to Phoenix UI?
I think it depends on the language you choose but the frontend is fully OSS so definitely take a peek!
Can I make changes to the Phoenix server main.py file to make use of options like "primary and reference"?
Totally understand your ask - the problem with inferences is that they require schemas and we don't really have parameterized support for that right now. But maybe you could try in a fork and let us know if it works out? We probably will mainly support embedding spans rather than dataframes going forward. I think the simple script to wrap main might be the best option right now.
Neeraja S.
·
Hi Mikyo, the python script to call launch_app is not working. The script would be executed without errors, but I am not able to access the Phoenix UI.
Neeraja S.
·
Hi Mikyo, Is there anyway to save the embeddings to either persistence storage or to the json file once you export a cluster?
Mikyo
·
Hey Neeraja S. sorry for the delay. We don’t support logging of embeddings via inferences right now and focusing on the traces. We have some embedding work coming up but not immediately. Maybe you could tell me a bit more about your use case and I might be able to offer a solution!
Mikyo
·
Yes, launch app actually exits the second it’s called in a script. If it’s in a script, you will have to sleep the main thread as the application will be running in a separate thread.
Mikyo
·
I can write you a small example snippet it in the morning
Neeraja S.
·
Mikyo, Thank you for your response. I am working on creating a UI to visualize the embeddings of LLM models, similar to Phoenix. To facilitate this, I am considering saving the embeddings to a database, which would make it easier to use the embeddings for visualization purposes
Neeraja S.
·
Hi Mikyo, I was trying to use PostgreSQL as persistence storage, but now I want to use the default SQLite persistence. What modifications needs to done so as to use SQLite?
Mikyo
·
Have you tried changing the config Neeraja? https://docs.arize.com/phoenix/deployment/persistence#sqlite
Neeraja S.
·
Yes, Mikyo. I exported the SQLite database URL and its working.

Running Phoenix CLI with Inference Sets for LLM Summarization

34 comments

Running Phoenix CLI with Inference Sets for LLM Summarization

34 comments