Troubleshooting Phoenix Session Trace Exports and Timeouts

·Aug 01, 2024 05:45 PM

Hello, I have a running pheonix session, which keeps collecting traces. However, it is running on a remote server, and the python process responsible for this session, occasionally tries to export the traces. It was working fine for some time. Recently (probably due to changes in versions), on every export/save trial an error is raised. This is a Timed Out exception, that is a result of the get_trace_dataset call. Couldn't find in the documentation any guidance to change the way in which trace datasets are saved. traces = px.Client().get_trace_dataset() traces.save(trace_data_path) Is there a better way to handle this step? Moreover, to avoid loss of valuable data (traces):

is it possible to connect to a running session on a remote server, in order to export the traces?
Is it possible to somehow export the traces via the web UI?

Many thanks in advance

13 comments

· Sorted by Oldest

Xander S.
·
Hey Dvir B., it sounds like you have some sort of cronjob running to occasionally persist to a file. This is a bit of an unexpected pattern. I'm curious what are you using these files for?
Dvir B.
·
Thanks for the response Xander S. The tracing instruments an application, implemented with LangChain, in which calls are made. We are using the traces to investigate the raw content of the calls (with all the context), and the responses.
Xander S.
·
Curious, what version of Phoenix are you using?
Dvir B.
·
Downgraded back to 4.14.1, due to other types of exceptions. What is the preferred and better pattern for exporting the traces?
Dvir B.
·
It's not a cron job, just a simple python script that launches the app, and every X seconds/mins asks for the traces from the app, and saves them.
👍1
Xander S.
·
I would recommend running Phoenix from the command line with python -m phoenix.server.main serve. If you launch in this way, your traces will be persisted to disk for you (by default, SQLite, but we also support Postgres), and you can stop and start Phoenix without losing your traces. No need to manually read and write your traces to disk. phoenix.launch_app is intended primarily for notebooks. It creates a temporary session, so your traces will be lost at the end of the session. If you do wish to write your traces to a Parquet file, e.g., to share with someone else, you still have that option with get_trace_dataset. In recent versions of Phoenix, this method has a timeout parameter you can adjust. Hope this helps!
👍1
Xander S.
·
By the way, phoenix.launch_app has a use_temp_dir parameter you can set to False if you want to use that method for launching Phoenix. But many users find it simpler to run from the command line.
Dvir B.
·
if i run the app using python -m phoenix.server.main serve from which entity do i ask get_trace_dataset?
Xander S.
·
You can run the client in a separate process.
Xander S.
·
Just to be clear, if all you're trying to do is persist your trace data, you shouldn't need get_trace_dataset. Just run Phoenix and send traces to it 🙂
Dvir B.
·
I'm trying to find this way of launching the app in the documentation
Xander S.
·
https://docs.arize.com/phoenix/setup/environments
👍1
Dvir B.
·
just found it now i will try that now. Thanks a lot for the assistance! should have joined much earlier
🙌1

Troubleshooting Phoenix Session Trace Exports and Timeouts | Arize AI Community

Xander S.
·
Hey Dvir B., it sounds like you have some sort of cronjob running to occasionally persist to a file. This is a bit of an unexpected pattern. I'm curious what are you using these files for?
Dvir B.
·
Thanks for the response Xander S. The tracing instruments an application, implemented with LangChain, in which calls are made. We are using the traces to investigate the raw content of the calls (with all the context), and the responses.
Xander S.
·
Curious, what version of Phoenix are you using?
Dvir B.
·
Downgraded back to 4.14.1, due to other types of exceptions. What is the preferred and better pattern for exporting the traces?
Dvir B.
·
It's not a cron job, just a simple python script that launches the app, and every X seconds/mins asks for the traces from the app, and saves them.
👍1
Xander S.
·
I would recommend running Phoenix from the command line with python -m phoenix.server.main serve. If you launch in this way, your traces will be persisted to disk for you (by default, SQLite, but we also support Postgres), and you can stop and start Phoenix without losing your traces. No need to manually read and write your traces to disk. phoenix.launch_app is intended primarily for notebooks. It creates a temporary session, so your traces will be lost at the end of the session. If you do wish to write your traces to a Parquet file, e.g., to share with someone else, you still have that option with get_trace_dataset. In recent versions of Phoenix, this method has a timeout parameter you can adjust. Hope this helps!
👍1
Xander S.
·
By the way, phoenix.launch_app has a use_temp_dir parameter you can set to False if you want to use that method for launching Phoenix. But many users find it simpler to run from the command line.
Dvir B.
·
if i run the app using python -m phoenix.server.main serve from which entity do i ask get_trace_dataset?
Xander S.
·
You can run the client in a separate process.
Xander S.
·
Just to be clear, if all you're trying to do is persist your trace data, you shouldn't need get_trace_dataset. Just run Phoenix and send traces to it 🙂
Dvir B.
·
I'm trying to find this way of launching the app in the documentation
Xander S.
·
https://docs.arize.com/phoenix/setup/environments
👍1
Dvir B.
·
just found it now i will try that now. Thanks a lot for the assistance! should have joined much earlier
🙌1