Log Rotation for Phoenix App in Docker: Challenges & Solutions

·Mar 21, 2024 06:00 PM

Hey Arize Team! I wanted to ask if there is a way to log rotate the trace datasets from my docker container that is running phoenix app (We have a separate application hosted elsewhere that sends trace data to this phoenix server in a docker container) I know there is a api to get trace data from the phoenix client but this leads to unpredictable and bursty demand for resources (CPU, I/O, Memory, etc). We instead wanted to rotate the underlying log files ourselves and have it be stored elsewhere I found the working directory within my docker container /root/.phoenix but it contains 3 empty directories datasets, exports, and trace_datasets with no files inside it. Though my phoenix server is showing 23 traces Few clarifying questions

1.
Is all the trace data actually stored in the above directory? Or is there some other one, or are these directories not used the way i thought they were?
2.
Is the above alternative to get_spans_dataframe a good one? any challenges you think (pending 1 is actually how I think about it)

Any help here would be much appreciated, without log rotation, we are having a hard time maintaining the server running phoenix in a production setting

4 comments

· Sorted by Oldest

Dustin N.
·
hi Swaraj R.! thanks so much for using Phoenix! Unfortunately right now the Phoenix server stores all of its data in-memory. We're actively working on persistence right now to enable use cases like yours. Hopefully we can find a solution for you long term very soon
Swaraj R.
·
Thanks for the quick response Dustin Couple more clarifications based on that
1.
Any tentative ETA you folks can share on the persisting trace data? Or is there a github issue I should watch?
2.
Any idea why disk read bytes spikes when calling get_spans_dataframe if we aren't reading from disk? (this may be swap?)
3.
What is the purpose of these folders today or setting PHOENIX_WORKING_DIR? Is this for some other use-case or set up for the persisting feature?
Dustin N.
·
1.
I don't have an ETA currently, but it's one of our highest priorities right now.
2.
That's a good question and I don't have a great answer for you off the top of my head. I'll have to do some digging.
3.
PHOENIX_WORKING_DIR is the directory for dumping data to disk and loading it from disk when starting phoenix, it's not quite persistence, but you can save parquet files with all your existing spans and start the server fresh to keep a copy of your old data.
Swaraj R.
·
Sounds good 👍 Thanks for the help Dustin!

Dustin N.
·
hi Swaraj R.! thanks so much for using Phoenix! Unfortunately right now the Phoenix server stores all of its data in-memory. We're actively working on persistence right now to enable use cases like yours. Hopefully we can find a solution for you long term very soon
Swaraj R.
·
Thanks for the quick response Dustin Couple more clarifications based on that
1.
Any tentative ETA you folks can share on the persisting trace data? Or is there a github issue I should watch?
2.
Any idea why disk read bytes spikes when calling get_spans_dataframe if we aren't reading from disk? (this may be swap?)
3.
What is the purpose of these folders today or setting PHOENIX_WORKING_DIR? Is this for some other use-case or set up for the persisting feature?
Dustin N.
·
1.
I don't have an ETA currently, but it's one of our highest priorities right now.
2.
That's a good question and I don't have a great answer for you off the top of my head. I'll have to do some digging.
3.
PHOENIX_WORKING_DIR is the directory for dumping data to disk and loading it from disk when starting phoenix, it's not quite persistence, but you can save parquet files with all your existing spans and start the server fresh to keep a copy of your old data.
Swaraj R.
·
Sounds good 👍 Thanks for the help Dustin!