I ended up writing custom code that periodically saves traces to disk and when starts the phoenix then it takes all the trace files(panda data frames) combines and imports them. Using the rest API, did not work since it was failing to serialize data frames, because of different data types for same column. To solve that I used pickle instead of pyarrow.