Handling Future Timestamps in Trace Data for Arize

·Mar 07, 2024 03:29 PM

I get the below exception with trying to push trace data to the platform. Detected future timestamp in column 'end_time'. Caution when sending spans with future timestamps. Arize only stores 2 years worth of data. For example, if you sent spans to Arize from 1.5 years ago, and now send spans with timestamps of a year in the future, the oldest 0.5 years will be dropped to maintain the 2 years worth of data requirement.

13 comments

· Sorted by Oldest

Tammy L.
·
Kiko C. Aman K. (thread moved into this channel)
Kiko C.
·
Raja A. are you trying to send traces to Phoenix or Arize?
Raja A.
·
to Arize platfom
Raja A.
·
Kiko C. - Hi Kiko - Any inputs or help ? I am running the app in Ubuntu VM which is hosted in different timezone.
Aman K.
·
Hi Raja A. is this exception a warning or an error that blocks ingestion? And would you have a notebook/code snippet you could send to us to reproduce? This error is a warning so it shouldn’t block ingestion but just want to confirm
Kiko C.
·
I can confirm the above message is a warning not an error
Raja A.
·
Kiko C. and Aman K.. I am not able to see the traces. Not sure whether the timestamp is getting messed up somewhere. I am going to take a deeper look and will come back with my findings. Appreciate the response.
Kiko C.
·
Raja A. could you share steps to reproduce the error? Data sample and code?
Raja A.
·
Kiko C. - I can share the piece of code that is used for sending the data to platform

Raja A.

if __name__ == '__main__':
    session = px.launch_app()
    LangChainInstrumentor().instrument()
    main()
    df = px.Client().get_spans_dataframe()
    df.reset_index(drop=True, inplace=True)
    if df is not None:
        print(df.head())
    else:
        print("************** NULL ")
    SPACE_KEY = "<>"
    API_KEY = "<>"

    if SPACE_KEY == "SPACE_KEY" or API_KEY == "API_KEY":
        raise ValueError("❌ NEED TO CHANGE SPACE AND/OR API_KEY")
    else:
        print("✅ Import and Setup Arize Client Done! Now we can start using Arize!")
    
    arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)
    model_id = "generative-spans-tutorial-test" #the model name in Arize
    model_version = "1.0" #(optional) the model version
    
    response = arize_client.log_spans(
        dataframe=df,
        model_id=model_id,
        model_version=model_version, #optional
    )

Kiko C.
·
Hi Raja A., there are a few problems with that code snippet:
1.
I don't know what main() does. Hence I can't evaluate correctness on it
2.
session = px.launch_app() and other lines with px are phoenix logic, but you mentioned you want to send traces to Arize
3.
The model_id you seem to be trying is from one of our tutorials but this is not the logic in them
4.
You are using both LangChainInstrumentator and arize_client.log_spans when they are meant for 2 different things:
a.
You use the instrumentator if you want to log spans 1-by-1, streaming them via OTLP
b.
You use log_spans if you have the spans in a dataframe and want to upload all of them as 1 batch of data
Raja A.
·
Thanks for this debugging . If there is any code that I can refer to send traces to Arize. please let me know.
Kiko C.
·
This teaches how to send traces to Arize via OTLP https://colab.research.google.com/github/Arize-ai/tutorials_python/blob/main/Arize_Tutorials/Tracing/Arize_Tutorial_OTLP_Tracing_Llama_Index.ipynb

Handling Future Timestamps in Trace Data for Arize | Arize AI Community

Tammy L.
·
Kiko C. Aman K. (thread moved into this channel)
Kiko C.
·
Raja A. are you trying to send traces to Phoenix or Arize?
Raja A.
·
to Arize platfom
Raja A.
·
Kiko C. - Hi Kiko - Any inputs or help ? I am running the app in Ubuntu VM which is hosted in different timezone.
Aman K.
·
Hi Raja A. is this exception a warning or an error that blocks ingestion? And would you have a notebook/code snippet you could send to us to reproduce? This error is a warning so it shouldn’t block ingestion but just want to confirm
Kiko C.
·
I can confirm the above message is a warning not an error
Raja A.
·
Kiko C. and Aman K.. I am not able to see the traces. Not sure whether the timestamp is getting messed up somewhere. I am going to take a deeper look and will come back with my findings. Appreciate the response.
Kiko C.
·
Raja A. could you share steps to reproduce the error? Data sample and code?
Raja A.
·
Kiko C. - I can share the piece of code that is used for sending the data to platform

Raja A.

if __name__ == '__main__':
    session = px.launch_app()
    LangChainInstrumentor().instrument()
    main()
    df = px.Client().get_spans_dataframe()
    df.reset_index(drop=True, inplace=True)
    if df is not None:
        print(df.head())
    else:
        print("************** NULL ")
    SPACE_KEY = "<>"
    API_KEY = "<>"

    if SPACE_KEY == "SPACE_KEY" or API_KEY == "API_KEY":
        raise ValueError("❌ NEED TO CHANGE SPACE AND/OR API_KEY")
    else:
        print("✅ Import and Setup Arize Client Done! Now we can start using Arize!")
    
    arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)
    model_id = "generative-spans-tutorial-test" #the model name in Arize
    model_version = "1.0" #(optional) the model version
    
    response = arize_client.log_spans(
        dataframe=df,
        model_id=model_id,
        model_version=model_version, #optional
    )

Kiko C.
·
Hi Raja A., there are a few problems with that code snippet:
1.
I don't know what main() does. Hence I can't evaluate correctness on it
2.
session = px.launch_app() and other lines with px are phoenix logic, but you mentioned you want to send traces to Arize
3.
The model_id you seem to be trying is from one of our tutorials but this is not the logic in them
4.
You are using both LangChainInstrumentator and arize_client.log_spans when they are meant for 2 different things:
a.
You use the instrumentator if you want to log spans 1-by-1, streaming them via OTLP
b.
You use log_spans if you have the spans in a dataframe and want to upload all of them as 1 batch of data
Raja A.
·
Thanks for this debugging . If there is any code that I can refer to send traces to Arize. please let me know.
Kiko C.
·
This teaches how to send traces to Arize via OTLP https://colab.research.google.com/github/Arize-ai/tutorials_python/blob/main/Arize_Tutorials/Tracing/Arize_Tutorial_OTLP_Tracing_Llama_Index.ipynb