Clarification on Using suppress_tracing Context Manager in OpenInference

Trevor L. · 2024-06-04T01:15:50.312Z

Quick clarification question on the suppress_tracing (https://github.com/Arize-ai/openinference/blob/aa8f655c57ba0c7ad93936110d8f275e447309e5/python/openinference-instrumentation/src/openinference/instrumentation/__init__.py#L34) context manager. Could we use that to wrap any function we don't want spans to be captured for (e.g., even with spans that are generated outside of an openinference autoinstrumentor)? There are a few calls to get data from Snowflake tables (using snowpark) that happen in the application, and spans for those function calls are still showing up in the console and Phoenix even when wrapping the functions with suppress_tracing , so not sure if it's being used correctly.

Mikyo
·
Hey Trevor L. supress_tracing is a flag that gets set on the context manager for OTEL. It's actually a reserved flag you can set and part of the OTel SDKs. That being said it doesn't prevent you from being able to create spans manually. So it will not supress tracing not created by instrumentors. You will have to perform the check in your code if you want to suppress tracing in your code.
from opentelemetry.context import _SUPPRESS_INSTRUMENTATION_KEY context_api.get_value(_SUPPRESS_INSTRUMENTATION_KEY): return None
Trevor L.
·
Thanks for the response Mikyo! We aren't creating any spans manually which is why I was a bit confused still. Does the above code check if the suppress instrumentation key is set on the current context?
Mikyo
·
Yeah it does - My guess is you have some sort of HTTP instrumentor setup. Can you check what instrumentors are configured for OTel?
👍1
Mikyo
·
The unfortunate thing is that I think this flag is opt-in
Trevor L.
·
Yeah I'm reading through some of the Snowflake docs because I think it's coming from there. Not too familiar w/ Snowpark but trying to see how to suppress tracing directly there instead
Mikyo
·
Sg, let us know what you find. I'm sure you can turn it off
👍1
1
Trevor L.
·
When using the to_pandas() method, it looks like that's where the spans are being created. I'm still trying to figure out how to turn it off though
Mikyo
·
Looks like it’s natively instrumented https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/snowpark/api/snowflake.snowpark.Session.telemetry_enabled
Trevor L.
·
Yeah we tried setting that to False but the spans were still showing up in the console. It looked like that parameter just disables them from showing up in Snowflake
Trevor L.
·
We're also testing setting the TRACE_LEVEL to OFF to see if that resolves
Mikyo
·
Trevor L. you might try this https://opentelemetry.io/docs/languages/python/automatic/configuration/#:~:text=Disabling%20Specific%20Instrumentations,-The%20Python%20agent&text=You%20can%20omit%20specific%20packages,names%20to%20exclude%20from%20instrumentation.
Mikyo
·
I also filed an issue with snowpark https://github.com/snowflakedb/snowpark-python/issues/1730
1
Trevor L.
·
Thanks Mikyo! Just to confirm, there's 2 things we can try:
1.
We could try adding the snowpark package to the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS env variable to see if that disables it
2.
We could patch snowpark and add the ability to disable the creation of spans in the above file even if the otel package is found
Mikyo
·
1.
We could try adding the snowpark package to the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS env variable to see if that disables it
My guess is this will not work as they have "native" instrumentation but it's a good argument that they should support the flag I would say I would try 2 if this is really important. Maybe the team can upvote the above ticket
👍1
Trevor L.
·
as they have "native" instrumentation
This refers to the package creating spans with this tracer.start_as_current_span(name) as cur_span: , right?

16 comments