Error in Exporting Traces: gRPC Channel Issue with HTML Size Limit
2024-07-27 10:27:03,498 - ERROR - Failed to export traces to otlp.arize.com, error code: StatusCode.UNKNOWN
Traceback (most recent call last):
File "/home/user/venv/lib/python3.12/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 262, in _export
self._client.Export(
File "/home/user/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "All spans failed. Found 1 errors: rpc error: code = InvalidArgument desc = user defined attribute value cannot contain more than 4000000 characters. Found 1612376 for key env.page_html"
debug_error_string = "UNKNOWN:Error received from peer ipv4:34.49.228.251:443 {created_time:"2024-07-27T10:27:03.497684563+00:00", grpc_status:2, grpc_message:"All spans failed. Found 1 errors: rpc error: code = InvalidArgument desc = user defined attribute value cannot contain more than 4000000 characters. Found 1612376 for key env.page_html"}"
>our prompts are usually about 6k tokens, but we do log a full web page too sometimes which is what broke the GRPC channel. We need the webspage to be able to reconstruct what might have gone wrong in hindsight and to backtest changes as our agents parses the webpage itself Is there a way to increase the limit? should we try to compress the html somehow? the raw html is already compressed -> base64, also this is not logged at every interaction only on specific exceptions
