Improve Early Failure for Tracer on Unreachable Endpoints

·Jul 31, 2024 11:10 AM

Is there a way to make the tracer fail early if the endpoint is not reachable? Right now it waits for a span to be over to communicate, which means it fails after an unit of work rather than failing before. Something like:

tracer.span_processor.force_flush()

Would have been a good way to fail early out but it's not implemented.

✅1

28 comments

· Sorted by Oldest

Manuel D.

This is how we do it now:

requests.get(tracer.span_processor._span_processors[0].span_exporter._endpoint)

very brittle but achieves the early failure

Roger Y.
·
for my understanding, what exactly does “fail early” imply? does it halt the whole process besides spans?
Roger Y.
·
in other words, what’s the drawback of “failing late” (i.e. if the span is generated but can’t be exported)?
Roger Y.
·
technically you can also temporarily store the spans elsewhere and re-export them when the endpoint is back
Manuel D.
·
https://gist.github.com/manuel-delverme/06ff5b2ad5d25ccd6cf89e6b8a7f1295
Manuel D.
·
the second is the idea behavior so we detect the failure early rather than compute and then fail
Manuel D.
·
since the main thread doesn't fail, the current way is also OK usually so it's not high importance or urgent
Roger Y.
·
i see. well, if the operation is long enough, the endpoint can still be gone by the time the operation is finished, so it’s not guaranteed either
Roger Y.
·
you can attache a second and local exporter as a backup
Roger Y.
·
if the export is successful you can erase the backup
Roger Y.
·
if export fails you can try again later
Roger Y.
·
an example is the in memory exporter
Manuel D.
·
it's ok if networking is off or sluggish in the end, the issue is that we might have a machine that cannot connect to the exporter at all, the other alternative would be to export a fake trace but then we have fake traces everywhere
Roger Y.
·
machine that cannot connect to the exporter at all
i see. alternatively you can also write the spans to a file
Roger Y.
·
the protobuf format is serializable to strings