Request for Access to Tokens Used in Evaluation Process

Hello, is there any way to get the tokens used during evaluation process?

· Sorted by Oldest

Dustin N.
·
Hi Kushal! Some SDKs will return the number of (like OpenAI's) tokens used after making a request. Because the tokenizer and token counting strategies vary from model to model you might have to refer to the model vendor's documentation to estimate token counts
Gaurav J.
·
Dustin N. What if we want tokens used by phoenix during evaluation of hallucination,relevancy?
Dustin N.
·
For now we don't provide that functionality, you might consider extending our model wrapper classes to log this information manually or submit an issue to Phoenix

Dustin N.
·
Hi Kushal! Some SDKs will return the number of (like OpenAI's) tokens used after making a request. Because the tokenizer and token counting strategies vary from model to model you might have to refer to the model vendor's documentation to estimate token counts
Gaurav J.
·
Dustin N. What if we want tokens used by phoenix during evaluation of hallucination,relevancy?
Dustin N.
·
For now we don't provide that functionality, you might consider extending our model wrapper classes to log this information manually or submit an issue to Phoenix