Daniel

Commented on Enhancing Playground: Dynamic Examples in Prompt U...·Posted inPhoenix Support

Thanks for sharing your thoughts 🙏

Commented on Enhancing Playground: Dynamic Examples in Prompt U...·Posted inPhoenix Support

The issue with a potential first class integration that I see is that you not only have to take care of the retrieval but you also have to come up with an option that allows the user to perform the indexing as well (as opposed to the option to execute custom code where this would be passed towards the user). Curious to hear further thoughts on this 🙂

Commented on Enhancing Playground: Dynamic Examples in Prompt U...·Posted inPhoenix Support

Daniel

No nothing specific. In fact I perform a hybrid search at the moment (bm25, embeddings), but does the type of retrieval really matter? Actually wouldn't it be easier to provide access to a function (and then the user can decide what this function returns). Or do you have any ideas in mind?

Commented on Enhancing Playground: Dynamic Examples in Prompt U...·Posted inPhoenix Support

Daniel

Anthony P. Thanks for your reply. The use case is information extraction. Depending on the input text I would like to inject those few-shot examples that are close in terms of semantic similarity, so that the model can learn from examples that show successful extractions of similar input texts. Does this make sense?

Posted in Phoenix Support·

Daniel

Enhancing Playground: Dynamic Examples in Prompt Usage

Hi, I really love the Playground feature where you can compare different prompt versions, exchange models etc.. From what I've seen so far, though, the prompt in use is always static (particularly, you can create variables and inject them into the prompt), but there is no way to integrate e.g. dynamic few shot examples (meaning: having a retrieval system in place that depending on the input retrieves semantically similar examples and injects the best candidates into the prompt) is it? Thanks for a short reply! 🙏

22Comments

Commented on Troubleshooting AsyncIO Event Loop Error in Phoeni...·Posted inPhoenix Support

Daniel

Sure will keep you updated. Your help is highly appreciated 🙏

Commented on Troubleshooting AsyncIO Event Loop Error in Phoeni...·Posted inPhoenix Support

Daniel

That sounds promising! Thanks a lot, will check that out 🙌

Commented on Troubleshooting AsyncIO Event Loop Error in Phoeni...·Posted inPhoenix Support

Daniel

Dustin N. Yes that's correct because I want the user to decide which metrics he wants to calculate and thus encapsulate it as much as possible (of course at some costs)

Commented on Troubleshooting AsyncIO Event Loop Error in Phoeni...·Posted inPhoenix Support

Daniel

The program is organized as follows:

    extractor: Extractor = Extractor()
    metrics_and_loaders: list[tuple[Metric, Loader]] = [(AnswerRelevancyMetric(), AnswerRelevancyLoader())]

    for metric, loader in metrics_and_loaders:
        for evaluation_input_with_id in extractor.extract(extraction_strategy=PhoenixSpansAsEvaluationInput()):
            try:
                loader.load(
                    evaluation_output=EvaluationOutput.from_evaluation_result(
                        evaluation_result=metric.evaluate(evaluation_input_with_id),
                        span_id=evaluation_input_with_id.span_id,
                    )
                )

extractor gets the span_df from phoenix client
metrics contain phoenix eval metrics, ragas metrics and huggingface evaluate metrics such as bert_score
loader writes every metric back to the phoenix client
evaluate is a wrapper around the calculation depending on the chosen framework, e.g. for phoenix:

self.__single_evaluation_dataframe_to_evaluation_result(
            run_evals(
                dataframe=PhoenixEvaluationAdapter.evaluation_input_to_phoenix_input(evaluation_input).to_dataframe(),
                evaluators=[self.get_evaluator(self.__chat_model)],
                provide_explanation=True,
                verbose=True,
                concurrency=1,
            )
        )

Does this make sense?

Posted in Phoenix Support·

Daniel

Troubleshooting AsyncIO Event Loop Error in PhoenixEvals Implementation

Hi, I implemented PhoenixEvals as follows:

for index, d in enumerate(self.__ragas_dicts):
     if d.get("ground_truth")[0] is not None:
         result = evaluate(
               dataset=Dataset.from_dict(d),
               metrics=self.__metrics)
         results.append(result)

As part of my main script there are also some other calculations ongoing, e.g., calculating a bert_score.

bert_score_dict = metric.compute(predictions=[pred], references=[ref], lang="de")

Depending on the order I sometimes get the error: ERROR:asyncio:Task exception was never retrieved future: <Task finished name='Task-54' coro=<AsyncClient.aclose() done, RuntimeError: Event loop is closed Anyone with an idea how to avoid this issue? What's happening?

10Comments