I'm sure I've done something wrong... anyone have any suggestion?

Pal B.

·Mar 25, 2024 12:11 PM

6 comments

· Sorted by Oldest

Aman K.
·
Hi Pal B.! Thanks for checking out Arize - I see you’ve logged traces already, which is great! I noticed you also logged “flat” rows i.e. inferences (or generations) in addition to traces. The issue lies in the data model for dolphin-mixtral being set to tracing , but then also receiving inferences. I’m just curious, was there a reason to log both types of data models to the same model in platform? For what its worth - going forward you could ingest traces in 1 model and inferences/rows/generations to another model in the platform and that should work fine. Also if its helpful, feel free to grab time on my calendar here and I can try to help - would also love to learn more about your use case: https://calendly.com/aman-arize/15-min?back=1&month=2024-03
Pal B.
·
Thanks for coming back, Aman. As for tracing & inference monitoring, no particular reason really, although I have been through the docs but can't recall seeing it's be a twilight zone. my bad. So essentially, if clear the mess out of my code, it would work again (either/or tracing ofc)
Pal B.
·
my use case is baselining on drifts and hallucinations with a select list of models
Pal B.
·
I have done a few years sec engineer delivering on various q/a testing of software and equipment etc. - the point being, I want to find out how an uncensored v censored model performs under certain stress scenarios, hence the choice of dolphin-mixtral, to begin width
Pal B.
·
this is all off the back of a research project I'm leading at work into reducing model inversion/injection risks, and so on
Pal B.
·
I find your platform very pleasant, easy to use. Saying this as I had gone through a list of llm observability tools recently

Aman K.
·
Hi Pal B.! Thanks for checking out Arize - I see you’ve logged traces already, which is great! I noticed you also logged “flat” rows i.e. inferences (or generations) in addition to traces. The issue lies in the data model for dolphin-mixtral being set to tracing , but then also receiving inferences. I’m just curious, was there a reason to log both types of data models to the same model in platform? For what its worth - going forward you could ingest traces in 1 model and inferences/rows/generations to another model in the platform and that should work fine. Also if its helpful, feel free to grab time on my calendar here and I can try to help - would also love to learn more about your use case: https://calendly.com/aman-arize/15-min?back=1&month=2024-03
Pal B.
·
Thanks for coming back, Aman. As for tracing & inference monitoring, no particular reason really, although I have been through the docs but can't recall seeing it's be a twilight zone. my bad. So essentially, if clear the mess out of my code, it would work again (either/or tracing ofc)
Pal B.
·
my use case is baselining on drifts and hallucinations with a select list of models
Pal B.
·
I have done a few years sec engineer delivering on various q/a testing of software and equipment etc. - the point being, I want to find out how an uncensored v censored model performs under certain stress scenarios, hence the choice of dolphin-mixtral, to begin width
Pal B.
·
this is all off the back of a research project I'm leading at work into reducing model inversion/injection risks, and so on
Pal B.
·
I find your platform very pleasant, easy to use. Saying this as I had gone through a list of llm observability tools recently