Implementing a Model Router for LLM Selection Based on User Queries

Is there any model router implementation that folks use for determining which LLM to use based on user's query?

· Sorted by Oldest

Jason
·
That is a good question ^
Deepanshu
·
Any ideas?
Jason
·
Deepanshu Do you currently use a gateway like LiteLLM? Is your question on the LLM gateway layer or more on the dynamic “choice” of what to use I think it’s pretty use case dependent how you might build this. Depending on volume and delays needed In co-pilot in Arize we leverage function calling to route to skills, some skills leverage reasoning models now. The decision on what to use is the function call “routing”
Jason
·
GPT4o is main routing model
Deepanshu
·
Hi Jason its more on the dynamic choice of which model to use out of GPT-4o, GPT-4o-mini, Claude 3.5, Llama 3.5, Mistral Small etc
Jason
·
We use GPT-4o for this purpose in our product. In our case we use tools definition for routing to "skills" but you could also just use a normal LLM call as well (if you don't need parameter extraction)
Deepanshu
·
Yeah makes sense

Jason
·
That is a good question ^
Deepanshu
·
Any ideas?
Jason
·
Deepanshu Do you currently use a gateway like LiteLLM? Is your question on the LLM gateway layer or more on the dynamic “choice” of what to use I think it’s pretty use case dependent how you might build this. Depending on volume and delays needed In co-pilot in Arize we leverage function calling to route to skills, some skills leverage reasoning models now. The decision on what to use is the function call “routing”
Jason
·
GPT4o is main routing model
Deepanshu
·
Hi Jason its more on the dynamic choice of which model to use out of GPT-4o, GPT-4o-mini, Claude 3.5, Llama 3.5, Mistral Small etc
Jason
·
We use GPT-4o for this purpose in our product. In our case we use tools definition for routing to "skills" but you could also just use a normal LLM call as well (if you don't need parameter extraction)
Deepanshu
·
Yeah makes sense