Feature Request: Control Prompt Version Validity in Phoenix Cache

Graham R. · 2025-09-24T18:47:47.639Z

Hi — just yapping about my Phoenix setup and maybe a feature request here. I’ve set up my AI product to use Prompt Management in Phoenix, and to cache outputs using the prompt version marked as ‘production’. Only problem with my setup now is that previous versions of my prompt that aren’t tagged as ‘production’ are immediately invalidated in my cache. Invalidating old results using an old prompt could cost my company a good amount of money. So I’d love to be able to control which prompt versions are valid in my cache with Phoenix. I know this is probably a very niche use case for prompt versioning 😆 but I’d love if there was a way to label prompt versions with a non-unique label, similar to what was just released with the prompts themselves.

Feature Request: Control Prompt Version Validity in Phoenix Cache | Arize AI Community

8 comments

· Sorted by Oldest

RunLLM
APP
·
·
Phoenix currently supports tagging prompt versions with unique tags per prompt (e.g., "production", "staging", or custom tags), but does not support non-unique or shared labels across multiple prompt versions. Tags are unique per prompt, and you cannot have two tags with the same name for the same prompt. This means you cannot use a non-unique label to group multiple prompt versions for cache validation purposes as described in your use case. However, prompt labels (for organizing/searching prompts themselves, not versions) are under development, and prompt version tagging is the recommended way to mark versions for environments or releases at this time. Feature requests for more flexible labeling have been discussed, but are not yet implemented for prompt versions specifically—only unique tags are supported per version per prompt as of now. See Phoenix prompt tagging docs and GitHub issue on prompt management improvements.Would you like more detail on current prompt version tagging, or on the roadmap for prompt labels and related features?Sources: - Phoenix prompt tagging docs - GitHub issue on prompt management improvements
💡 Hint: Mention RunLLM in the thread for followups.
Graham R.
·
·
Regardless I’d be willing to contribute a PR for something like this!
🔥1
Mikyo
·
·
Hey Graham R. - great to hear you've started leveraging the prompt management in phoenix! Glad to capture your feedback as feature request and. figure out what caching capabilities you're looking for. I'm not 100% sure I understand what you've setup. So you have some sort of ttl cache for prompt that you pull via a tag. I'm not sure I fully grok this line:
So I’d love to be able to control which prompt versions are valid in my cache with Phoenix.
You're looking for caches in the client SDK specifically or server-side caching?
Graham R.
·
·
Yeah, apologies, I’m being pretty vague about what I’ve built 🙂 Basically, some AI process runs on a bunch of queued data, say it produces a summary for each record. I currently store info about the phoenix prompt version used for the summary. If the record is re-queued, and a summary already exists with the existing ‘production’ tagged prompt, we just return the already created summary. Say I update my prompt, because I like what it outputs better, and set that new version to production. My pipeline would need to generate new summaries since no summaries exist using the new production prompt (nothing to get from our DB / caching system). But, if I was able to check in Phoenix for prompt versions that are still labeled ‘valid’, I could give the system calling me an already generated summary. It’d also give me control of dropping old versions of prompts. Make sense? Again, the problem and solution might be niche, but I’m willing to implement it.
Mikyo
·
·
I see, that does make sense. I wonder if I can suggest a possible alternative approach? You could keep your production tagging in place but you could also create your own custom tag for the prompt at the begging of your job and tag the prompt as having run on XYZ. You could then always pull this "pool" of prompts and keep those cached? I think this sorta solves the "my prompt is still valid" question.
👍1
Graham R.
·
·
Right 🤔 I’ve looked into using custom tags a couple weeks ago. Version tags are forced to be unique, which is unfortunate for this, but we can work around it. So a potential flow could be using your rest API to get all prompt versions for a prompt, then calling it again for each version to see which ones have a tag with a ‘valid-’ prefix…
Mikyo
·
·
right, it would work kinda like git release tags. They do have to be unique ofc.
Graham R.
·
·
I appreciate your thoughts
1

RunLLM
APP
·
·
Phoenix currently supports tagging prompt versions with unique tags per prompt (e.g., "production", "staging", or custom tags), but does not support non-unique or shared labels across multiple prompt versions. Tags are unique per prompt, and you cannot have two tags with the same name for the same prompt. This means you cannot use a non-unique label to group multiple prompt versions for cache validation purposes as described in your use case. However, prompt labels (for organizing/searching prompts themselves, not versions) are under development, and prompt version tagging is the recommended way to mark versions for environments or releases at this time. Feature requests for more flexible labeling have been discussed, but are not yet implemented for prompt versions specifically—only unique tags are supported per version per prompt as of now. See Phoenix prompt tagging docs and GitHub issue on prompt management improvements.Would you like more detail on current prompt version tagging, or on the roadmap for prompt labels and related features?Sources: - Phoenix prompt tagging docs - GitHub issue on prompt management improvements
💡 Hint: Mention RunLLM in the thread for followups.
Graham R.
·
·
Regardless I’d be willing to contribute a PR for something like this!
🔥1
Mikyo
·
·
Hey Graham R. - great to hear you've started leveraging the prompt management in phoenix! Glad to capture your feedback as feature request and. figure out what caching capabilities you're looking for. I'm not 100% sure I understand what you've setup. So you have some sort of ttl cache for prompt that you pull via a tag. I'm not sure I fully grok this line:
So I’d love to be able to control which prompt versions are valid in my cache with Phoenix.
You're looking for caches in the client SDK specifically or server-side caching?
Graham R.
·
·
Yeah, apologies, I’m being pretty vague about what I’ve built 🙂 Basically, some AI process runs on a bunch of queued data, say it produces a summary for each record. I currently store info about the phoenix prompt version used for the summary. If the record is re-queued, and a summary already exists with the existing ‘production’ tagged prompt, we just return the already created summary. Say I update my prompt, because I like what it outputs better, and set that new version to production. My pipeline would need to generate new summaries since no summaries exist using the new production prompt (nothing to get from our DB / caching system). But, if I was able to check in Phoenix for prompt versions that are still labeled ‘valid’, I could give the system calling me an already generated summary. It’d also give me control of dropping old versions of prompts. Make sense? Again, the problem and solution might be niche, but I’m willing to implement it.
Mikyo
·
·
I see, that does make sense. I wonder if I can suggest a possible alternative approach? You could keep your production tagging in place but you could also create your own custom tag for the prompt at the begging of your job and tag the prompt as having run on XYZ. You could then always pull this "pool" of prompts and keep those cached? I think this sorta solves the "my prompt is still valid" question.
👍1
Graham R.
·
·
Right 🤔 I’ve looked into using custom tags a couple weeks ago. Version tags are forced to be unique, which is unfortunate for this, but we can work around it. So a potential flow could be using your rest API to get all prompt versions for a prompt, then calling it again for each version to see which ones have a tag with a ‘valid-’ prefix…
Mikyo
·
·
right, it would work kinda like git release tags. They do have to be unique ofc.
Graham R.
·
·
I appreciate your thoughts
1