Skip to content
NewSecurity Audit for AI-Built Apps. Fixed scope, clear report
GeekTech
AI Engineering

Prompt Versioning: How to Manage Prompts in Production

Prompt versioning in practice: keep prompts in code, log the version with every response, gate changes on evals and roll out behind a flag you can revert.

7 min read

GeekTech Engineering Team18 years building and securing software

Manage prompts in production the way you manage code. Keep each prompt in your repository as a named builder function, give it a version, review every change in a pull request and run your evaluation set before it ships. Log the prompt version with every response, and put risky changes behind a flag so rollback is a config change.

Prompt versioning matters because the prompt is the part of an LLM feature that changes most often and fails most silently. A one-line edit can shift output format or refusal behavior across every request, and nothing raises an error.

Why prompt changes break production features

A prompt edit is a behavior change with no type checker. Nothing fails at build time. The damage shows up later, in a parser or a support queue.

Some hypothetical edits that look harmless:

  • Adding "be concise" to a support assistant, which drops the refund policy reference from replies.
  • Rewording a classification instruction, which moves borderline tickets into another category.
  • Editing a shared "base instructions" fragment for one feature, which changes four others that import it.

Each is easy to read in a diff and hard to catch in a few manual tries.

Where to store prompts: code, config or a prompt registry

Where a prompt lives decides who can change behavior and whether those changes pass review and tests.

Where the prompt livesWho can change itReview and testsRollback
Repository codeEngineers, through pull requestsCode review and evals in CIRevert, or switch a flag
Config or feature flag serviceAnyone with access to the serviceOnly if you add an approval and eval stepSwitch the flag back
Vendor-hosted prompt registryAnyone with dashboard accessVendor tooling, outside your CIPin a version, if supported

Code is the default we'd recommend for product features. OpenAI's guide to migrating from prompt objects now says the same: move prompt content into source code so changes go through the same review and release process as product logic, with versioning handled by commits, pull request review and evals.

Config storage suits wording that product or support staff adjust often, as long as a change there triggers the same eval run and approval. A hosted registry adds a runtime dependency: if the service is down or retired, requests that reference it fail.

How to version prompts in production

Keep prompts in one module, one named function per prompt. Pass dynamic values as typed arguments, so a missing field fails in your code before it reaches the model.

// prompts/support-reply.ts (example)
const SYSTEM = "You answer customer support tickets. Follow the refund policy below exactly.";
 
export const supportReply = {
  name: "support-reply",
  version: "2026-11-18.1",
  build: (input: { customerName: string; ticketText: string }) => [
    { role: "system", content: SYSTEM },
    { role: "user", content: `<ticket>\n${input.ticketText}\n</ticket>\nReply to ${input.customerName}.` },
  ],
};

A few rules make the version useful:

  • Bump the version in the same pull request as the text change. A content hash computed at build time also works and can't be forgotten.
  • Release the prompt, model ID, request parameters and tool definitions together, so production always runs a known combination. Our post on switching LLM models without breaking production covers the model side.
  • Log the prompt name and version with every response, next to the model ID. You can then rebuild the exact prompt behind a bad output from the version and the logged inputs.
  • Give each prompt an owner who approves changes, the way code owners guard a payments module.

Put static instructions first and per-request content last, because provider caches match on the exact prefix. Anthropic's prompt caching documentation walks through a timestamp inside the cached block: the prefix differs on every request, so there's never a cache hit. Each edit to the static part also invalidates the cache once.

Keep secrets and access rules out of the prompt

A prompt ends up in your repository, your logs and sometimes in front of users who ask the model to repeat it. Write it assuming it will be read.

The OWASP entry on system prompt leakage says the system prompt should not be considered a secret or used as a security control. Its mitigations include keeping credentials, database names, user roles and permission structures out of prompts, and enforcing authorization checks outside the model.

Add these checks to prompt reviews:

  1. No API keys, connection strings or internal hostnames in the template.
  2. No instruction that stands in for an access check, such as "only show admins billing data". Filter the data in code before it reaches the prompt.
  3. Untrusted input is wrapped and labeled as data, never mixed into the instructions. Prompt injection explained for product teams covers why delimiters reduce mistakes but don't stop a determined attacker.

Test every prompt change before release

Run the evaluation set on every pull request that touches a prompt, a shared fragment or model settings. The set should hold real production inputs, hard cases and every past failure, each with a pass rule. Our guide to evaluating LLM output quality in production explains how to build it from logged traffic.

Compare the new version with the current one on the same inputs:

  • Schema validation pass rate.
  • Accuracy on labeled cases, with hard cases reported separately.
  • Refusal rate, including refusals that shouldn't happen.
  • Token usage and response length.
  • Results on your security test cases.

Agree on the pass bar before running, for example "no drop on hard cases and no new schema failures". A change to a shared fragment should run the suites of every feature that imports it.

How to roll out a prompt change

Production traffic contains inputs your eval set doesn't. For high-volume or high-impact prompts, release in stages:

  1. Ship both versions. Deploy the new prompt next to the old one, selected by a flag.
  2. Assign by user or tenant. Route a small share of traffic to the new version, keeping each user on one version.
  3. Watch the same metrics. Track validation failures, user corrections, escalations and token usage, split by prompt version.
  4. Widen in steps. Increase the share after each step holds through a normal week of traffic.
  5. Delete the old version. Remove it once the new one is at full traffic, so the module doesn't fill with dead variants.

Small fixes to low-risk prompts can ship once evals pass. Decide which prompts need staged rollout when you assign owners.

Moving off OpenAI prompt objects before November 30, 2026

If your product calls saved prompts by ID and version through the OpenAI API, this has a deadline. OpenAI's migration guide states that reusable prompt objects are deprecated and the v1/prompts endpoint is scheduled to shut down on November 30, 2026.

A migration checklist:

  1. Find every API call that passes a saved prompt reference instead of messages.
  2. Record which prompt ID and version each environment calls. Staging and production often differ.
  3. Copy the exact text of each version in use, with its variables. In OpenAI's before-and-after example, the saved-prompt request names no model while the migrated one does, so confirm each version's model and settings and set them explicitly.
  4. Rebuild each prompt as a typed builder function.
  5. Run your eval set through both paths and compare before the shutdown date.
  6. Deploy the code-based prompt behind a flag, then remove the saved-prompt call.

Common mistakes in prompt management

  • Editing a production prompt in a dashboard with no record of what changed.
  • Assembling prompts from fragments spread across the codebase, so nobody can see the full text a request sent.
  • Logging the model ID but not the prompt version, which leaves incident investigations guessing.
  • Changing the prompt and the model in one release, so a regression has two possible causes.
  • Putting dates or request IDs at the start of the system prompt, which defeats caching.
  • Writing access rules into the prompt instead of enforcing them in code.

Frequently asked questions

Should non-engineers be allowed to edit production prompts?

Yes, if their edits pass the same gate as code. Product and support staff often write better domain instructions. Give them a way to propose changes that runs the eval set and needs the prompt owner's approval before release.

Do we need a dedicated prompt management tool?

Not to start. A prompts module, a version field, logging and evals in CI cover the core needs. Consider a tool when many people outside engineering edit prompts, and check that it can pin versions and trigger your evals.

Can we A/B test two prompt versions in production?

Yes. Both versions should pass your eval set first, and the metric that picks the winner should be fixed in advance. Assign versions per user rather than per request, and run the test across a full week of traffic.

If you're setting up prompt versioning and evals for an LLM feature, our LLM apps, agents and RAG service builds evaluation suites and guardrails that catch regressions on every change.

Insights

Related insights

More on AI Engineering →
  • AI Engineering

    How to Switch LLM Models Without Breaking Production

    How to switch LLM models safely: pin model IDs, track retirement dates, rerun your evals, roll out behind a flag and keep a rollback path for every change.

    7 min read

  • AI Engineering

    AI Agent vs Workflow: When to Use an AI Agent

    AI agent vs workflow: when a fixed LLM workflow is enough, when an agent is worth the extra latency and risk, and how to add autonomy one step at a time.

    7 min read

  • AI Engineering

    Build vs. Buy AI Features: How to Decide

    Build vs. buy for AI features: when a vendor product fits, when to build on a model API, and the running costs and exit risks to check before you commit.

    7 min read

Get AI security and automation insights, monthly

AI security, automation and governance, written by engineers. No spam, unsubscribe any time.

Book a free 30-min call