[{"data":1,"prerenderedAt":1132},["ShallowReactive",2],{"article-alternates":3,"article-\u002Fen\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops-discipline":13},{"i18nKey":4,"paths":5},"ai-004-2026-07",{"de":6,"en":7,"es":8,"fr":9,"it":10,"ru":11,"tr":12},"\u002Fde\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops","\u002Fen\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops-discipline","\u002Fes\u002Fai\u002Fversionamiento-prompts-ab-testing-llm","\u002Ffr\u002Fai\u002Fversionierung-und-ab-tests-von-prompts","\u002Fit\u002Fai\u002Fprompt-versiyonlama-ab-testi-llm-disiplini","\u002Fru\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops","\u002Ftr\u002Fai\u002Fprompt-versiyonlama-ve-a-b-testi-llm-operasyonun-disiplini",{"_path":7,"_dir":14,"_draft":15,"_partial":15,"_locale":16,"title":17,"description":18,"publishedAt":19,"modifiedAt":19,"category":14,"i18nKey":4,"tags":20,"readingTime":26,"author":27,"body":28,"_type":811,"_id":1127,"_source":1128,"_file":1129,"_stem":1130,"_extension":1131},"ai",false,"","Prompt Versioning and A\u002FB Testing: The Discipline of LLM Operations","How to systematically test LLM outputs with Promptfoo and LangSmith. Building evaluation pipelines for production-grade AI applications.","2026-07-30",[21,22,23,24,25],"llm-ops","prompt-engineering","evaluation","ab-testing","mlops",8,"Roibase",{"type":29,"children":30,"toc":1113},"root",[31,39,46,51,64,69,75,89,102,434,439,446,451,456,462,467,483,489,494,499,505,510,543,553,558,593,598,604,609,614,766,771,776,782,795,803,808,983,988,1001,1007,1012,1017,1022,1077,1082,1096,1102,1107],{"type":32,"tag":33,"props":34,"children":35},"element","p",{},[36],{"type":37,"value":38},"text","The moment you deploy LLMs to production, you realize you need the same test suite discipline of classical software engineering. When you change a prompt, what happens to output consistency? How does the cost-quality tradeoff shift when you upgrade your model version? How do you turn the feeling that \"Claude gave a better answer\" into a measurable metric? In 2026, when LLM operations have matured, those who answer these questions systematically—not manually—win. Tools like Promptfoo and LangSmith, plus evaluation pipelines, are the insurance policy for keeping LLMs in production.",{"type":32,"tag":40,"props":41,"children":43},"h2",{"id":42},"prompt-change-code-change",[44],{"type":37,"value":45},"Prompt Change = Code Change",{"type":32,"tag":33,"props":47,"children":48},{},[49],{"type":37,"value":50},"You have a marketing content generation workflow. You send a prompt to Claude's API, get a blog draft back. Version one says \"write,\" version two adds \"write for Roibase, use engineering tone\" to the system prompt. Version three adds a \"BANNED WORDS\" list. Each change affects the output—but how do you measure the impact?",{"type":32,"tag":33,"props":52,"children":53},{},[54,56,62],{"type":37,"value":55},"In classical software, you have unit tests—fixed input, deterministic output. With LLMs, input is fixed but output is stochastic. You can't decide based on a single run. You run the same prompt 10 times with different seeds, look at average token count, latency, coherence score. That's why ",{"type":32,"tag":57,"props":58,"children":59},"strong",{},[60],{"type":37,"value":61},"prompt versioning",{"type":37,"value":63}," is as critical as code versioning. You track prompt changes with Git commits but might not track output changes. Evaluation suites solve this: every commit triggers automatic tests, you see metric regressions instantly.",{"type":32,"tag":33,"props":65,"children":66},{},[67],{"type":37,"value":68},"Concrete scenario: your n8n workflow generates content with Claude. When you change \"1500 words\" to \"1400-1600 words,\" average output drops from 1520 to 1480 words, token cost drops 3%, but readability score loses 0.2 points. Seeing this tradeoff without manual trial-and-error requires an automated eval pipeline.",{"type":32,"tag":40,"props":70,"children":72},{"id":71},"promptfoo-regression-test-suite-for-prompts",[73],{"type":37,"value":74},"Promptfoo: Regression Test Suite for Prompts",{"type":32,"tag":33,"props":76,"children":77},{},[78,80,87],{"type":37,"value":79},"Promptfoo is an open-source CLI tool—you define prompts in YAML config, provide test cases in CSV or JSON, write assertions. The ",{"type":32,"tag":81,"props":82,"children":84},"code",{"className":83},[],[85],{"type":37,"value":86},"promptfoo eval",{"type":37,"value":88}," command runs all variants, outputs a success\u002Ffailure table.",{"type":32,"tag":33,"props":90,"children":91},{},[92,94,100],{"type":37,"value":93},"A typical ",{"type":32,"tag":81,"props":95,"children":97},{"className":96},[],[98],{"type":37,"value":99},"promptfoo.yaml",{"type":37,"value":101}," looks like this:",{"type":32,"tag":103,"props":104,"children":108},"pre",{"code":105,"language":106,"meta":16,"className":107,"style":16},"prompts:\n  - id: baseline\n    text: \"Write a blog post about {{topic}}\"\n  - id: roibase-tone\n    text: \"Write a blog post about {{topic}}. Use engineering discipline tone. No hype words.\"\n\nproviders:\n  - anthropic:messages:claude-3-5-sonnet-20241022\n\ntests:\n  - vars:\n      topic: \"server-side GTM setup\"\n    assert:\n      - type: contains\n        value: \"first-party\"\n      - type: javascript\n        value: output.length > 1400 && output.length \u003C 1600\n      - type: cost\n        threshold: 0.05\n","yaml","language-yaml shiki shiki-themes github-dark",[109],{"type":32,"tag":81,"props":110,"children":111},{"__ignoreMap":16},[112,130,155,173,194,211,221,234,246,254,267,284,302,315,338,356,377,394,415],{"type":32,"tag":113,"props":114,"children":117},"span",{"class":115,"line":116},"line",1,[118,124],{"type":32,"tag":113,"props":119,"children":121},{"style":120},"--shiki-default:#85E89D",[122],{"type":37,"value":123},"prompts",{"type":32,"tag":113,"props":125,"children":127},{"style":126},"--shiki-default:#E1E4E8",[128],{"type":37,"value":129},":\n",{"type":32,"tag":113,"props":131,"children":133},{"class":115,"line":132},2,[134,139,144,149],{"type":32,"tag":113,"props":135,"children":136},{"style":126},[137],{"type":37,"value":138},"  - ",{"type":32,"tag":113,"props":140,"children":141},{"style":120},[142],{"type":37,"value":143},"id",{"type":32,"tag":113,"props":145,"children":146},{"style":126},[147],{"type":37,"value":148},": ",{"type":32,"tag":113,"props":150,"children":152},{"style":151},"--shiki-default:#9ECBFF",[153],{"type":37,"value":154},"baseline\n",{"type":32,"tag":113,"props":156,"children":158},{"class":115,"line":157},3,[159,164,168],{"type":32,"tag":113,"props":160,"children":161},{"style":120},[162],{"type":37,"value":163},"    text",{"type":32,"tag":113,"props":165,"children":166},{"style":126},[167],{"type":37,"value":148},{"type":32,"tag":113,"props":169,"children":170},{"style":151},[171],{"type":37,"value":172},"\"Write a blog post about {{topic}}\"\n",{"type":32,"tag":113,"props":174,"children":176},{"class":115,"line":175},4,[177,181,185,189],{"type":32,"tag":113,"props":178,"children":179},{"style":126},[180],{"type":37,"value":138},{"type":32,"tag":113,"props":182,"children":183},{"style":120},[184],{"type":37,"value":143},{"type":32,"tag":113,"props":186,"children":187},{"style":126},[188],{"type":37,"value":148},{"type":32,"tag":113,"props":190,"children":191},{"style":151},[192],{"type":37,"value":193},"roibase-tone\n",{"type":32,"tag":113,"props":195,"children":197},{"class":115,"line":196},5,[198,202,206],{"type":32,"tag":113,"props":199,"children":200},{"style":120},[201],{"type":37,"value":163},{"type":32,"tag":113,"props":203,"children":204},{"style":126},[205],{"type":37,"value":148},{"type":32,"tag":113,"props":207,"children":208},{"style":151},[209],{"type":37,"value":210},"\"Write a blog post about {{topic}}. Use engineering discipline tone. No hype words.\"\n",{"type":32,"tag":113,"props":212,"children":214},{"class":115,"line":213},6,[215],{"type":32,"tag":113,"props":216,"children":218},{"emptyLinePlaceholder":217},true,[219],{"type":37,"value":220},"\n",{"type":32,"tag":113,"props":222,"children":224},{"class":115,"line":223},7,[225,230],{"type":32,"tag":113,"props":226,"children":227},{"style":120},[228],{"type":37,"value":229},"providers",{"type":32,"tag":113,"props":231,"children":232},{"style":126},[233],{"type":37,"value":129},{"type":32,"tag":113,"props":235,"children":236},{"class":115,"line":26},[237,241],{"type":32,"tag":113,"props":238,"children":239},{"style":126},[240],{"type":37,"value":138},{"type":32,"tag":113,"props":242,"children":243},{"style":151},[244],{"type":37,"value":245},"anthropic:messages:claude-3-5-sonnet-20241022\n",{"type":32,"tag":113,"props":247,"children":249},{"class":115,"line":248},9,[250],{"type":32,"tag":113,"props":251,"children":252},{"emptyLinePlaceholder":217},[253],{"type":37,"value":220},{"type":32,"tag":113,"props":255,"children":257},{"class":115,"line":256},10,[258,263],{"type":32,"tag":113,"props":259,"children":260},{"style":120},[261],{"type":37,"value":262},"tests",{"type":32,"tag":113,"props":264,"children":265},{"style":126},[266],{"type":37,"value":129},{"type":32,"tag":113,"props":268,"children":270},{"class":115,"line":269},11,[271,275,280],{"type":32,"tag":113,"props":272,"children":273},{"style":126},[274],{"type":37,"value":138},{"type":32,"tag":113,"props":276,"children":277},{"style":120},[278],{"type":37,"value":279},"vars",{"type":32,"tag":113,"props":281,"children":282},{"style":126},[283],{"type":37,"value":129},{"type":32,"tag":113,"props":285,"children":287},{"class":115,"line":286},12,[288,293,297],{"type":32,"tag":113,"props":289,"children":290},{"style":120},[291],{"type":37,"value":292},"      topic",{"type":32,"tag":113,"props":294,"children":295},{"style":126},[296],{"type":37,"value":148},{"type":32,"tag":113,"props":298,"children":299},{"style":151},[300],{"type":37,"value":301},"\"server-side GTM setup\"\n",{"type":32,"tag":113,"props":303,"children":305},{"class":115,"line":304},13,[306,311],{"type":32,"tag":113,"props":307,"children":308},{"style":120},[309],{"type":37,"value":310},"    assert",{"type":32,"tag":113,"props":312,"children":313},{"style":126},[314],{"type":37,"value":129},{"type":32,"tag":113,"props":316,"children":318},{"class":115,"line":317},14,[319,324,329,333],{"type":32,"tag":113,"props":320,"children":321},{"style":126},[322],{"type":37,"value":323},"      - ",{"type":32,"tag":113,"props":325,"children":326},{"style":120},[327],{"type":37,"value":328},"type",{"type":32,"tag":113,"props":330,"children":331},{"style":126},[332],{"type":37,"value":148},{"type":32,"tag":113,"props":334,"children":335},{"style":151},[336],{"type":37,"value":337},"contains\n",{"type":32,"tag":113,"props":339,"children":341},{"class":115,"line":340},15,[342,347,351],{"type":32,"tag":113,"props":343,"children":344},{"style":120},[345],{"type":37,"value":346},"        value",{"type":32,"tag":113,"props":348,"children":349},{"style":126},[350],{"type":37,"value":148},{"type":32,"tag":113,"props":352,"children":353},{"style":151},[354],{"type":37,"value":355},"\"first-party\"\n",{"type":32,"tag":113,"props":357,"children":359},{"class":115,"line":358},16,[360,364,368,372],{"type":32,"tag":113,"props":361,"children":362},{"style":126},[363],{"type":37,"value":323},{"type":32,"tag":113,"props":365,"children":366},{"style":120},[367],{"type":37,"value":328},{"type":32,"tag":113,"props":369,"children":370},{"style":126},[371],{"type":37,"value":148},{"type":32,"tag":113,"props":373,"children":374},{"style":151},[375],{"type":37,"value":376},"javascript\n",{"type":32,"tag":113,"props":378,"children":380},{"class":115,"line":379},17,[381,385,389],{"type":32,"tag":113,"props":382,"children":383},{"style":120},[384],{"type":37,"value":346},{"type":32,"tag":113,"props":386,"children":387},{"style":126},[388],{"type":37,"value":148},{"type":32,"tag":113,"props":390,"children":391},{"style":151},[392],{"type":37,"value":393},"output.length > 1400 && output.length \u003C 1600\n",{"type":32,"tag":113,"props":395,"children":397},{"class":115,"line":396},18,[398,402,406,410],{"type":32,"tag":113,"props":399,"children":400},{"style":126},[401],{"type":37,"value":323},{"type":32,"tag":113,"props":403,"children":404},{"style":120},[405],{"type":37,"value":328},{"type":32,"tag":113,"props":407,"children":408},{"style":126},[409],{"type":37,"value":148},{"type":32,"tag":113,"props":411,"children":412},{"style":151},[413],{"type":37,"value":414},"cost\n",{"type":32,"tag":113,"props":416,"children":418},{"class":115,"line":417},19,[419,424,428],{"type":32,"tag":113,"props":420,"children":421},{"style":120},[422],{"type":37,"value":423},"        threshold",{"type":32,"tag":113,"props":425,"children":426},{"style":126},[427],{"type":37,"value":148},{"type":32,"tag":113,"props":429,"children":431},{"style":430},"--shiki-default:#79B8FF",[432],{"type":37,"value":433},"0.05\n",{"type":32,"tag":33,"props":435,"children":436},{},[437],{"type":37,"value":438},"When you run this config, Promptfoo sends both prompts to Claude, checks assertions: Does \"first-party\" appear? Is output between 1400-1600 words? Is API cost under $0.05? If failures exist, it shows which prompt failed. Integrate into CI\u002FCD and prompt changes are automatically tested on every pull request—just like unit tests.",{"type":32,"tag":440,"props":441,"children":443},"h3",{"id":442},"why-automation-not-manual-testing",[444],{"type":37,"value":445},"Why Automation, Not Manual Testing?",{"type":32,"tag":33,"props":447,"children":448},{},[449],{"type":37,"value":450},"Manual: You send Claude 5 different topics, scan outputs by eye, say \"good.\" Next day you change the prompt, test manually again. By iteration 10, you've forgotten which change affected which metric.",{"type":32,"tag":33,"props":452,"children":453},{},[454],{"type":37,"value":455},"Automation: You have 50 test cases (real keywords from GSC), automatic runs on every prompt change. Regression table: \"baseline prompt averages 1520 words, new prompt 1480—2.6% drop.\" Decision is metric-based, not gut-based.",{"type":32,"tag":40,"props":457,"children":459},{"id":458},"langsmith-production-observability",[460],{"type":37,"value":461},"LangSmith: Production Observability",{"type":32,"tag":33,"props":463,"children":464},{},[465],{"type":37,"value":466},"Promptfoo is a development-time testing tool. LangSmith (built by the LangChain team) lets you observe what happens in production. Every LLM call is logged to LangSmith: input, output, latency, token count, metadata. You see traces on the dashboard—retrieval, prompt construction, LLM call, post-processing chain, all step-by-step.",{"type":32,"tag":33,"props":468,"children":469},{},[470,472,481],{"type":37,"value":471},"Example: In Roibase's ",{"type":32,"tag":473,"props":474,"children":478},"a",{"href":475,"rel":476},"https:\u002F\u002Fwww.roibase.com.tr\u002Fen\u002Fgeo",[477],"nofollow",[479],{"type":37,"value":480},"Generative Engine Optimization",{"type":37,"value":482}," work, we track ChatGPT citations in an LLM pipeline. Pipeline: user question → embedding → Pinecone retrieval → context injection → Claude → citation extraction. LangSmith logs every step. If citation rate drops below 15%, an alert fires—prompt drift or retrieval quality issue caught instantly.",{"type":32,"tag":440,"props":484,"children":486},{"id":485},"tracing-vs-logging",[487],{"type":37,"value":488},"Tracing vs Logging",{"type":32,"tag":33,"props":490,"children":491},{},[492],{"type":37,"value":493},"Classical logging: \"Sent prompt to Claude API, got response back.\" Trace: \"Retrieval took 120ms, returned 5 documents, prompt construction 15ms, Claude 2.3s, total latency 2.45s—no SLA breach.\" Traces let you see the end-to-end pipeline. In LLM chains, finding bottlenecks is critical: if retrieval is slow, optimize database index; if LLM is slow, consider model version or reduce prompt token count.",{"type":32,"tag":33,"props":495,"children":496},{},[497],{"type":37,"value":498},"In production A\u002FB tests, LangSmith is indispensable: 50% of traffic gets baseline prompt, 50% gets new prompt—separate trace groups per variant, real-time metric comparison. Baseline 2.1s average latency, new prompt 1.9s but output quality score drops from 0.85 to 0.80—tradeoff table live.",{"type":32,"tag":40,"props":500,"children":502},{"id":501},"evaluation-pipeline-automated-quality-score",[503],{"type":37,"value":504},"Evaluation Pipeline: Automated Quality Score",{"type":32,"tag":33,"props":506,"children":507},{},[508],{"type":37,"value":509},"LLM output is subjective—how do you automate \"is it good or bad?\" Two approaches: rule-based assertions and LLM-as-a-judge.",{"type":32,"tag":33,"props":511,"children":512},{},[513,518,520,526,528,534,535,541],{"type":32,"tag":57,"props":514,"children":515},{},[516],{"type":37,"value":517},"Rule-based:",{"type":37,"value":519}," Promptfoo's ",{"type":32,"tag":81,"props":521,"children":523},{"className":522},[],[524],{"type":37,"value":525},"contains",{"type":37,"value":527},", ",{"type":32,"tag":81,"props":529,"children":531},{"className":530},[],[532],{"type":37,"value":533},"length",{"type":37,"value":527},{"type":32,"tag":81,"props":536,"children":538},{"className":537},[],[539],{"type":37,"value":540},"regex-match",{"type":37,"value":542}," assertions. Rules like \"1400-1600 words,\" \"no exclamation marks,\" \"at least 1 internal link.\" Fast, deterministic—but doesn't measure semantic quality.",{"type":32,"tag":33,"props":544,"children":545},{},[546,551],{"type":32,"tag":57,"props":547,"children":548},{},[549],{"type":37,"value":550},"LLM-as-a-judge:",{"type":37,"value":552}," Use another LLM (usually GPT-4 or Claude) to evaluate output. Example: \"Is this blog post in engineering tone? Score 1-10.\" If judge gives 7.5, pass; if 6, fail. This captures semantic quality but is non-deterministic—the judge model itself is stochastic. Solution: run each eval 3 times, average the scores.",{"type":32,"tag":33,"props":554,"children":555},{},[556],{"type":37,"value":557},"Roibase's content generation pipeline uses this eval flow:",{"type":32,"tag":559,"props":560,"children":561},"ol",{},[562,568,573,578,583,588],{"type":32,"tag":563,"props":564,"children":565},"li",{},[566],{"type":37,"value":567},"Generate blog draft with Claude",{"type":32,"tag":563,"props":569,"children":570},{},[571],{"type":37,"value":572},"Send to Promptfoo",{"type":32,"tag":563,"props":574,"children":575},{},[576],{"type":37,"value":577},"Rule-based checks: word count, internal link count, banned word control",{"type":32,"tag":563,"props":579,"children":580},{},[581],{"type":37,"value":582},"LLM-as-a-judge: GPT-4 scores \"tone alignment 1-10\"",{"type":32,"tag":563,"props":584,"children":585},{},[586],{"type":37,"value":587},"All metrics logged to Notion",{"type":32,"tag":563,"props":589,"children":590},{},[591],{"type":37,"value":592},"Slack alert if average score drops below 8",{"type":32,"tag":33,"props":594,"children":595},{},[596],{"type":37,"value":597},"With this pipeline, when you generate 1000 articles, quality standards hold. Instead of manual QA reading every piece, the team only reviews eval failures—90% time savings.",{"type":32,"tag":40,"props":599,"children":601},{"id":600},"ab-test-two-prompts-two-cost-quality-tradeoffs",[602],{"type":37,"value":603},"A\u002FB Test: Two Prompts, Two Cost-Quality Tradeoffs",{"type":32,"tag":33,"props":605,"children":606},{},[607],{"type":37,"value":608},"In production, prompt A\u002FB testing works like classic feature flagging. Use LaunchDarkly or a custom flag service: 50% of users get prompt_v1, 50% get prompt_v2. Collect metrics per variant: average token count, latency, downstream conversion (does editor approve the draft?).",{"type":32,"tag":33,"props":610,"children":611},{},[612],{"type":37,"value":613},"Concrete example: We're testing a new prompt version with category-specific guidance. Baseline prompt is generic; new prompt includes category-specific instructions. A\u002FB test runs 2 weeks:",{"type":32,"tag":615,"props":616,"children":617},"table",{},[618,647],{"type":32,"tag":619,"props":620,"children":621},"thead",{},[622],{"type":32,"tag":623,"props":624,"children":625},"tr",{},[626,632,637,642],{"type":32,"tag":627,"props":628,"children":629},"th",{},[630],{"type":37,"value":631},"Metric",{"type":32,"tag":627,"props":633,"children":634},{},[635],{"type":37,"value":636},"Baseline",{"type":32,"tag":627,"props":638,"children":639},{},[640],{"type":37,"value":641},"New Prompt",{"type":32,"tag":627,"props":643,"children":644},{},[645],{"type":37,"value":646},"Delta",{"type":32,"tag":648,"props":649,"children":650},"tbody",{},[651,675,698,720,743],{"type":32,"tag":623,"props":652,"children":653},{},[654,660,665,670],{"type":32,"tag":655,"props":656,"children":657},"td",{},[658],{"type":37,"value":659},"Avg tokens (input+output)",{"type":32,"tag":655,"props":661,"children":662},{},[663],{"type":37,"value":664},"3200",{"type":32,"tag":655,"props":666,"children":667},{},[668],{"type":37,"value":669},"3450",{"type":32,"tag":655,"props":671,"children":672},{},[673],{"type":37,"value":674},"+7.8%",{"type":32,"tag":623,"props":676,"children":677},{},[678,683,688,693],{"type":32,"tag":655,"props":679,"children":680},{},[681],{"type":37,"value":682},"Avg latency (sec)",{"type":32,"tag":655,"props":684,"children":685},{},[686],{"type":37,"value":687},"2.1",{"type":32,"tag":655,"props":689,"children":690},{},[691],{"type":37,"value":692},"2.3",{"type":32,"tag":655,"props":694,"children":695},{},[696],{"type":37,"value":697},"+9.5%",{"type":32,"tag":623,"props":699,"children":700},{},[701,706,711,716],{"type":32,"tag":655,"props":702,"children":703},{},[704],{"type":37,"value":705},"Cost\u002Farticle ($)",{"type":32,"tag":655,"props":707,"children":708},{},[709],{"type":37,"value":710},"0.042",{"type":32,"tag":655,"props":712,"children":713},{},[714],{"type":37,"value":715},"0.046",{"type":32,"tag":655,"props":717,"children":718},{},[719],{"type":37,"value":697},{"type":32,"tag":623,"props":721,"children":722},{},[723,728,733,738],{"type":32,"tag":655,"props":724,"children":725},{},[726],{"type":37,"value":727},"Editor approval rate",{"type":32,"tag":655,"props":729,"children":730},{},[731],{"type":37,"value":732},"72%",{"type":32,"tag":655,"props":734,"children":735},{},[736],{"type":37,"value":737},"81%",{"type":32,"tag":655,"props":739,"children":740},{},[741],{"type":37,"value":742},"+12.5%",{"type":32,"tag":623,"props":744,"children":745},{},[746,751,756,761],{"type":32,"tag":655,"props":747,"children":748},{},[749],{"type":37,"value":750},"Internal link accuracy",{"type":32,"tag":655,"props":752,"children":753},{},[754],{"type":37,"value":755},"65%",{"type":32,"tag":655,"props":757,"children":758},{},[759],{"type":37,"value":760},"89%",{"type":32,"tag":655,"props":762,"children":763},{},[764],{"type":37,"value":765},"+36.9%",{"type":32,"tag":33,"props":767,"children":768},{},[769],{"type":37,"value":770},"New prompt is 10% more expensive but editor approval climbs 12.5%—editor revision costs drop. Internal link accuracy surges 36.9%—SEO gains justify the cost. Decision: new prompt wins, move to production.",{"type":32,"tag":33,"props":772,"children":773},{},[774],{"type":37,"value":775},"During the A\u002FB test, LangSmith creates separate trace groups per variant. Spot an anomaly (e.g., new prompt has 5% HTTP 429 rate limit errors)? You catch it immediately.",{"type":32,"tag":40,"props":777,"children":779},{"id":778},"versioning-git-metadata",[780],{"type":37,"value":781},"Versioning: Git + Metadata",{"type":32,"tag":33,"props":783,"children":784},{},[785,787,793],{"type":37,"value":786},"Keep prompt versions in Git like code—but with separate metadata. ",{"type":32,"tag":81,"props":788,"children":790},{"className":789},[],[791],{"type":37,"value":792},"prompts\u002F",{"type":37,"value":794}," folder:",{"type":32,"tag":103,"props":796,"children":798},{"code":797},"prompts\u002F\n  roibase-blog-v1.md\n  roibase-blog-v2.md\n  roibase-blog-v3.md\n",[799],{"type":32,"tag":81,"props":800,"children":801},{"__ignoreMap":16},[802],{"type":37,"value":797},{"type":32,"tag":33,"props":804,"children":805},{},[806],{"type":37,"value":807},"Each file includes frontmatter metadata:",{"type":32,"tag":103,"props":809,"children":813},{"code":810,"language":811,"meta":16,"className":812,"style":16},"---\nversion: 3\nmodel: claude-3-5-sonnet-20241022\ntemperature: 0.7\nmax_tokens: 8000\ncreated: 2026-07-15\ndeprecated: false\ntest_suite: promptfoo-blog-eval.yaml\n---\n\n# ROLE\nYou write for Roibase.\n...\n","markdown","language-markdown shiki shiki-themes github-dark",[814],{"type":32,"tag":81,"props":815,"children":816},{"__ignoreMap":16},[817,825,842,859,876,893,910,927,944,951,958,967,975],{"type":32,"tag":113,"props":818,"children":819},{"class":115,"line":116},[820],{"type":32,"tag":113,"props":821,"children":822},{"style":126},[823],{"type":37,"value":824},"---\n",{"type":32,"tag":113,"props":826,"children":827},{"class":115,"line":132},[828,833,837],{"type":32,"tag":113,"props":829,"children":830},{"style":120},[831],{"type":37,"value":832},"version",{"type":32,"tag":113,"props":834,"children":835},{"style":126},[836],{"type":37,"value":148},{"type":32,"tag":113,"props":838,"children":839},{"style":430},[840],{"type":37,"value":841},"3\n",{"type":32,"tag":113,"props":843,"children":844},{"class":115,"line":157},[845,850,854],{"type":32,"tag":113,"props":846,"children":847},{"style":120},[848],{"type":37,"value":849},"model",{"type":32,"tag":113,"props":851,"children":852},{"style":126},[853],{"type":37,"value":148},{"type":32,"tag":113,"props":855,"children":856},{"style":151},[857],{"type":37,"value":858},"claude-3-5-sonnet-20241022\n",{"type":32,"tag":113,"props":860,"children":861},{"class":115,"line":175},[862,867,871],{"type":32,"tag":113,"props":863,"children":864},{"style":120},[865],{"type":37,"value":866},"temperature",{"type":32,"tag":113,"props":868,"children":869},{"style":126},[870],{"type":37,"value":148},{"type":32,"tag":113,"props":872,"children":873},{"style":430},[874],{"type":37,"value":875},"0.7\n",{"type":32,"tag":113,"props":877,"children":878},{"class":115,"line":196},[879,884,888],{"type":32,"tag":113,"props":880,"children":881},{"style":120},[882],{"type":37,"value":883},"max_tokens",{"type":32,"tag":113,"props":885,"children":886},{"style":126},[887],{"type":37,"value":148},{"type":32,"tag":113,"props":889,"children":890},{"style":430},[891],{"type":37,"value":892},"8000\n",{"type":32,"tag":113,"props":894,"children":895},{"class":115,"line":213},[896,901,905],{"type":32,"tag":113,"props":897,"children":898},{"style":120},[899],{"type":37,"value":900},"created",{"type":32,"tag":113,"props":902,"children":903},{"style":126},[904],{"type":37,"value":148},{"type":32,"tag":113,"props":906,"children":907},{"style":430},[908],{"type":37,"value":909},"2026-07-15\n",{"type":32,"tag":113,"props":911,"children":912},{"class":115,"line":223},[913,918,922],{"type":32,"tag":113,"props":914,"children":915},{"style":120},[916],{"type":37,"value":917},"deprecated",{"type":32,"tag":113,"props":919,"children":920},{"style":126},[921],{"type":37,"value":148},{"type":32,"tag":113,"props":923,"children":924},{"style":430},[925],{"type":37,"value":926},"false\n",{"type":32,"tag":113,"props":928,"children":929},{"class":115,"line":26},[930,935,939],{"type":32,"tag":113,"props":931,"children":932},{"style":120},[933],{"type":37,"value":934},"test_suite",{"type":32,"tag":113,"props":936,"children":937},{"style":126},[938],{"type":37,"value":148},{"type":32,"tag":113,"props":940,"children":941},{"style":151},[942],{"type":37,"value":943},"promptfoo-blog-eval.yaml\n",{"type":32,"tag":113,"props":945,"children":946},{"class":115,"line":248},[947],{"type":32,"tag":113,"props":948,"children":949},{"style":126},[950],{"type":37,"value":824},{"type":32,"tag":113,"props":952,"children":953},{"class":115,"line":256},[954],{"type":32,"tag":113,"props":955,"children":956},{"emptyLinePlaceholder":217},[957],{"type":37,"value":220},{"type":32,"tag":113,"props":959,"children":960},{"class":115,"line":269},[961],{"type":32,"tag":113,"props":962,"children":964},{"style":963},"--shiki-default:#79B8FF;--shiki-default-font-weight:bold",[965],{"type":37,"value":966},"# ROLE\n",{"type":32,"tag":113,"props":968,"children":969},{"class":115,"line":286},[970],{"type":32,"tag":113,"props":971,"children":972},{"style":126},[973],{"type":37,"value":974},"You write for Roibase.\n",{"type":32,"tag":113,"props":976,"children":977},{"class":115,"line":304},[978],{"type":32,"tag":113,"props":979,"children":980},{"style":126},[981],{"type":37,"value":982},"...\n",{"type":32,"tag":33,"props":984,"children":985},{},[986],{"type":37,"value":987},"Git commit message: \"prompt v3: add category-specific guidance, expand banned word list.\" When CI\u002FCD sees this commit, it automatically runs the Promptfoo test suite. If tests pass, deploy to staging, run 24-hour A\u002FB test, then promote to production.",{"type":32,"tag":33,"props":989,"children":990},{},[991,993,999],{"type":37,"value":992},"Versioning makes rollback fast: production issue? ",{"type":32,"tag":81,"props":994,"children":996},{"className":995},[],[997],{"type":37,"value":998},"git revert",{"type":37,"value":1000},", old prompt is live in 5 minutes.",{"type":32,"tag":40,"props":1002,"children":1004},{"id":1003},"cost-optimization-token-audit",[1005],{"type":37,"value":1006},"Cost Optimization: Token Audit",{"type":32,"tag":33,"props":1008,"children":1009},{},[1010],{"type":37,"value":1011},"LLM application costs typically split into input tokens + output tokens. Claude Sonnet 3.5 API pricing: $3\u002F1M input tokens, $15\u002F1M output tokens (2026 rates). A 1500-word blog draft is ~2000 output tokens; system prompt + user prompt is ~1200 input tokens—about $0.042 per article.",{"type":32,"tag":33,"props":1013,"children":1014},{},[1015],{"type":37,"value":1016},"Generate 1000 articles\u002Fmonth? That's $42. Optimize the prompt to cut output tokens 10%? Save $6.30\u002Fmonth—$75.60\u002Fyear. Looks small until you scale: 10,000 articles\u002Fmonth gives $756\u002Fyear.",{"type":32,"tag":33,"props":1018,"children":1019},{},[1020],{"type":37,"value":1021},"Add cost assertions to your Promptfoo eval suite:",{"type":32,"tag":103,"props":1023,"children":1025},{"code":1024,"language":106,"meta":16,"className":107,"style":16},"assert:\n  - type: cost\n    threshold: 0.045\n",[1026],{"type":32,"tag":81,"props":1027,"children":1028},{"__ignoreMap":16},[1029,1041,1060],{"type":32,"tag":113,"props":1030,"children":1031},{"class":115,"line":116},[1032,1037],{"type":32,"tag":113,"props":1033,"children":1034},{"style":120},[1035],{"type":37,"value":1036},"assert",{"type":32,"tag":113,"props":1038,"children":1039},{"style":126},[1040],{"type":37,"value":129},{"type":32,"tag":113,"props":1042,"children":1043},{"class":115,"line":132},[1044,1048,1052,1056],{"type":32,"tag":113,"props":1045,"children":1046},{"style":126},[1047],{"type":37,"value":138},{"type":32,"tag":113,"props":1049,"children":1050},{"style":120},[1051],{"type":37,"value":328},{"type":32,"tag":113,"props":1053,"children":1054},{"style":126},[1055],{"type":37,"value":148},{"type":32,"tag":113,"props":1057,"children":1058},{"style":151},[1059],{"type":37,"value":414},{"type":32,"tag":113,"props":1061,"children":1062},{"class":115,"line":157},[1063,1068,1072],{"type":32,"tag":113,"props":1064,"children":1065},{"style":120},[1066],{"type":37,"value":1067},"    threshold",{"type":32,"tag":113,"props":1069,"children":1070},{"style":126},[1071],{"type":37,"value":148},{"type":32,"tag":113,"props":1073,"children":1074},{"style":430},[1075],{"type":37,"value":1076},"0.045\n",{"type":32,"tag":33,"props":1078,"children":1079},{},[1080],{"type":37,"value":1081},"If a prompt change pushes cost above $0.045, the test fails. Calibrate this threshold against business metrics (editor approval rate, conversion).",{"type":32,"tag":33,"props":1083,"children":1084},{},[1085,1087,1094],{"type":37,"value":1086},"For token audits, check LangSmith traces: which prompt component consumes the most tokens? For example, your system prompt's \"BANNED WORDS\" section is 300 tokens—do you really need it on every call, or can you inject it contextually via retrieval? In Roibase's ",{"type":32,"tag":473,"props":1088,"children":1091},{"href":1089,"rel":1090},"https:\u002F\u002Fwww.roibase.com.tr\u002Fen\u002Ffirstparty",[477],[1092],{"type":37,"value":1093},"First-Party Data & Measurement Architecture",{"type":37,"value":1095}," work, we use context injection: modularize the prompt, inject only necessary sections based on user segment—15-20% token savings.",{"type":32,"tag":40,"props":1097,"children":1099},{"id":1098},"what-to-do-now",[1100],{"type":37,"value":1101},"What to Do Now",{"type":32,"tag":33,"props":1103,"children":1104},{},[1105],{"type":37,"value":1106},"If you're running LLMs in production, stop testing prompt changes manually. Start with Promptfoo: 10 test cases, 3 assertions (word count, cost, semantic keyword check). Integrate into CI\u002FCD—automatic tests on every PR. Next: add LangSmith or similar observability, monitor production traces. For A\u002FB testing, set up a feature flag system, pilot new prompt versions at 10% traffic. This discipline moves LLM operations from \"it works\" to \"measurable, optimizable.\" Prompts are code—test them like code, version them, deploy them.",{"type":32,"tag":1108,"props":1109,"children":1110},"style",{},[1111],{"type":37,"value":1112},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":16,"searchDepth":157,"depth":157,"links":1114},[1115,1116,1119,1122,1123,1124,1125,1126],{"id":42,"depth":132,"text":45},{"id":71,"depth":132,"text":74,"children":1117},[1118],{"id":442,"depth":157,"text":445},{"id":458,"depth":132,"text":461,"children":1120},[1121],{"id":485,"depth":157,"text":488},{"id":501,"depth":132,"text":504},{"id":600,"depth":132,"text":603},{"id":778,"depth":132,"text":781},{"id":1003,"depth":132,"text":1006},{"id":1098,"depth":132,"text":1101},"content:en:ai:prompt-versioning-ab-testing-llm-ops-discipline.md","content","en\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops-discipline.md","en\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops-discipline","md",1785967475483]