[{"data":1,"prerenderedAt":858},["ShallowReactive",2],{"article-alternates":3,"article-\u002Fde\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops":13},{"i18nKey":4,"paths":5},"ai-004-2026-07",{"de":6,"en":7,"es":8,"fr":9,"it":10,"ru":11,"tr":12},"\u002Fde\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops","\u002Fen\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops","\u002Fes\u002Fai\u002Fprompt-versionamiento-y-pruebas-ab","\u002Ffr\u002Fai\u002Fversionierung-und-ab-tests-von-prompts","\u002Fit\u002Fai\u002Fprompt-versiyonlama-ab-testi-llm-disiplini","\u002Fru\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops","\u002Ftr\u002Fai\u002Fprompt-versiyonlama-ve-a-b-testi-llm-operasyonun-disiplini",{"_path":6,"_dir":14,"_draft":15,"_partial":15,"_locale":16,"title":17,"description":18,"publishedAt":19,"modifiedAt":19,"category":14,"i18nKey":4,"tags":20,"readingTime":26,"author":27,"body":28,"_type":852,"_id":853,"_source":854,"_file":855,"_stem":856,"_extension":857},"ai",false,"","Prompt-Versionierung und A\u002FB-Tests: Engineering-Disziplin für LLM-Operationen","Prompt-Änderungen in Production-LLM-Systemen erfordern Testen, Versionierung und Rollback-Mechanismen. Mit Promptfoo und LangSmith strukturiert evaluieren.","2026-07-11",[21,22,23,24,25],"llm-ops","prompt-engineering","evaluation","ab-testing","langsmith",9,"Roibase",{"type":29,"children":30,"toc":840},"root",[31,39,46,68,73,78,84,89,94,115,122,127,481,508,521,527,540,545,550,566,572,577,582,679,684,703,709,714,719,732,738,743,748,783,788,794,799,804,818,824,829,834],{"type":32,"tag":33,"props":34,"children":35},"element","p",{},[36],{"type":37,"value":38},"text","Jedes Team, das LLMs in Production einsetzt, stellt sich die gleiche Frage: Ist die Prompt-Änderung wirklich besser geworden? „Wirkt irgendwie kohärenter\" ist nicht ausreichend. Wenn eine Marketing-Abteilung täglich 500 Blog-Überschriften aus der Claude API generiert, kann der Unterschied zwischen „sei kreativ\" und „sei verkäuferisch\" tausende Dollar Conversion-Differenz bedeuten. Eine solche Änderung ohne Messung in Production zu gehen, ist kein Engineering – es ist Glücksspiel. Prompt-Versionierung und Evaluation Pipelines verwandeln LLM-Operationen von spekulativer Experimenterei in datengesteuerte Disziplin.",{"type":32,"tag":40,"props":41,"children":43},"h2",{"id":42},"warum-prompt-änderungen-anders-sind-als-code-änderungen",[44],{"type":37,"value":45},"Warum Prompt-Änderungen anders sind als Code-Änderungen",{"type":32,"tag":33,"props":47,"children":48},{},[49,51,58,60,66],{"type":37,"value":50},"Wenn du im klassischen Code ",{"type":32,"tag":52,"props":53,"children":55},"code",{"className":54},[],[56],{"type":37,"value":57},"if (x > 5)",{"type":37,"value":59}," in ",{"type":32,"tag":52,"props":61,"children":63},{"className":62},[],[64],{"type":37,"value":65},"if (x >= 5)",{"type":37,"value":67}," änderst, bricht ein Unit-Test, das Verhalten ist deterministisch. Eine Prompt-Änderung ist stochastisch: Der gleiche Input liefert unterschiedliche Outputs, es gibt keine Regressionstests, „schlechter als vorher\" ist vage. GPT-4 „sei kurz\" schreiben bedeutet eines Tages 50 Wörter, nächsten Tag 120 Wörter. Diese Unsicherheit macht eine „produktiv nehmen \u002F nicht nehmen\"-Entscheidung ohne Metriken unmöglich.",{"type":32,"tag":33,"props":69,"children":70},{},[71],{"type":37,"value":72},"Der zweite Unterschied ist die Kontrollpunkt-Anzahl. Eine Code-Änderung durchläuft Unit-Tests, Integration-Tests, Staging in der Deployment Pipeline. Eine Prompt-Änderung geht bei den meisten Teams mit „habe es in Claude UI getestet, sieht gut aus\" direkt in Production. Ergebnis: Zwei Wochen später kommt die Beschwerde „neue Prompts verwenden zu viel Jargon\", und du musst ein Git-Commit durchsuchen, um zur alten Version zurückzugehen.",{"type":32,"tag":33,"props":74,"children":75},{},[76],{"type":37,"value":77},"Der dritte Unterschied ist die verzögerte Auswirkung. Neuer Prompt mit generiertem Content führt zwei Monate später zu SEO-Rückgang, Chatbot-Output erodiert langsam Kundenzufriedenheit. Ein Code-Bug löst sofort Sentry-Alarm aus, Prompt-Regression sammelt sich stillschweigend an.",{"type":32,"tag":40,"props":79,"children":81},{"id":80},"anatomy-einer-evaluation-pipeline",[82],{"type":37,"value":83},"Anatomy einer Evaluation Pipeline",{"type":32,"tag":33,"props":85,"children":86},{},[87],{"type":37,"value":88},"Eine Evaluation Pipeline besteht aus drei Schichten: Dataset, Judge, Report. Das Dataset sind aus Production gesampelte Eingaben – keine generischen „Test Prompts\", sondern echte Benutzerabfragen. Beispiel: Bei einem Customer-Support-Chatbot besteht das Dataset aus 100 Ticket-Input-Output-Paaren. Du wirst diese manuell labeln: „Halluzination vorhanden\", „Ton falsch\", „faktisch korrekt\". Das Dataset ist keine statische Fixture, sondern wird wöchentlich aus Production aktualisiert.",{"type":32,"tag":33,"props":90,"children":91},{},[92],{"type":37,"value":93},"Der Judge ist der Mechanismus, der Outputs scored. Einfacher Weg: Regex\u002FKeyword-Matching („Output muss ‚entschuldigen' enthalten\"). Mittlerer Weg: Ein anderes LLM als Judge (GPT-4-turbo: „Ist dieser Output hilfreich? Score 1–5\"). Fortgeschrittener Weg: Custom Classifier trainieren (BERT-basierte binäre Klassifizierung: Halluzination ja\u002Fnein). Der Judge selbst muss versioniert werden – wenn sich der Judge ändert, ändern sich die Scores, Trends brechen.",{"type":32,"tag":33,"props":95,"children":96},{},[97,99,105,107,113],{"type":37,"value":98},"Die Report-Schicht verwandelt A\u002FB-Tests in Entscheidungen. Du hast zwei Prompt-Versionen: ",{"type":32,"tag":52,"props":100,"children":102},{"className":101},[],[103],{"type":37,"value":104},"baseline",{"type":37,"value":106}," (Production) und ",{"type":32,"tag":52,"props":108,"children":110},{"className":109},[],[111],{"type":37,"value":112},"candidate",{"type":37,"value":114}," (Test). Beide auf den gleichen Dataset laufen lassen, Judge-Scores sammeln. Report: „Candidate 12% höhere faktische Genauigkeit, aber 8% höhere Latenz\". Entscheidung: Ist die Latenz-Steigerung akzeptabel? Das beantwortest du mit einer Metrik (z.B. Durchsatz-SLA überschritten?).",{"type":32,"tag":116,"props":117,"children":119},"h3",{"id":118},"setup-mit-promptfoo",[120],{"type":37,"value":121},"Setup mit Promptfoo",{"type":32,"tag":33,"props":123,"children":124},{},[125],{"type":37,"value":126},"Promptfoo ist ein Open-Source CLI-Tool, mit dem du config-basierte Evaluationen durchführst:",{"type":32,"tag":128,"props":129,"children":133},"pre",{"className":130,"code":131,"language":132,"meta":16,"style":16},"language-yaml shiki shiki-themes github-dark","# promptfoo.yaml\nprompts:\n  - file:\u002F\u002Fprompts\u002Fv1-baseline.txt\n  - file:\u002F\u002Fprompts\u002Fv2-candidate.txt\n\nproviders:\n  - openai:gpt-4\n  - anthropic:claude-3-opus-20240229\n\ntests:\n  - vars:\n      user_query: \"Wann kommt meine Bestellung an?\"\n    assert:\n      - type: contains\n        value: \"Verfolgung\"\n      - type: llm-rubric\n        value: \"Zeigt die Antwort Empathie für den Kunden?\"\n\n  - vars:\n      user_query: \"Wie führe ich eine Rücksendung durch?\"\n    assert:\n      - type: not-contains\n        value: \"Das können wir leider nicht\"\n","yaml",[134],{"type":32,"tag":52,"props":135,"children":136},{"__ignoreMap":16},[137,149,165,180,193,203,216,229,242,249,262,279,298,311,334,352,373,390,398,414,431,443,464],{"type":32,"tag":138,"props":139,"children":142},"span",{"class":140,"line":141},"line",1,[143],{"type":32,"tag":138,"props":144,"children":146},{"style":145},"--shiki-default:#6A737D",[147],{"type":37,"value":148},"# promptfoo.yaml\n",{"type":32,"tag":138,"props":150,"children":152},{"class":140,"line":151},2,[153,159],{"type":32,"tag":138,"props":154,"children":156},{"style":155},"--shiki-default:#85E89D",[157],{"type":37,"value":158},"prompts",{"type":32,"tag":138,"props":160,"children":162},{"style":161},"--shiki-default:#E1E4E8",[163],{"type":37,"value":164},":\n",{"type":32,"tag":138,"props":166,"children":168},{"class":140,"line":167},3,[169,174],{"type":32,"tag":138,"props":170,"children":171},{"style":161},[172],{"type":37,"value":173},"  - ",{"type":32,"tag":138,"props":175,"children":177},{"style":176},"--shiki-default:#9ECBFF",[178],{"type":37,"value":179},"file:\u002F\u002Fprompts\u002Fv1-baseline.txt\n",{"type":32,"tag":138,"props":181,"children":183},{"class":140,"line":182},4,[184,188],{"type":32,"tag":138,"props":185,"children":186},{"style":161},[187],{"type":37,"value":173},{"type":32,"tag":138,"props":189,"children":190},{"style":176},[191],{"type":37,"value":192},"file:\u002F\u002Fprompts\u002Fv2-candidate.txt\n",{"type":32,"tag":138,"props":194,"children":196},{"class":140,"line":195},5,[197],{"type":32,"tag":138,"props":198,"children":200},{"emptyLinePlaceholder":199},true,[201],{"type":37,"value":202},"\n",{"type":32,"tag":138,"props":204,"children":206},{"class":140,"line":205},6,[207,212],{"type":32,"tag":138,"props":208,"children":209},{"style":155},[210],{"type":37,"value":211},"providers",{"type":32,"tag":138,"props":213,"children":214},{"style":161},[215],{"type":37,"value":164},{"type":32,"tag":138,"props":217,"children":219},{"class":140,"line":218},7,[220,224],{"type":32,"tag":138,"props":221,"children":222},{"style":161},[223],{"type":37,"value":173},{"type":32,"tag":138,"props":225,"children":226},{"style":176},[227],{"type":37,"value":228},"openai:gpt-4\n",{"type":32,"tag":138,"props":230,"children":232},{"class":140,"line":231},8,[233,237],{"type":32,"tag":138,"props":234,"children":235},{"style":161},[236],{"type":37,"value":173},{"type":32,"tag":138,"props":238,"children":239},{"style":176},[240],{"type":37,"value":241},"anthropic:claude-3-opus-20240229\n",{"type":32,"tag":138,"props":243,"children":244},{"class":140,"line":26},[245],{"type":32,"tag":138,"props":246,"children":247},{"emptyLinePlaceholder":199},[248],{"type":37,"value":202},{"type":32,"tag":138,"props":250,"children":252},{"class":140,"line":251},10,[253,258],{"type":32,"tag":138,"props":254,"children":255},{"style":155},[256],{"type":37,"value":257},"tests",{"type":32,"tag":138,"props":259,"children":260},{"style":161},[261],{"type":37,"value":164},{"type":32,"tag":138,"props":263,"children":265},{"class":140,"line":264},11,[266,270,275],{"type":32,"tag":138,"props":267,"children":268},{"style":161},[269],{"type":37,"value":173},{"type":32,"tag":138,"props":271,"children":272},{"style":155},[273],{"type":37,"value":274},"vars",{"type":32,"tag":138,"props":276,"children":277},{"style":161},[278],{"type":37,"value":164},{"type":32,"tag":138,"props":280,"children":282},{"class":140,"line":281},12,[283,288,293],{"type":32,"tag":138,"props":284,"children":285},{"style":155},[286],{"type":37,"value":287},"      user_query",{"type":32,"tag":138,"props":289,"children":290},{"style":161},[291],{"type":37,"value":292},": ",{"type":32,"tag":138,"props":294,"children":295},{"style":176},[296],{"type":37,"value":297},"\"Wann kommt meine Bestellung an?\"\n",{"type":32,"tag":138,"props":299,"children":301},{"class":140,"line":300},13,[302,307],{"type":32,"tag":138,"props":303,"children":304},{"style":155},[305],{"type":37,"value":306},"    assert",{"type":32,"tag":138,"props":308,"children":309},{"style":161},[310],{"type":37,"value":164},{"type":32,"tag":138,"props":312,"children":314},{"class":140,"line":313},14,[315,320,325,329],{"type":32,"tag":138,"props":316,"children":317},{"style":161},[318],{"type":37,"value":319},"      - ",{"type":32,"tag":138,"props":321,"children":322},{"style":155},[323],{"type":37,"value":324},"type",{"type":32,"tag":138,"props":326,"children":327},{"style":161},[328],{"type":37,"value":292},{"type":32,"tag":138,"props":330,"children":331},{"style":176},[332],{"type":37,"value":333},"contains\n",{"type":32,"tag":138,"props":335,"children":337},{"class":140,"line":336},15,[338,343,347],{"type":32,"tag":138,"props":339,"children":340},{"style":155},[341],{"type":37,"value":342},"        value",{"type":32,"tag":138,"props":344,"children":345},{"style":161},[346],{"type":37,"value":292},{"type":32,"tag":138,"props":348,"children":349},{"style":176},[350],{"type":37,"value":351},"\"Verfolgung\"\n",{"type":32,"tag":138,"props":353,"children":355},{"class":140,"line":354},16,[356,360,364,368],{"type":32,"tag":138,"props":357,"children":358},{"style":161},[359],{"type":37,"value":319},{"type":32,"tag":138,"props":361,"children":362},{"style":155},[363],{"type":37,"value":324},{"type":32,"tag":138,"props":365,"children":366},{"style":161},[367],{"type":37,"value":292},{"type":32,"tag":138,"props":369,"children":370},{"style":176},[371],{"type":37,"value":372},"llm-rubric\n",{"type":32,"tag":138,"props":374,"children":376},{"class":140,"line":375},17,[377,381,385],{"type":32,"tag":138,"props":378,"children":379},{"style":155},[380],{"type":37,"value":342},{"type":32,"tag":138,"props":382,"children":383},{"style":161},[384],{"type":37,"value":292},{"type":32,"tag":138,"props":386,"children":387},{"style":176},[388],{"type":37,"value":389},"\"Zeigt die Antwort Empathie für den Kunden?\"\n",{"type":32,"tag":138,"props":391,"children":393},{"class":140,"line":392},18,[394],{"type":32,"tag":138,"props":395,"children":396},{"emptyLinePlaceholder":199},[397],{"type":37,"value":202},{"type":32,"tag":138,"props":399,"children":401},{"class":140,"line":400},19,[402,406,410],{"type":32,"tag":138,"props":403,"children":404},{"style":161},[405],{"type":37,"value":173},{"type":32,"tag":138,"props":407,"children":408},{"style":155},[409],{"type":37,"value":274},{"type":32,"tag":138,"props":411,"children":412},{"style":161},[413],{"type":37,"value":164},{"type":32,"tag":138,"props":415,"children":417},{"class":140,"line":416},20,[418,422,426],{"type":32,"tag":138,"props":419,"children":420},{"style":155},[421],{"type":37,"value":287},{"type":32,"tag":138,"props":423,"children":424},{"style":161},[425],{"type":37,"value":292},{"type":32,"tag":138,"props":427,"children":428},{"style":176},[429],{"type":37,"value":430},"\"Wie führe ich eine Rücksendung durch?\"\n",{"type":32,"tag":138,"props":432,"children":434},{"class":140,"line":433},21,[435,439],{"type":32,"tag":138,"props":436,"children":437},{"style":155},[438],{"type":37,"value":306},{"type":32,"tag":138,"props":440,"children":441},{"style":161},[442],{"type":37,"value":164},{"type":32,"tag":138,"props":444,"children":446},{"class":140,"line":445},22,[447,451,455,459],{"type":32,"tag":138,"props":448,"children":449},{"style":161},[450],{"type":37,"value":319},{"type":32,"tag":138,"props":452,"children":453},{"style":155},[454],{"type":37,"value":324},{"type":32,"tag":138,"props":456,"children":457},{"style":161},[458],{"type":37,"value":292},{"type":32,"tag":138,"props":460,"children":461},{"style":176},[462],{"type":37,"value":463},"not-contains\n",{"type":32,"tag":138,"props":465,"children":467},{"class":140,"line":466},23,[468,472,476],{"type":32,"tag":138,"props":469,"children":470},{"style":155},[471],{"type":37,"value":342},{"type":32,"tag":138,"props":473,"children":474},{"style":161},[475],{"type":37,"value":292},{"type":32,"tag":138,"props":477,"children":478},{"style":176},[479],{"type":37,"value":480},"\"Das können wir leider nicht\"\n",{"type":32,"tag":33,"props":482,"children":483},{},[484,490,492,498,500,506],{"type":32,"tag":52,"props":485,"children":487},{"className":486},[],[488],{"type":37,"value":489},"promptfoo eval",{"type":37,"value":491}," führt jeden Prompt × jeden Test-Fall aus, kontrolliert Assertions, gibt eine Tabelle aus: Welcher Prompt schlägt bei welchem Test fehl. Hier evaluiert ",{"type":32,"tag":52,"props":493,"children":495},{"className":494},[],[496],{"type":37,"value":497},"llm-rubric",{"type":37,"value":499}," ein anderes LLM (Promptfoo ruft dies automatisch auf). Für A\u002FB-Differenzen öffnet ",{"type":32,"tag":52,"props":501,"children":503},{"className":502},[],[504],{"type":37,"value":505},"promptfoo view",{"type":37,"value":507}," eine Web-UI, wo du beide Prompts nebeneinander vergleichst.",{"type":32,"tag":33,"props":509,"children":510},{},[511,513,519],{"type":37,"value":512},"Promptfoo's Vorteil ist die Geschwindigkeit: 50 Test-Cases in 2 Minuten, in CI\u002FCD integriert (",{"type":32,"tag":52,"props":514,"children":516},{"className":515},[],[517],{"type":37,"value":518},"promptfoo eval --assertions",{"type":37,"value":520}," gibt Exit-Code 1 bei Fehler). Nachteil: Nicht mit Production Traces integriert, musst manuell exportieren.",{"type":32,"tag":40,"props":522,"children":524},{"id":523},"production-trace-basierte-evaluation-mit-langsmith",[525],{"type":37,"value":526},"Production Trace-basierte Evaluation mit LangSmith",{"type":32,"tag":33,"props":528,"children":529},{},[530,532,538],{"type":37,"value":531},"LangSmith (vom LangChain-Team) loggt automatisch Production-LLM-Runs, dann führst du Evaluationen über diese Logs aus. Ablauf: Läuft deine App mit LangChain SDK, geht jeder LLM-Call an LangSmith (Input, Output, Latenz, Kosten). In der LangSmith UI filterst du „Runs mit Tag customer_support in den letzten 7 Tagen\", wählst 200 Beispiele, sagst „create dataset\". Dieses Dataset ist jetzt versioniert – gespeichert als ",{"type":32,"tag":52,"props":533,"children":535},{"className":534},[],[536],{"type":37,"value":537},"2026-07-01-support-sample",{"type":37,"value":539},".",{"type":32,"tag":33,"props":541,"children":542},{},[543],{"type":37,"value":544},"Jetzt möchtest du einen neuen Prompt testen. In LangSmith's Playground änderst du den Prompt, sagst „Run on dataset\", es läuft alle 200 Beispiele mit diesem neuen Prompt. Ergebnisse nebeneinander: Alter Output vs. neuer Output. Du oder ein Judge-Modell annotiert: Thumbs-up\u002Fdown, oder Custom-Score (1–5). LangSmith aggregiert diese Scores – z.B. „neuer Prompt Thumbs-up Rate 78%, alter 65%\".",{"type":32,"tag":33,"props":546,"children":547},{},[548],{"type":37,"value":549},"LangSmith's Stärke ist der Trace-Kontext. Nicht nur Prompt, sondern auch Retrieval-Schritte sind im Trace sichtbar. Beispiel: Du änderst einen Prompt in einem RAG-System, aber eigentlich war das Retrieval das Problem – falsche Dokumente kamen herein. Im Trace siehst du: „Neuer Prompt antwortet besser, weil ich die Retrieval-Query geändert habe\". Diesen Einblick hat Promptfoo nicht (schaut nur auf den Final Output).",{"type":32,"tag":33,"props":551,"children":552},{},[553,555,564],{"type":37,"value":554},"LangSmith's Trade-off ist Vendor Lock-in: Du musst das LangChain-Ökosystem verwenden. Mit reinem Anthropic oder OpenAI SDK brauchst du manuelles Tracing (jeden Call an LangSmith API senden). Alternative: Ansatz wie ",{"type":32,"tag":556,"props":557,"children":561},"a",{"href":558,"rel":559},"https:\u002F\u002Fwww.roibase.com.tr\u002Fde\u002Ffirstparty",[560],"nofollow",[562],{"type":37,"value":563},"First-Party Data & Measurement Architecture",{"type":37,"value":565}," – LLM Traces zu deinem Data Warehouse, Evaluation aus BigQuery.",{"type":32,"tag":40,"props":567,"children":569},{"id":568},"eval-metriken-nach-use-case-auswählen",[570],{"type":37,"value":571},"Eval-Metriken nach Use Case auswählen",{"type":32,"tag":33,"props":573,"children":574},{},[575],{"type":37,"value":576},"Die Metrik-Auswahl hängt vom Use Case ab. Bei Content Generation: „Liegt Output-Keyword-Dichte im Ziel?\", „Passt Ton zu Brand Guidelines?\", „Faktische Halluzinationen vorhanden?\". Bei Chatbots: „Query gelöst?\", „Response-Latenz im SLA?\", „Fragt Nutzer Follow-up?\". Für jede Metrik musst du einen Judge definieren.",{"type":32,"tag":33,"props":578,"children":579},{},[580],{"type":37,"value":581},"Eine solide Eval Suite hat mindestens 3 Metrik-Schichten:",{"type":32,"tag":583,"props":584,"children":585},"table",{},[586,610],{"type":32,"tag":587,"props":588,"children":589},"thead",{},[590],{"type":32,"tag":591,"props":592,"children":593},"tr",{},[594,600,605],{"type":32,"tag":595,"props":596,"children":597},"th",{},[598],{"type":37,"value":599},"Schicht",{"type":32,"tag":595,"props":601,"children":602},{},[603],{"type":37,"value":604},"Beispiel-Metriken",{"type":32,"tag":595,"props":606,"children":607},{},[608],{"type":37,"value":609},"Judge-Typ",{"type":32,"tag":611,"props":612,"children":613},"tbody",{},[614,637,658],{"type":32,"tag":591,"props":615,"children":616},{},[617,627,632],{"type":32,"tag":618,"props":619,"children":620},"td",{},[621],{"type":32,"tag":622,"props":623,"children":624},"strong",{},[625],{"type":37,"value":626},"Functional",{"type":32,"tag":618,"props":628,"children":629},{},[630],{"type":37,"value":631},"Output-Format korrekt (JSON parsebar?), Mandatory Keywords vorhanden?",{"type":32,"tag":618,"props":633,"children":634},{},[635],{"type":37,"value":636},"Regex\u002Fdeterministisch",{"type":32,"tag":591,"props":638,"children":639},{},[640,648,653],{"type":32,"tag":618,"props":641,"children":642},{},[643],{"type":32,"tag":622,"props":644,"children":645},{},[646],{"type":37,"value":647},"Quality",{"type":32,"tag":618,"props":649,"children":650},{},[651],{"type":37,"value":652},"Ton-Eignung, faktische Genauigkeit, Halluzinationen",{"type":32,"tag":618,"props":654,"children":655},{},[656],{"type":37,"value":657},"LLM-as-Judge (GPT-4 scoret)",{"type":32,"tag":591,"props":659,"children":660},{},[661,669,674],{"type":32,"tag":618,"props":662,"children":663},{},[664],{"type":32,"tag":622,"props":665,"children":666},{},[667],{"type":37,"value":668},"Business",{"type":32,"tag":618,"props":670,"children":671},{},[672],{"type":37,"value":673},"Conversion-Vorhersage, Engagement-Estimate",{"type":32,"tag":618,"props":675,"children":676},{},[677],{"type":37,"value":678},"Custom-Modell (XGBoost: wird dieser Output Umsatz generieren?)",{"type":32,"tag":33,"props":680,"children":681},{},[682],{"type":37,"value":683},"Functional-Metriken sind billig, schnell, Regression-Guard. Quality-Metriken sind teuer (Judge-LLM-Calls), aber nächster Proxy zu menschlicher Bewertung. Business-Metriken sind wertvollst, aber schwer zu trainieren – musst Conversion-Daten mit Outputs abgleichen.",{"type":32,"tag":33,"props":685,"children":686},{},[687,689,694,696,701],{"type":37,"value":688},"Sowohl Promptfoo als auch LangSmith unterstützen LLM-as-Judge. Beispiel: Promptfoo's ",{"type":32,"tag":52,"props":690,"children":692},{"className":691},[],[693],{"type":37,"value":497},{"type":37,"value":695}," Assertion sendet GPT-4 diesen Prompt: „Bewerte den folgenden Output ",{"type":32,"tag":138,"props":697,"children":698},{},[699],{"type":37,"value":700},"Kriterium",{"type":37,"value":702}," auf Skala 1–10, antworte nur mit Zahl\". In LangSmith definierst du „Evaluators\" – z.B. „frage mit Anthropic Claude Haiku: ‚Hat Antwort Empathie?', konvertiere Antwort zu bool\".",{"type":32,"tag":40,"props":704,"children":706},{"id":705},"ab-tests-in-production-verschieben",[707],{"type":37,"value":708},"A\u002FB-Tests in Production verschieben",{"type":32,"tag":33,"props":710,"children":711},{},[712],{"type":37,"value":713},"Nach bestandener Offline-Eval kommt Production A\u002FB-Test. Zwei Strategien: Shadow Deployment und Gradual Rollout. Beim Shadow Deployment nimmt der neue Prompt Production-Traffic, Output wird aber dem Nutzer nicht gezeigt – nur geloggt, mit Baseline verglichen. Eine Woche Shadow, wenn Metriken keinen signifikanten Unterschied zeigen, ist der Prompt tot.",{"type":32,"tag":33,"props":715,"children":716},{},[717],{"type":37,"value":718},"Gradual Rollout: 5% Traffic neuer Prompt, 95% Baseline. Zwei Wochen kochen lassen, Business-Metriken (z.B. Session-Resolution-Rate im Chatbot) überwachen. Kein Problem bei 5%? Auf 25% erhöhen, dann 50%, dann 100%. Jede Phase – wenn KPIs sinken, Rollback.",{"type":32,"tag":33,"props":720,"children":721},{},[722,724,730],{"type":37,"value":723},"Rollback-Mechanismus ist essentiell. Git-Commit-Versionierung des Prompts reicht nicht – du musst auch Production-Deployment versionieren. Beispiel: Wenn dein n8n-Workflow den Prompt von GitHub mit Raw URL lädt, muss die URL einen Commit-Hash haben: ",{"type":32,"tag":52,"props":725,"children":727},{"className":726},[],[728],{"type":37,"value":729},"github.com\u002F...\u002Fprompt.md?ref=abc123",{"type":37,"value":731},". Rollback: Hash auf alten Commit, Workflow redeploy (30 Sekunden). Noch sofistikierter: Feature-Flag-System wie LaunchDarkly – ändere Prompt-Version zur Laufzeit, kein Deployment.",{"type":32,"tag":40,"props":733,"children":735},{"id":734},"eval-budget-und-automation",[736],{"type":37,"value":737},"Eval-Budget und Automation",{"type":32,"tag":33,"props":739,"children":740},{},[741],{"type":37,"value":742},"Das Eval-Budget eines Production-LLM-Systems sollte 10–20% der LLM-API-Kosten sein. Wenn du monatlich 5.000$ Claude-Calls machst, allokiere 500–1.000$ für Eval. Budget-Posten: Dataset-Refresh (wöchentlich 100 neue Beispiele), Judge-LLM-Calls (2 Calls pro Beispiel = 200 Beispiele × 2 × 0.01$ = 4$), Human Labeling (kritische Edge Cases).",{"type":32,"tag":33,"props":744,"children":745},{},[746],{"type":37,"value":747},"Automation so strukturieren:",{"type":32,"tag":749,"props":750,"children":751},"ol",{},[752,763,773],{"type":32,"tag":753,"props":754,"children":755},"li",{},[756,761],{"type":32,"tag":622,"props":757,"children":758},{},[759],{"type":37,"value":760},"CI Eval:",{"type":37,"value":762}," Jeder Prompt-Commit: Promptfoo gegen Baseline, Functional-Metriken fail → PR kann nicht gemerged werden.",{"type":32,"tag":753,"props":764,"children":765},{},[766,771],{"type":32,"tag":622,"props":767,"children":768},{},[769],{"type":37,"value":770},"Nightly Eval:",{"type":37,"value":772}," Jede Nacht: Neuer Dataset-Sample aus Production, Candidate Prompts kosten, Report an Slack.",{"type":32,"tag":753,"props":774,"children":775},{},[776,781],{"type":32,"tag":622,"props":777,"children":778},{},[779],{"type":37,"value":780},"Weekly Review:",{"type":37,"value":782}," Montagmorgen: Ekteam schaut LangSmith Dashboard, Quality-Metric Trends, neue Experiment-Entscheidung.",{"type":32,"tag":33,"props":784,"children":785},{},[786],{"type":37,"value":787},"Ohne Automation stirbt Eval beim Geburt. „Werden manuell testen\" bedeutet: Niemand macht's, in zwei Monaten ist Production-Prompt Chaos.",{"type":32,"tag":40,"props":789,"children":791},{"id":790},"gegenargument-eval-erfasst-echte-nutzer-nicht",[792],{"type":37,"value":793},"Gegenargument: Eval erfasst echte Nutzer nicht",{"type":32,"tag":33,"props":795,"children":796},{},[797],{"type":37,"value":798},"Eval's Limitation: So gut der Judge auch ist, er kann echtes Nutzer-Verhalten nicht vorhersagen. LLM-as-Judge sagt „dieser Ton ist gut\", Nutzer springt aber ab. Lösung: Eval mit A\u002FB-Tests kombinieren, Evaluation nicht als „go\u002Fno-go Gate\" sondern als „Risiko-Filter\" verwenden. Eval bestanden = verdient 5% Production-Traffic, aber finale Entscheidung kommt von KPIs.",{"type":32,"tag":33,"props":800,"children":801},{},[802],{"type":37,"value":803},"Zweites Gegenargument: Kosten. Eval-Pipeline aufzubauen dauert (2–3 Wochen), Judge-LLM-Calls addieren sich. Wenn Prompt-Änderungen monatlich einmal passieren, ist Pipeline-Overhead nicht gerechtfertigt. Antwort: Wenn Prompt-Änderungen nur monatlich einmal passieren, überdenk deine LLM-Strategie – deine Production-Iterationsgeschwindigkeit ist langsam, das ist kein Growth Engineering.",{"type":32,"tag":33,"props":805,"children":806},{},[807,809,816],{"type":37,"value":808},"Endfrage: Ist es risikoreicher, ohne Eval zu gehen, oder ist Eval-Overhead größer? Wenn LLM-Output Revenue-Critical ist (z.B. Product Recommendation, Customer Support, ",{"type":32,"tag":556,"props":810,"children":813},{"href":811,"rel":812},"https:\u002F\u002Fwww.roibase.com.tr\u002Fde\u002Fgeo",[560],[814],{"type":37,"value":815},"Generative Engine Optimization",{"type":37,"value":817}," Citation), ist die Antwort klar: Ohne Eval fährst du blind. Wenn Output sekundär ist (z.B. interne Tool-Zusammenfassung), ist manuelles QA ausreichend.",{"type":32,"tag":40,"props":819,"children":821},{"id":820},"womit-fängst-du-diese-woche-an",[822],{"type":37,"value":823},"Womit fängst du diese Woche an",{"type":32,"tag":33,"props":825,"children":826},{},[827],{"type":37,"value":828},"Falls LLM in Production läuft, aber keine Eval-Pipeline existiert: Diese Woche Promptfoo setup, 20 Test Cases schreiben, in CI adden. Git-Commit-Message: „Add baseline prompt eval\". Nächsten Monat: 100 Beispiele aus Production als Dataset, LangSmith Trial starten (oder eigenes Trace-Log zu BigQuery), erste A\u002FB im Shadow-Mode. In drei Monaten: Eval-Automation live, jede Prompt-Änderung mit Metric-Diff gemerged, Rollback ein Befehl.",{"type":32,"tag":33,"props":830,"children":831},{},[832],{"type":37,"value":833},"Prompt-Versionierung und Eval heben LLM-Operationen aus der Spekulation in Engineering-Disziplin. Statt „neuer",{"type":32,"tag":835,"props":836,"children":837},"style",{},[838],{"type":37,"value":839},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":16,"searchDepth":167,"depth":167,"links":841},[842,843,846,847,848,849,850,851],{"id":42,"depth":151,"text":45},{"id":80,"depth":151,"text":83,"children":844},[845],{"id":118,"depth":167,"text":121},{"id":523,"depth":151,"text":526},{"id":568,"depth":151,"text":571},{"id":705,"depth":151,"text":708},{"id":734,"depth":151,"text":737},{"id":790,"depth":151,"text":793},{"id":820,"depth":151,"text":823},"markdown","content:de:ai:prompt-versioning-ab-testing-llm-ops.md","content","de\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops.md","de\u002Fai\u002Fprompt-versioning-ab-testing-llm-ops","md",1785103518029]