[{"data":1,"prerenderedAt":941},["ShallowReactive",2],{"article-alternates":3,"article-\u002Fde\u002Fai\u002Fprompt-versionierung-und-ab-test-llm-operations-disziplin":13},{"i18nKey":4,"paths":5},"ai-004-2026-06",{"de":6,"en":7,"es":8,"fr":9,"it":10,"ru":11,"tr":12},"\u002Fde\u002Fai\u002Fprompt-versionierung-und-ab-test-llm-operations-disziplin","\u002Fen\u002Fai\u002Fprompt-versioning-and-ab-testing-llm-operations-discipline","\u002Fes\u002Fai\u002Fversionamiento-de-prompts-y-pruebas-ab-disciplina-en-operaciones-llm","\u002Ffr\u002Fai\u002Fversionnage-des-prompts-et-tests-ab-discipline-des-operations-llm","\u002Fit\u002Fai\u002Fversionamento-prompt-e-test-ab-discipline-operativa-llm","\u002Fru\u002Fai\u002Fversiyonlama-t","\u002Ftr\u002Fai\u002Fprompt-versiyonlama-ve-a-b-testi-llm-operasyonun-disiplini",{"_path":6,"_dir":14,"_draft":15,"_partial":15,"_locale":16,"title":17,"description":18,"publishedAt":19,"modifiedAt":19,"category":14,"i18nKey":4,"tags":20,"readingTime":26,"author":27,"body":28,"_type":935,"_id":936,"_source":937,"_file":938,"_stem":939,"_extension":940},"ai",false,"","Prompt-Versionierung und A\u002FB-Testing: Die Disziplin des LLM-Betriebs","Mit Promptfoo, LangSmith und Evaluation Pipelines machen Sie Prompt-Änderungen messbar. So etablieren Sie Versionierung und A\u002FB-Testing im Production-LLM-Betrieb.","2026-06-22",[21,22,23,24,25],"prompt-engineering","llm-ops","evaluation","ab-testing","promptfoo",9,"Roibase",{"type":29,"children":30,"toc":926},"root",[31,39,46,51,56,61,67,72,293,298,303,316,412,417,423,428,441,479,484,570,575,581,586,591,738,743,748,754,759,772,838,851,865,871,876,881,889,894,899,905,910,915,920],{"type":32,"tag":33,"props":34,"children":35},"element","p",{},[36],{"type":37,"value":38},"text","LLMs in der Production zu betreiben bedeutet nicht mehr, nur ein paar API-Calls zu machen. Wenn Sie einen Prompt ändern, kann die Output-Qualität um 15 % sinken oder um 22 % steigen — aber wenn Sie das nicht bemerken, wird der Deployment zur Lotterie. Prompt-Versionierung und A\u002FB-Testing transportieren die Disziplin des klassischen Software-Deployments in den LLM-Betrieb. Dieser Artikel zeigt, wie Sie mit Evaluation-Frameworks wie Promptfoo und LangSmith Prompt-Änderungen messbar machen.",{"type":32,"tag":40,"props":41,"children":43},"h2",{"id":42},"eine-prompt-änderung-ist-kein-deployment",[44],{"type":37,"value":45},"Eine Prompt-Änderung ist kein Deployment",{"type":32,"tag":33,"props":47,"children":48},{},[49],{"type":37,"value":50},"Im klassischen Software Engineering durchlaufen Funktionsänderungen Unit-Tests, Integration-Tests und Canary-Deployments. Im LLM-Betrieb ändern die meisten Teams den Prompt in einer Textdatei, führen ein paar manuelle Tests durch und pushen in Production. Das Ergebnis: die User-Sentiment sinkt um 8 %, aber niemand kann die Ursache ermitteln.",{"type":32,"tag":33,"props":52,"children":53},{},[54],{"type":37,"value":55},"Das Problem: LLM-Output ist nicht deterministisch. Sie erhalten auf den gleichen Prompt unterschiedliche Antworten, was Single-Sample-Tests sinnlos macht. Ohne Versionierungssystem können Sie nicht auf die Frage „war der alte oder neue Prompt besser?\" antworten. Selbst ein Git Commit reicht nicht — Sie können die semantischen Unterschiede nicht aus der Commit-Message ableiten.",{"type":32,"tag":33,"props":57,"children":58},{},[59],{"type":37,"value":60},"Die Lösung: Versioning jeder Prompt-Änderung, Evaluierung desselben Eval-Sets vor und nach der Änderung, Vergleich der Metriken. Diese Disziplin erreicht zwei Ziele: Regression Detection (ob der neue Prompt alte Aufgaben zerstört) und Improvement Measurement (ob sich die Zielmetrik wirklich verbessert).",{"type":32,"tag":40,"props":62,"children":64},{"id":63},"wie-eine-evaluation-pipeline-funktioniert",[65],{"type":37,"value":66},"Wie eine Evaluation Pipeline funktioniert",{"type":32,"tag":33,"props":68,"children":69},{},[70],{"type":37,"value":71},"Eine Evaluation Pipeline besteht aus drei Komponenten: Eval Set, Eval-Metrik und Runner. Das Eval Set ist eine Liste von Inputs, die an das LLM gesendet werden, sowie erwartete Outputs (oder Output-Eigenschaften). Im JSON-Format sieht es so aus:",{"type":32,"tag":73,"props":74,"children":78},"pre",{"code":75,"language":76,"meta":16,"className":77,"style":16},"[\n  {\n    \"input\": \"Fasse den Revenue-Trend Q1 2025 zusammen\",\n    \"expected_topics\": [\"revenue\", \"growth\", \"quarter\"],\n    \"expected_sentiment\": \"neutral\"\n  },\n  {\n    \"input\": \"Erkläre, warum die Churn Rate gestiegen ist\",\n    \"expected_topics\": [\"churn\", \"retention\"],\n    \"expected_sentiment\": \"analytical\"\n  }\n]\n","json","language-json shiki shiki-themes github-dark",[79],{"type":32,"tag":80,"props":81,"children":82},"code",{"__ignoreMap":16},[83,95,104,130,173,191,200,208,229,258,275,284],{"type":32,"tag":84,"props":85,"children":88},"span",{"class":86,"line":87},"line",1,[89],{"type":32,"tag":84,"props":90,"children":92},{"style":91},"--shiki-default:#E1E4E8",[93],{"type":37,"value":94},"[\n",{"type":32,"tag":84,"props":96,"children":98},{"class":86,"line":97},2,[99],{"type":32,"tag":84,"props":100,"children":101},{"style":91},[102],{"type":37,"value":103},"  {\n",{"type":32,"tag":84,"props":105,"children":107},{"class":86,"line":106},3,[108,114,119,125],{"type":32,"tag":84,"props":109,"children":111},{"style":110},"--shiki-default:#79B8FF",[112],{"type":37,"value":113},"    \"input\"",{"type":32,"tag":84,"props":115,"children":116},{"style":91},[117],{"type":37,"value":118},": ",{"type":32,"tag":84,"props":120,"children":122},{"style":121},"--shiki-default:#9ECBFF",[123],{"type":37,"value":124},"\"Fasse den Revenue-Trend Q1 2025 zusammen\"",{"type":32,"tag":84,"props":126,"children":127},{"style":91},[128],{"type":37,"value":129},",\n",{"type":32,"tag":84,"props":131,"children":133},{"class":86,"line":132},4,[134,139,144,149,154,159,163,168],{"type":32,"tag":84,"props":135,"children":136},{"style":110},[137],{"type":37,"value":138},"    \"expected_topics\"",{"type":32,"tag":84,"props":140,"children":141},{"style":91},[142],{"type":37,"value":143},": [",{"type":32,"tag":84,"props":145,"children":146},{"style":121},[147],{"type":37,"value":148},"\"revenue\"",{"type":32,"tag":84,"props":150,"children":151},{"style":91},[152],{"type":37,"value":153},", ",{"type":32,"tag":84,"props":155,"children":156},{"style":121},[157],{"type":37,"value":158},"\"growth\"",{"type":32,"tag":84,"props":160,"children":161},{"style":91},[162],{"type":37,"value":153},{"type":32,"tag":84,"props":164,"children":165},{"style":121},[166],{"type":37,"value":167},"\"quarter\"",{"type":32,"tag":84,"props":169,"children":170},{"style":91},[171],{"type":37,"value":172},"],\n",{"type":32,"tag":84,"props":174,"children":176},{"class":86,"line":175},5,[177,182,186],{"type":32,"tag":84,"props":178,"children":179},{"style":110},[180],{"type":37,"value":181},"    \"expected_sentiment\"",{"type":32,"tag":84,"props":183,"children":184},{"style":91},[185],{"type":37,"value":118},{"type":32,"tag":84,"props":187,"children":188},{"style":121},[189],{"type":37,"value":190},"\"neutral\"\n",{"type":32,"tag":84,"props":192,"children":194},{"class":86,"line":193},6,[195],{"type":32,"tag":84,"props":196,"children":197},{"style":91},[198],{"type":37,"value":199},"  },\n",{"type":32,"tag":84,"props":201,"children":203},{"class":86,"line":202},7,[204],{"type":32,"tag":84,"props":205,"children":206},{"style":91},[207],{"type":37,"value":103},{"type":32,"tag":84,"props":209,"children":211},{"class":86,"line":210},8,[212,216,220,225],{"type":32,"tag":84,"props":213,"children":214},{"style":110},[215],{"type":37,"value":113},{"type":32,"tag":84,"props":217,"children":218},{"style":91},[219],{"type":37,"value":118},{"type":32,"tag":84,"props":221,"children":222},{"style":121},[223],{"type":37,"value":224},"\"Erkläre, warum die Churn Rate gestiegen ist\"",{"type":32,"tag":84,"props":226,"children":227},{"style":91},[228],{"type":37,"value":129},{"type":32,"tag":84,"props":230,"children":231},{"class":86,"line":26},[232,236,240,245,249,254],{"type":32,"tag":84,"props":233,"children":234},{"style":110},[235],{"type":37,"value":138},{"type":32,"tag":84,"props":237,"children":238},{"style":91},[239],{"type":37,"value":143},{"type":32,"tag":84,"props":241,"children":242},{"style":121},[243],{"type":37,"value":244},"\"churn\"",{"type":32,"tag":84,"props":246,"children":247},{"style":91},[248],{"type":37,"value":153},{"type":32,"tag":84,"props":250,"children":251},{"style":121},[252],{"type":37,"value":253},"\"retention\"",{"type":32,"tag":84,"props":255,"children":256},{"style":91},[257],{"type":37,"value":172},{"type":32,"tag":84,"props":259,"children":261},{"class":86,"line":260},10,[262,266,270],{"type":32,"tag":84,"props":263,"children":264},{"style":110},[265],{"type":37,"value":181},{"type":32,"tag":84,"props":267,"children":268},{"style":91},[269],{"type":37,"value":118},{"type":32,"tag":84,"props":271,"children":272},{"style":121},[273],{"type":37,"value":274},"\"analytical\"\n",{"type":32,"tag":84,"props":276,"children":278},{"class":86,"line":277},11,[279],{"type":32,"tag":84,"props":280,"children":281},{"style":91},[282],{"type":37,"value":283},"  }\n",{"type":32,"tag":84,"props":285,"children":287},{"class":86,"line":286},12,[288],{"type":32,"tag":84,"props":289,"children":290},{"style":91},[291],{"type":37,"value":292},"]\n",{"type":32,"tag":33,"props":294,"children":295},{},[296],{"type":37,"value":297},"Das Eval Set können Sie manuell erstellen (durch Sampling aus Production-Logs) oder synthetisch generieren (ein anderes LLM fragen: „generiere 50 Abwandlungen dieses Prompts\"). Das Wichtige: Das Set sollte Edge Cases abdecken — lange Inputs, mehrdeutige Anfragen, mehrere Sprachen.",{"type":32,"tag":33,"props":299,"children":300},{},[301],{"type":37,"value":302},"Die Eval-Metrik definiert, wie Sie Output bewerten. Es gibt zwei verbreitete Typen: Rule-based (prüfe auf bestimmte Wörter im Output) und LLM-as-Judge (frage ein anderes LLM: „beantwortet dieser Output die Frage korrekt, bewerte 1-5\"). LLM-as-Judge ist flexibler, aber teurer und langsamer. Für Balance zwischen Geschwindigkeit und Genauigkeit kombinieren Sie Rule-based + leichte Klassifizierer (wie BERT-basierte Sentiment-Modelle).",{"type":32,"tag":33,"props":304,"children":305},{},[306,308,314],{"type":37,"value":307},"Der Runner nimmt das Eval Set, führt jeden Input mit altem und neuem Prompt aus, vergleicht Outputs mit der Metrik und produziert eine Diff-Tabelle. Promptfoo macht das vom Terminal mit ",{"type":32,"tag":80,"props":309,"children":311},{"className":310},[],[312],{"type":37,"value":313},"promptfoo eval",{"type":37,"value":315},":",{"type":32,"tag":73,"props":317,"children":321},{"code":318,"language":319,"meta":16,"className":320,"style":16},"promptfoo eval \\\n  --prompts prompts\u002Fv1.txt prompts\u002Fv2.txt \\\n  --providers openai:gpt-4 \\\n  --tests evals\u002Fsummarization.json \\\n  --output results.json\n","bash","language-bash shiki shiki-themes github-dark",[322],{"type":32,"tag":80,"props":323,"children":324},{"__ignoreMap":16},[325,343,365,382,399],{"type":32,"tag":84,"props":326,"children":327},{"class":86,"line":87},[328,333,338],{"type":32,"tag":84,"props":329,"children":331},{"style":330},"--shiki-default:#B392F0",[332],{"type":37,"value":25},{"type":32,"tag":84,"props":334,"children":335},{"style":121},[336],{"type":37,"value":337}," eval",{"type":32,"tag":84,"props":339,"children":340},{"style":110},[341],{"type":37,"value":342}," \\\n",{"type":32,"tag":84,"props":344,"children":345},{"class":86,"line":97},[346,351,356,361],{"type":32,"tag":84,"props":347,"children":348},{"style":110},[349],{"type":37,"value":350},"  --prompts",{"type":32,"tag":84,"props":352,"children":353},{"style":121},[354],{"type":37,"value":355}," prompts\u002Fv1.txt",{"type":32,"tag":84,"props":357,"children":358},{"style":121},[359],{"type":37,"value":360}," prompts\u002Fv2.txt",{"type":32,"tag":84,"props":362,"children":363},{"style":110},[364],{"type":37,"value":342},{"type":32,"tag":84,"props":366,"children":367},{"class":86,"line":106},[368,373,378],{"type":32,"tag":84,"props":369,"children":370},{"style":110},[371],{"type":37,"value":372},"  --providers",{"type":32,"tag":84,"props":374,"children":375},{"style":121},[376],{"type":37,"value":377}," openai:gpt-4",{"type":32,"tag":84,"props":379,"children":380},{"style":110},[381],{"type":37,"value":342},{"type":32,"tag":84,"props":383,"children":384},{"class":86,"line":132},[385,390,395],{"type":32,"tag":84,"props":386,"children":387},{"style":110},[388],{"type":37,"value":389},"  --tests",{"type":32,"tag":84,"props":391,"children":392},{"style":121},[393],{"type":37,"value":394}," evals\u002Fsummarization.json",{"type":32,"tag":84,"props":396,"children":397},{"style":110},[398],{"type":37,"value":342},{"type":32,"tag":84,"props":400,"children":401},{"class":86,"line":175},[402,407],{"type":32,"tag":84,"props":403,"children":404},{"style":110},[405],{"type":37,"value":406},"  --output",{"type":32,"tag":84,"props":408,"children":409},{"style":121},[410],{"type":37,"value":411}," results.json\n",{"type":32,"tag":33,"props":413,"children":414},{},[415],{"type":37,"value":416},"Die Ausgabe zeigt für jeden Test Case, welcher Prompt besser abschneidet. Wenn der neue Prompt in 80 % der Eval-Set-Cases die Metrik verbessert, ist er deployment-ready. Andernfalls liegt eine Regression vor — überarbeiten Sie den Prompt.",{"type":32,"tag":40,"props":418,"children":420},{"id":419},"ab-testing-zwei-prompts-parallel-in-production",[421],{"type":37,"value":422},"A\u002FB-Testing: Zwei Prompts parallel in Production",{"type":32,"tag":33,"props":424,"children":425},{},[426],{"type":37,"value":427},"Die Eval Pipeline gibt Offline-Ergebnisse — echte Benutzerdaten fehlen. Um in Production zu messen, welcher Prompt besser funktioniert, führen Sie zwei Prompts parallel aus. Dafür brauchen Sie Traffic Splitting und Metric Collection.",{"type":32,"tag":33,"props":429,"children":430},{},[431,433,439],{"type":37,"value":432},"Traffic Splitting ist einfach: Nehmen Sie die User-ID oder Session-ID jedes Request, hashen Sie sie und nutzen Sie Modulo, um den Request prompt A oder B zuzuweisen. Beispiel: ",{"type":32,"tag":80,"props":434,"children":436},{"className":435},[],[437],{"type":37,"value":438},"hash(user_id) % 100 \u003C 50",{"type":37,"value":440}," → Prompt A, sonst Prompt B. Das ergibt ein 50\u002F50 Split. Wichtig: derselbe User sollte immer denselben Prompt sehen (Sticky Assignment) — sonst wird die User Experience inkonsistent.",{"type":32,"tag":33,"props":442,"children":443},{},[444,446,452,453,459,460,466,468,477],{"type":37,"value":445},"Bei Metric Collection fügen Sie zum LLM-Response Metadaten hinzu: ",{"type":32,"tag":80,"props":447,"children":449},{"className":448},[],[450],{"type":37,"value":451},"prompt_version",{"type":37,"value":153},{"type":32,"tag":80,"props":454,"children":456},{"className":455},[],[457],{"type":37,"value":458},"latency",{"type":37,"value":153},{"type":32,"tag":80,"props":461,"children":463},{"className":462},[],[464],{"type":37,"value":465},"token_count",{"type":37,"value":467},". Diese Daten fließen in Data Warehouse (BigQuery, Snowflake). Hier greift Roibases ",{"type":32,"tag":469,"props":470,"children":474},"a",{"href":471,"rel":472},"https:\u002F\u002Fwww.roibase.com.tr\u002Fde\u002Fverianalizi",[473],"nofollow",[475],{"type":37,"value":476},"Veri Analizi & İçgörü Mühendisliği",{"type":37,"value":478}," ein — Sie kombinieren LLM-Logs mit anderen Event-Daten (Nutzeraktionen, Conversions, Churn) und messen die Downstream-Effekte des Prompts.",{"type":32,"tag":33,"props":480,"children":481},{},[482],{"type":37,"value":483},"Welche Metriken verfolgen Sie beim A\u002FB-Test? Drei Kategorien:",{"type":32,"tag":485,"props":486,"children":487},"table",{},[488,512],{"type":32,"tag":489,"props":490,"children":491},"thead",{},[492],{"type":32,"tag":493,"props":494,"children":495},"tr",{},[496,502,507],{"type":32,"tag":497,"props":498,"children":499},"th",{},[500],{"type":37,"value":501},"Metrik-Typ",{"type":32,"tag":497,"props":503,"children":504},{},[505],{"type":37,"value":506},"Beispiel",{"type":32,"tag":497,"props":508,"children":509},{},[510],{"type":37,"value":511},"Ziel",{"type":32,"tag":513,"props":514,"children":515},"tbody",{},[516,535,553],{"type":32,"tag":493,"props":517,"children":518},{},[519,525,530],{"type":32,"tag":520,"props":521,"children":522},"td",{},[523],{"type":37,"value":524},"Qualität",{"type":32,"tag":520,"props":526,"children":527},{},[528],{"type":37,"value":529},"LLM-as-Judge Score, Hallucination Rate",{"type":32,"tag":520,"props":531,"children":532},{},[533],{"type":37,"value":534},"Hoch",{"type":32,"tag":493,"props":536,"children":537},{},[538,543,548],{"type":32,"tag":520,"props":539,"children":540},{},[541],{"type":37,"value":542},"Kosten",{"type":32,"tag":520,"props":544,"children":545},{},[546],{"type":37,"value":547},"Token Count, API-Kosten",{"type":32,"tag":520,"props":549,"children":550},{},[551],{"type":37,"value":552},"Niedrig",{"type":32,"tag":493,"props":554,"children":555},{},[556,561,566],{"type":32,"tag":520,"props":557,"children":558},{},[559],{"type":37,"value":560},"Downstream",{"type":32,"tag":520,"props":562,"children":563},{},[564],{"type":37,"value":565},"Conversion Rate, User Engagement",{"type":32,"tag":520,"props":567,"children":568},{},[569],{"type":37,"value":534},{"type":32,"tag":33,"props":571,"children":572},{},[573],{"type":37,"value":574},"Beispiel: Prompt B hebt den LLM-as-Judge Score um 12 % vs. Prompt A, erhöht aber die Token Count um 35 %. Es gibt einen Tradeoff. Wenn die Downstream-Conversion gleich bleibt, ist Prompt A effizienter.",{"type":32,"tag":40,"props":576,"children":578},{"id":577},"langsmith-und-observability",[579],{"type":37,"value":580},"LangSmith und Observability",{"type":32,"tag":33,"props":582,"children":583},{},[584],{"type":37,"value":585},"LangSmith ist eine LLM-Observability-Plattform vom LangChain-Team. Sie geht über Evaluation hinaus: Sie erfasst Production-Traces, visualisiert Prompt-Chains und zeigt, wo Latenz steigt. Besonders bei Multi-Step-Workflows (RAG + Summarization + JSON Parsing) ist Debugging kritisch.",{"type":32,"tag":33,"props":587,"children":588},{},[589],{"type":37,"value":590},"Traces zu LangSmith senden Sie über das SDK:",{"type":32,"tag":73,"props":592,"children":596},{"code":593,"language":594,"meta":16,"className":595,"style":16},"from langsmith import Client\nclient = Client(api_key=\"...\")\n\nwith client.trace(name=\"summarize_revenue\"):\n    result = llm.invoke(prompt)\n    client.log_metric(\"token_count\", result.usage.total_tokens)\n","python","language-python shiki shiki-themes github-dark",[597],{"type":32,"tag":80,"props":598,"children":599},{"__ignoreMap":16},[600,624,662,671,703,720],{"type":32,"tag":84,"props":601,"children":602},{"class":86,"line":87},[603,609,614,619],{"type":32,"tag":84,"props":604,"children":606},{"style":605},"--shiki-default:#F97583",[607],{"type":37,"value":608},"from",{"type":32,"tag":84,"props":610,"children":611},{"style":91},[612],{"type":37,"value":613}," langsmith ",{"type":32,"tag":84,"props":615,"children":616},{"style":605},[617],{"type":37,"value":618},"import",{"type":32,"tag":84,"props":620,"children":621},{"style":91},[622],{"type":37,"value":623}," Client\n",{"type":32,"tag":84,"props":625,"children":626},{"class":86,"line":97},[627,632,637,642,648,652,657],{"type":32,"tag":84,"props":628,"children":629},{"style":91},[630],{"type":37,"value":631},"client ",{"type":32,"tag":84,"props":633,"children":634},{"style":605},[635],{"type":37,"value":636},"=",{"type":32,"tag":84,"props":638,"children":639},{"style":91},[640],{"type":37,"value":641}," Client(",{"type":32,"tag":84,"props":643,"children":645},{"style":644},"--shiki-default:#FFAB70",[646],{"type":37,"value":647},"api_key",{"type":32,"tag":84,"props":649,"children":650},{"style":605},[651],{"type":37,"value":636},{"type":32,"tag":84,"props":653,"children":654},{"style":121},[655],{"type":37,"value":656},"\"...\"",{"type":32,"tag":84,"props":658,"children":659},{"style":91},[660],{"type":37,"value":661},")\n",{"type":32,"tag":84,"props":663,"children":664},{"class":86,"line":106},[665],{"type":32,"tag":84,"props":666,"children":668},{"emptyLinePlaceholder":667},true,[669],{"type":37,"value":670},"\n",{"type":32,"tag":84,"props":672,"children":673},{"class":86,"line":132},[674,679,684,689,693,698],{"type":32,"tag":84,"props":675,"children":676},{"style":605},[677],{"type":37,"value":678},"with",{"type":32,"tag":84,"props":680,"children":681},{"style":91},[682],{"type":37,"value":683}," client.trace(",{"type":32,"tag":84,"props":685,"children":686},{"style":644},[687],{"type":37,"value":688},"name",{"type":32,"tag":84,"props":690,"children":691},{"style":605},[692],{"type":37,"value":636},{"type":32,"tag":84,"props":694,"children":695},{"style":121},[696],{"type":37,"value":697},"\"summarize_revenue\"",{"type":32,"tag":84,"props":699,"children":700},{"style":91},[701],{"type":37,"value":702},"):\n",{"type":32,"tag":84,"props":704,"children":705},{"class":86,"line":175},[706,711,715],{"type":32,"tag":84,"props":707,"children":708},{"style":91},[709],{"type":37,"value":710},"    result ",{"type":32,"tag":84,"props":712,"children":713},{"style":605},[714],{"type":37,"value":636},{"type":32,"tag":84,"props":716,"children":717},{"style":91},[718],{"type":37,"value":719}," llm.invoke(prompt)\n",{"type":32,"tag":84,"props":721,"children":722},{"class":86,"line":193},[723,728,733],{"type":32,"tag":84,"props":724,"children":725},{"style":91},[726],{"type":37,"value":727},"    client.log_metric(",{"type":32,"tag":84,"props":729,"children":730},{"style":121},[731],{"type":37,"value":732},"\"token_count\"",{"type":32,"tag":84,"props":734,"children":735},{"style":91},[736],{"type":37,"value":737},", result.usage.total_tokens)\n",{"type":32,"tag":33,"props":739,"children":740},{},[741],{"type":37,"value":742},"Jeder Trace ist in der LangSmith UI sichtbar — Input, Output und Metadaten sind vollständig geloggt. Mit mehreren Prompt-Versionen öffnen Sie eine Vergleichsansicht. Sie sehen Insights wie: „Prompt v2 produziert durchschnittlich 8 % längere Outputs als v1, aber die Latenz ist 3 % niedriger.\"",{"type":32,"tag":33,"props":744,"children":745},{},[746],{"type":37,"value":747},"LangSmith bietet auch einen Playground — Sie ändern den Prompt und testen ihn mit einem Click gegen mehrere Inputs. Das erzeugt schnelles Feedback für Prototyping und Regressionstests. Aber Vorsicht: Playground-Tests ersetzen nicht das Production-A\u002FB-Testing, sie sind nur ein erstes Filter.",{"type":32,"tag":40,"props":749,"children":751},{"id":750},"der-zweite-effekt-der-prompt-versionierung-rollback",[752],{"type":37,"value":753},"Der zweite Effekt der Prompt-Versionierung: Rollback",{"type":32,"tag":33,"props":755,"children":756},{},[757],{"type":37,"value":758},"Im Fehlerfall können Sie auf eine frühere Prompt-Version zurückgehen. Im LLM-Betrieb bedeutet Rollback, zur vorherigen Prompt-Version zu wechseln. Aber dafür brauchen Sie eine Versionshistorie.",{"type":32,"tag":33,"props":760,"children":761},{},[762,764,770],{"type":37,"value":763},"Einfacher Ansatz: Jeden Prompt in einer separaten Git-Datei ablegen (",{"type":32,"tag":80,"props":765,"children":767},{"className":766},[],[768],{"type":37,"value":769},"prompts\u002Fsummarization_v3.txt",{"type":37,"value":771},"). Ein Deployment-Script speichert, welche Version in Production läuft:",{"type":32,"tag":73,"props":773,"children":777},{"code":774,"language":775,"meta":16,"className":776,"style":16},"# config\u002Fproduction.yaml\nprompts:\n  summarization: v3\n  classification: v2\n","yaml","language-yaml shiki shiki-themes github-dark",[778],{"type":32,"tag":80,"props":779,"children":780},{"__ignoreMap":16},[781,790,804,821],{"type":32,"tag":84,"props":782,"children":783},{"class":86,"line":87},[784],{"type":32,"tag":84,"props":785,"children":787},{"style":786},"--shiki-default:#6A737D",[788],{"type":37,"value":789},"# config\u002Fproduction.yaml\n",{"type":32,"tag":84,"props":791,"children":792},{"class":86,"line":97},[793,799],{"type":32,"tag":84,"props":794,"children":796},{"style":795},"--shiki-default:#85E89D",[797],{"type":37,"value":798},"prompts",{"type":32,"tag":84,"props":800,"children":801},{"style":91},[802],{"type":37,"value":803},":\n",{"type":32,"tag":84,"props":805,"children":806},{"class":86,"line":106},[807,812,816],{"type":32,"tag":84,"props":808,"children":809},{"style":795},[810],{"type":37,"value":811},"  summarization",{"type":32,"tag":84,"props":813,"children":814},{"style":91},[815],{"type":37,"value":118},{"type":32,"tag":84,"props":817,"children":818},{"style":121},[819],{"type":37,"value":820},"v3\n",{"type":32,"tag":84,"props":822,"children":823},{"class":86,"line":132},[824,829,833],{"type":32,"tag":84,"props":825,"children":826},{"style":795},[827],{"type":37,"value":828},"  classification",{"type":32,"tag":84,"props":830,"children":831},{"style":91},[832],{"type":37,"value":118},{"type":32,"tag":84,"props":834,"children":835},{"style":121},[836],{"type":37,"value":837},"v2\n",{"type":32,"tag":33,"props":839,"children":840},{},[841,843,849],{"type":37,"value":842},"Zum Rollback schreiben Sie ",{"type":32,"tag":80,"props":844,"children":846},{"className":845},[],[847],{"type":37,"value":848},"summarization: v2",{"type":37,"value":850}," und triggern das Deployment. Das ist aber ein manueller Prozess — bei Incidents zu langsam. Ein fortgeschrittener Ansatz: Feature-Flag-Systeme (LaunchDarkly, Unleash). Sie ändern die Prompt-Version zur Laufzeit, ohne Code zu deployen.",{"type":32,"tag":33,"props":852,"children":853},{},[854,856,863],{"type":37,"value":855},"Roibases Praktiken der ",{"type":32,"tag":469,"props":857,"children":860},{"href":858,"rel":859},"https:\u002F\u002Fwww.roibase.com.tr\u002Fde\u002Ffirstparty",[473],[861],{"type":37,"value":862},"First-Party-Datenverwaltung und Messung",{"type":37,"value":864}," greifen hier ein — Sie müssen Prompt-Änderungen mit Downstream-Events (Conversions, Churn) verknüpfen, um Rollback-Entscheidungen auf Daten zu gründen. Wenn die Churn Rate 6 Stunden nach dem neuen Prompt um 4 % steigt, ist das ein Signal zum Rollback.",{"type":32,"tag":40,"props":866,"children":868},{"id":867},"edge-case-mehrsprachige-prompt-versionierung",[869],{"type":37,"value":870},"Edge Case: Mehrsprachige Prompt-Versionierung",{"type":32,"tag":33,"props":872,"children":873},{},[874],{"type":37,"value":875},"Wenn Ihre LLM-Anwendung in mehreren Sprachen läuft (z.B. DE, EN, TR), müssen Sie separate Prompt-Versionen für jede Sprache führen. Ein englischer Prompt könnte auf Deutsch den gleichen Ton nicht treffen.",{"type":32,"tag":33,"props":877,"children":878},{},[879],{"type":37,"value":880},"Lösung: Prompt-Dateien nach Sprachcode organisieren:",{"type":32,"tag":73,"props":882,"children":884},{"code":883},"prompts\u002F\n  summarization\u002F\n    en_v3.txt\n    tr_v3.txt\n    de_v3.txt\n",[885],{"type":32,"tag":80,"props":886,"children":887},{"__ignoreMap":16},[888],{"type":37,"value":883},{"type":32,"tag":33,"props":890,"children":891},{},[892],{"type":37,"value":893},"Das Eval Set sollte auch sprachspezifisch sein — für deutsche Test Cases deutsche Output-Erwartungen setzen. A\u002FB-Tests pro Sprache ausführen, da deutschsprachige User anders agieren als englischsprachige. Vergessen Sie nicht, Language Segment bei der Metrik-Aggregation einzufügen.",{"type":32,"tag":33,"props":895,"children":896},{},[897],{"type":37,"value":898},"Ein weiterer Punkt: In mehrsprachigen Prompts variiert die Context-Länge je nach Sprache — deutschsprachige Sätze sind durchschnittlich 12 % länger (in Tokens). Das ist ein Token-Limit-Risiko. Fügen Sie Token-Count-Kontrollen in Ihre Eval Pipeline ein und warnen Sie bei Threshold-Überschreitungen.",{"type":32,"tag":40,"props":900,"children":902},{"id":901},"praktische-erste-schritte-ein-minimales-eval-set",[903],{"type":37,"value":904},"Praktische erste Schritte: Ein minimales Eval Set",{"type":32,"tag":33,"props":906,"children":907},{},[908],{"type":37,"value":909},"Um mit den beschriebenen Systemen zu starten, bauen Sie zunächst ein minimales Eval Set aus 20-30 echten User-Queries. Öffnen Sie Ihre Production-Logs, wählen Sie die häufigsten Queries, definieren Sie für jeden die erwarteten Output-Eigenschaften (Genauigkeit, Ton, Länge).",{"type":32,"tag":33,"props":911,"children":912},{},[913],{"type":37,"value":914},"Dann richten Sie Promptfoo oder LangSmith ein, führen Ihren aktuellen Prompt gegen das Set aus, erhalten einen Baseline-Score. Jetzt ändern Sie etwas im Prompt (z.B. „antworte kurz und knapp\"), evaluieren erneut und vergleichen die Scores. Wenn keine Regression über 5 % auftritt, deployen Sie die Änderung.",{"type":32,"tag":33,"props":916,"children":917},{},[918],{"type":37,"value":919},"Wenn dieser Zyklus automatisiert läuft, steigt Ihre Prompt-Iterations-Geschwindigkeit um 3x. Denn jetzt beantworten Sie „ist diese Änderung gut oder schlecht?\" nicht mit Vermutung, sondern mit Zahlen.",{"type":32,"tag":921,"props":922,"children":923},"style",{},[924],{"type":37,"value":925},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":16,"searchDepth":106,"depth":106,"links":927},[928,929,930,931,932,933,934],{"id":42,"depth":97,"text":45},{"id":63,"depth":97,"text":66},{"id":419,"depth":97,"text":422},{"id":577,"depth":97,"text":580},{"id":750,"depth":97,"text":753},{"id":867,"depth":97,"text":870},{"id":901,"depth":97,"text":904},"markdown","content:de:ai:prompt-versionierung-und-ab-test-llm-operations-disziplin.md","content","de\u002Fai\u002Fprompt-versionierung-und-ab-test-llm-operations-disziplin.md","de\u002Fai\u002Fprompt-versionierung-und-ab-test-llm-operations-disziplin","md",1783289075685]