[{"data":1,"prerenderedAt":957},["ShallowReactive",2],{"article-alternates":3,"article-\u002Fru\u002Fai\u002Fversiyonlama-a-b-test-llm-operatsii":12},{"i18nKey":4,"paths":5},"ai-004-2026-06",{"de":6,"en":7,"es":8,"fr":9,"it":10,"ru":11},"\u002Fde\u002Fai\u002Fprompt-versionierung-und-ab-test-llm-operations-disziplin","\u002Fen\u002Fai\u002Fprompt-versioning-and-ab-testing-llm-operations-discipline","\u002Fes\u002Fai\u002Fversionamiento-de-prompts-y-pruebas-ab-disciplina-en-operaciones-llm","\u002Ffr\u002Fai\u002Fversionnage-des-prompts-et-tests-ab-discipline-des-operations-llm","\u002Fit\u002Fai\u002Fversionamento-prompt-e-test-ab-discipline-operativa-llm","\u002Fru\u002Fai\u002Fversiyonlama-t",{"_path":13,"_dir":14,"_draft":15,"_partial":15,"_locale":16,"title":17,"description":18,"publishedAt":19,"modifiedAt":19,"category":14,"i18nKey":4,"tags":20,"readingTime":26,"author":27,"body":28,"_type":951,"_id":952,"_source":953,"_file":954,"_stem":955,"_extension":956},"\u002Fru\u002Fai\u002Fversiyonlama-a-b-test-llm-operatsii","ai",false,"","Версионирование Prompt'а и A\u002FB-тестирование: Дисциплина LLM-операций","С помощью Promptfoo, LangSmith и evaluation pipeline'ов сделайте изменения prompt'ов измеримыми. Как настроить версионирование и A\u002FB-тестирование в production LLM-операциях?","2026-06-22",[21,22,23,24,25],"prompt-engineering","llm-ops","evaluation","ab-testing","promptfoo",8,"Roibase",{"type":29,"children":30,"toc":942},"root",[31,39,46,51,56,61,67,72,293,298,303,316,412,417,423,428,457,495,500,586,591,597,602,607,754,759,764,770,775,788,854,867,881,887,892,897,905,910,915,921,926,931,936],{"type":32,"tag":33,"props":34,"children":35},"element","p",{},[36],{"type":37,"value":38},"text","Запуск LLM в production — это уже не просто несколько вызовов API. Когда вы меняете prompt, качество output может упасть на 15% или вырасти на 22% — но если вы этого не замечаете, развёртывание превращается в лотерею. Версионирование prompt'ов и A\u002FB-тестирование переносят дисциплину развёртывания ПО в LLM-операции. В этой статье рассказываем, как использовать evaluation framework'и вроде Promptfoo и LangSmith, чтобы сделать изменения prompt'ов измеримыми.",{"type":32,"tag":40,"props":41,"children":43},"h2",{"id":42},"изменение-promptа-это-не-развёртывание",[44],{"type":37,"value":45},"Изменение prompt'а — это не развёртывание",{"type":32,"tag":33,"props":47,"children":48},{},[49],{"type":37,"value":50},"В классическом software engineering'е при изменении функции в ход идут unit-тесты, integration-тесты и canary-развёртывание. В LLM-операциях же большинство команд меняют prompt в обычном текстовом файле, проводят несколько ручных проверок и выкатывают в production. Результат: user sentiment падает на 8%, но никто не видит связи.",{"type":32,"tag":33,"props":52,"children":53},{},[54],{"type":37,"value":55},"Проблема в том, что LLM-output недетерминирован. На один и тот же prompt вы получите разные ответы, что делает тестирование на одном примере бесполезным. Без системы версионирования вы не можете ответить на вопрос \"какой prompt был лучше — старый или новый?\". Даже коммит в Git недостаточен — семантическую разницу нельзя вытащить из сообщения коммита.",{"type":32,"tag":33,"props":57,"children":58},{},[59],{"type":37,"value":60},"Решение: каждое изменение prompt'а записывайте как версию, запускайте eval set до и после изменения, сравнивайте метрики. Такая дисциплина даёт два преимущества: detection regression'ов (не сломал ли новый prompt старые задачи) и measurement улучшений (действительно ли целевая метрика выросла).",{"type":32,"tag":40,"props":62,"children":64},{"id":63},"как-устроена-evaluation-pipeline",[65],{"type":37,"value":66},"Как устроена evaluation pipeline",{"type":32,"tag":33,"props":68,"children":69},{},[70],{"type":37,"value":71},"Evaluation pipeline состоит из трёх компонентов: eval set, eval метрика, runner. Eval set — это список input'ов, которые пойдут в LLM, и ожидаемых output'ов (или свойств output'ов). В JSON это выглядит так:",{"type":32,"tag":73,"props":74,"children":78},"pre",{"code":75,"language":76,"meta":16,"className":77,"style":16},"[\n  {\n    \"input\": \"Суммаризируй тренд revenue за Q1 2025\",\n    \"expected_topics\": [\"revenue\", \"growth\", \"quarter\"],\n    \"expected_sentiment\": \"neutral\"\n  },\n  {\n    \"input\": \"Объясни, почему выросла churn rate\",\n    \"expected_topics\": [\"churn\", \"retention\"],\n    \"expected_sentiment\": \"analytical\"\n  }\n]\n","json","language-json shiki shiki-themes github-dark",[79],{"type":32,"tag":80,"props":81,"children":82},"code",{"__ignoreMap":16},[83,95,104,130,173,191,200,208,228,258,275,284],{"type":32,"tag":84,"props":85,"children":88},"span",{"class":86,"line":87},"line",1,[89],{"type":32,"tag":84,"props":90,"children":92},{"style":91},"--shiki-default:#E1E4E8",[93],{"type":37,"value":94},"[\n",{"type":32,"tag":84,"props":96,"children":98},{"class":86,"line":97},2,[99],{"type":32,"tag":84,"props":100,"children":101},{"style":91},[102],{"type":37,"value":103},"  {\n",{"type":32,"tag":84,"props":105,"children":107},{"class":86,"line":106},3,[108,114,119,125],{"type":32,"tag":84,"props":109,"children":111},{"style":110},"--shiki-default:#79B8FF",[112],{"type":37,"value":113},"    \"input\"",{"type":32,"tag":84,"props":115,"children":116},{"style":91},[117],{"type":37,"value":118},": ",{"type":32,"tag":84,"props":120,"children":122},{"style":121},"--shiki-default:#9ECBFF",[123],{"type":37,"value":124},"\"Суммаризируй тренд revenue за Q1 2025\"",{"type":32,"tag":84,"props":126,"children":127},{"style":91},[128],{"type":37,"value":129},",\n",{"type":32,"tag":84,"props":131,"children":133},{"class":86,"line":132},4,[134,139,144,149,154,159,163,168],{"type":32,"tag":84,"props":135,"children":136},{"style":110},[137],{"type":37,"value":138},"    \"expected_topics\"",{"type":32,"tag":84,"props":140,"children":141},{"style":91},[142],{"type":37,"value":143},": [",{"type":32,"tag":84,"props":145,"children":146},{"style":121},[147],{"type":37,"value":148},"\"revenue\"",{"type":32,"tag":84,"props":150,"children":151},{"style":91},[152],{"type":37,"value":153},", ",{"type":32,"tag":84,"props":155,"children":156},{"style":121},[157],{"type":37,"value":158},"\"growth\"",{"type":32,"tag":84,"props":160,"children":161},{"style":91},[162],{"type":37,"value":153},{"type":32,"tag":84,"props":164,"children":165},{"style":121},[166],{"type":37,"value":167},"\"quarter\"",{"type":32,"tag":84,"props":169,"children":170},{"style":91},[171],{"type":37,"value":172},"],\n",{"type":32,"tag":84,"props":174,"children":176},{"class":86,"line":175},5,[177,182,186],{"type":32,"tag":84,"props":178,"children":179},{"style":110},[180],{"type":37,"value":181},"    \"expected_sentiment\"",{"type":32,"tag":84,"props":183,"children":184},{"style":91},[185],{"type":37,"value":118},{"type":32,"tag":84,"props":187,"children":188},{"style":121},[189],{"type":37,"value":190},"\"neutral\"\n",{"type":32,"tag":84,"props":192,"children":194},{"class":86,"line":193},6,[195],{"type":32,"tag":84,"props":196,"children":197},{"style":91},[198],{"type":37,"value":199},"  },\n",{"type":32,"tag":84,"props":201,"children":203},{"class":86,"line":202},7,[204],{"type":32,"tag":84,"props":205,"children":206},{"style":91},[207],{"type":37,"value":103},{"type":32,"tag":84,"props":209,"children":210},{"class":86,"line":26},[211,215,219,224],{"type":32,"tag":84,"props":212,"children":213},{"style":110},[214],{"type":37,"value":113},{"type":32,"tag":84,"props":216,"children":217},{"style":91},[218],{"type":37,"value":118},{"type":32,"tag":84,"props":220,"children":221},{"style":121},[222],{"type":37,"value":223},"\"Объясни, почему выросла churn rate\"",{"type":32,"tag":84,"props":225,"children":226},{"style":91},[227],{"type":37,"value":129},{"type":32,"tag":84,"props":229,"children":231},{"class":86,"line":230},9,[232,236,240,245,249,254],{"type":32,"tag":84,"props":233,"children":234},{"style":110},[235],{"type":37,"value":138},{"type":32,"tag":84,"props":237,"children":238},{"style":91},[239],{"type":37,"value":143},{"type":32,"tag":84,"props":241,"children":242},{"style":121},[243],{"type":37,"value":244},"\"churn\"",{"type":32,"tag":84,"props":246,"children":247},{"style":91},[248],{"type":37,"value":153},{"type":32,"tag":84,"props":250,"children":251},{"style":121},[252],{"type":37,"value":253},"\"retention\"",{"type":32,"tag":84,"props":255,"children":256},{"style":91},[257],{"type":37,"value":172},{"type":32,"tag":84,"props":259,"children":261},{"class":86,"line":260},10,[262,266,270],{"type":32,"tag":84,"props":263,"children":264},{"style":110},[265],{"type":37,"value":181},{"type":32,"tag":84,"props":267,"children":268},{"style":91},[269],{"type":37,"value":118},{"type":32,"tag":84,"props":271,"children":272},{"style":121},[273],{"type":37,"value":274},"\"analytical\"\n",{"type":32,"tag":84,"props":276,"children":278},{"class":86,"line":277},11,[279],{"type":32,"tag":84,"props":280,"children":281},{"style":91},[282],{"type":37,"value":283},"  }\n",{"type":32,"tag":84,"props":285,"children":287},{"class":86,"line":286},12,[288],{"type":32,"tag":84,"props":289,"children":290},{"style":91},[291],{"type":37,"value":292},"]\n",{"type":32,"tag":33,"props":294,"children":295},{},[296],{"type":37,"value":297},"Eval set можно создать вручную (sampling из production логов) или синтетически (попросить другую LLM \"сгенерируй 50 вариаций запросов для этого prompt'а\"). Главное — чтобы set покрывал edge case'ы: длинные input'ы, неоднозначные вопросы, мультиязычность.",{"type":32,"tag":33,"props":299,"children":300},{},[301],{"type":37,"value":302},"Eval метрика определяет, как оценивать output LLM. Два основных типа: rule-based (проверка наличия определённых слов в output'е) и LLM-as-judge (попросить другую LLM оценить ответ по шкале 1–5). LLM-as-judge гибче, но дороже и медленнее. Для баланса скорости и точности часто используют комбинацию: rule-based + лёгкий classifier (например, BERT для sentiment).",{"type":32,"tag":33,"props":304,"children":305},{},[306,308,314],{"type":37,"value":307},"Runner берёт eval set, запускает старый и новый prompt на каждом input'е, сравнивает output'ы метриками и выдаёт diff-таблицу. Promptfoo это делает командой ",{"type":32,"tag":80,"props":309,"children":311},{"className":310},[],[312],{"type":37,"value":313},"promptfoo eval",{"type":37,"value":315},":",{"type":32,"tag":73,"props":317,"children":321},{"code":318,"language":319,"meta":16,"className":320,"style":16},"promptfoo eval \\\n  --prompts prompts\u002Fv1.txt prompts\u002Fv2.txt \\\n  --providers openai:gpt-4 \\\n  --tests evals\u002Fsummarization.json \\\n  --output results.json\n","bash","language-bash shiki shiki-themes github-dark",[322],{"type":32,"tag":80,"props":323,"children":324},{"__ignoreMap":16},[325,343,365,382,399],{"type":32,"tag":84,"props":326,"children":327},{"class":86,"line":87},[328,333,338],{"type":32,"tag":84,"props":329,"children":331},{"style":330},"--shiki-default:#B392F0",[332],{"type":37,"value":25},{"type":32,"tag":84,"props":334,"children":335},{"style":121},[336],{"type":37,"value":337}," eval",{"type":32,"tag":84,"props":339,"children":340},{"style":110},[341],{"type":37,"value":342}," \\\n",{"type":32,"tag":84,"props":344,"children":345},{"class":86,"line":97},[346,351,356,361],{"type":32,"tag":84,"props":347,"children":348},{"style":110},[349],{"type":37,"value":350},"  --prompts",{"type":32,"tag":84,"props":352,"children":353},{"style":121},[354],{"type":37,"value":355}," prompts\u002Fv1.txt",{"type":32,"tag":84,"props":357,"children":358},{"style":121},[359],{"type":37,"value":360}," prompts\u002Fv2.txt",{"type":32,"tag":84,"props":362,"children":363},{"style":110},[364],{"type":37,"value":342},{"type":32,"tag":84,"props":366,"children":367},{"class":86,"line":106},[368,373,378],{"type":32,"tag":84,"props":369,"children":370},{"style":110},[371],{"type":37,"value":372},"  --providers",{"type":32,"tag":84,"props":374,"children":375},{"style":121},[376],{"type":37,"value":377}," openai:gpt-4",{"type":32,"tag":84,"props":379,"children":380},{"style":110},[381],{"type":37,"value":342},{"type":32,"tag":84,"props":383,"children":384},{"class":86,"line":132},[385,390,395],{"type":32,"tag":84,"props":386,"children":387},{"style":110},[388],{"type":37,"value":389},"  --tests",{"type":32,"tag":84,"props":391,"children":392},{"style":121},[393],{"type":37,"value":394}," evals\u002Fsummarization.json",{"type":32,"tag":84,"props":396,"children":397},{"style":110},[398],{"type":37,"value":342},{"type":32,"tag":84,"props":400,"children":401},{"class":86,"line":175},[402,407],{"type":32,"tag":84,"props":403,"children":404},{"style":110},[405],{"type":37,"value":406},"  --output",{"type":32,"tag":84,"props":408,"children":409},{"style":121},[410],{"type":37,"value":411}," results.json\n",{"type":32,"tag":33,"props":413,"children":414},{},[415],{"type":37,"value":416},"В результате для каждого test case видно, какой prompt показал лучше. Если новый prompt улучшил метрику на 80% test case'ов — готов к развёртыванию. Иначе есть regression, нужно доработать prompt.",{"type":32,"tag":40,"props":418,"children":420},{"id":419},"ab-тестирование-параллельный-запуск-двух-promptов-в-production",[421],{"type":37,"value":422},"A\u002FB-тестирование: параллельный запуск двух prompt'ов в production",{"type":32,"tag":33,"props":424,"children":425},{},[426],{"type":37,"value":427},"Evaluation pipeline даёт offline результаты — нет данных реальных пользователей. Чтобы измерить, какой prompt работает лучше на живых пользователях, нужно A\u002FB-тестирование: параллельно запускать оба prompt'а и смотреть, какой даёт лучший результат. Для этого нужна инфраструктура traffic splitting и сбора метрик.",{"type":32,"tag":33,"props":429,"children":430},{},[431,433,439,441,447,449,455],{"type":37,"value":432},"Traffic splitting просто: входящий request'ом берёте ",{"type":32,"tag":80,"props":434,"children":436},{"className":435},[],[437],{"type":37,"value":438},"user_id",{"type":37,"value":440}," или ",{"type":32,"tag":80,"props":442,"children":444},{"className":443},[],[445],{"type":37,"value":446},"session_id",{"type":37,"value":448},", хешируете, берёте модуль. Например, если ",{"type":32,"tag":80,"props":450,"children":452},{"className":451},[],[453],{"type":37,"value":454},"hash(user_id) % 100 \u003C 50",{"type":37,"value":456}," — prompt A, иначе — B. Так делаете 50\u002F50 split. Важный момент: один и тот же пользователь должен видеть один и тот же prompt в каждом request'е (sticky assignment) — иначе пользовательский опыт будет непредсказуем.",{"type":32,"tag":33,"props":458,"children":459},{},[460,462,468,469,475,476,482,484,493],{"type":37,"value":461},"Для сбора метрик к LLM-ответу добавляется метаинформация: ",{"type":32,"tag":80,"props":463,"children":465},{"className":464},[],[466],{"type":37,"value":467},"prompt_version",{"type":37,"value":153},{"type":32,"tag":80,"props":470,"children":472},{"className":471},[],[473],{"type":37,"value":474},"latency",{"type":37,"value":153},{"type":32,"tag":80,"props":477,"children":479},{"className":478},[],[480],{"type":37,"value":481},"token_count",{"type":37,"value":483},". Эти данные поступают в data warehouse (BigQuery, Snowflake). Здесь включается ",{"type":32,"tag":485,"props":486,"children":490},"a",{"href":487,"rel":488},"https:\u002F\u002Fwww.roibase.com.tr\u002Fru\u002Fverianalizi",[489],"nofollow",[491],{"type":37,"value":492},"Veri Analizi & İçgörü Mühendisliği",{"type":37,"value":494}," pipeline'а Roibase — вы объединяете LLM-логи с другими event'ами (действия пользователя, conversion, churn) и измеряете downstream эффект prompt'а.",{"type":32,"tag":33,"props":496,"children":497},{},[498],{"type":37,"value":499},"Какие метрики отслеживать в A\u002FB-тесте? Три категории:",{"type":32,"tag":501,"props":502,"children":503},"table",{},[504,528],{"type":32,"tag":505,"props":506,"children":507},"thead",{},[508],{"type":32,"tag":509,"props":510,"children":511},"tr",{},[512,518,523],{"type":32,"tag":513,"props":514,"children":515},"th",{},[516],{"type":37,"value":517},"Тип метрики",{"type":32,"tag":513,"props":519,"children":520},{},[521],{"type":37,"value":522},"Пример",{"type":32,"tag":513,"props":524,"children":525},{},[526],{"type":37,"value":527},"Цель",{"type":32,"tag":529,"props":530,"children":531},"tbody",{},[532,551,569],{"type":32,"tag":509,"props":533,"children":534},{},[535,541,546],{"type":32,"tag":536,"props":537,"children":538},"td",{},[539],{"type":37,"value":540},"Качество",{"type":32,"tag":536,"props":542,"children":543},{},[544],{"type":37,"value":545},"LLM-as-judge score, hallucination rate",{"type":32,"tag":536,"props":547,"children":548},{},[549],{"type":37,"value":550},"Высокая",{"type":32,"tag":509,"props":552,"children":553},{},[554,559,564],{"type":32,"tag":536,"props":555,"children":556},{},[557],{"type":37,"value":558},"Стоимость",{"type":32,"tag":536,"props":560,"children":561},{},[562],{"type":37,"value":563},"Token count, API cost",{"type":32,"tag":536,"props":565,"children":566},{},[567],{"type":37,"value":568},"Низкая",{"type":32,"tag":509,"props":570,"children":571},{},[572,577,582],{"type":32,"tag":536,"props":573,"children":574},{},[575],{"type":37,"value":576},"Downstream",{"type":32,"tag":536,"props":578,"children":579},{},[580],{"type":37,"value":581},"Conversion rate, user engagement",{"type":32,"tag":536,"props":583,"children":584},{},[585],{"type":37,"value":550},{"type":32,"tag":33,"props":587,"children":588},{},[589],{"type":37,"value":590},"Например, prompt B даёт LLM-as-judge score выше на 12%, чем prompt A, но требует на 35% больше токенов — есть trade-off. Если downstream conversion не отличается, prompt A эффективнее.",{"type":32,"tag":40,"props":592,"children":594},{"id":593},"langsmith-и-observability",[595],{"type":37,"value":596},"LangSmith и observability",{"type":32,"tag":33,"props":598,"children":599},{},[600],{"type":37,"value":601},"LangSmith — платформа LLM observability от команды LangChain. Помимо evaluation, она захватывает production trace'ы, визуализирует цепочки prompt'ов, показывает, где растёт latency. Особенно критично для multi-step LLM workflow'ов (RAG + summarization + JSON parsing) при отладке.",{"type":32,"tag":33,"props":603,"children":604},{},[605],{"type":37,"value":606},"Отправить trace в LangSmith можно через SDK:",{"type":32,"tag":73,"props":608,"children":612},{"code":609,"language":610,"meta":16,"className":611,"style":16},"from langsmith import Client\nclient = Client(api_key=\"...\")\n\nwith client.trace(name=\"summarize_revenue\"):\n    result = llm.invoke(prompt)\n    client.log_metric(\"token_count\", result.usage.total_tokens)\n","python","language-python shiki shiki-themes github-dark",[613],{"type":32,"tag":80,"props":614,"children":615},{"__ignoreMap":16},[616,640,678,687,719,736],{"type":32,"tag":84,"props":617,"children":618},{"class":86,"line":87},[619,625,630,635],{"type":32,"tag":84,"props":620,"children":622},{"style":621},"--shiki-default:#F97583",[623],{"type":37,"value":624},"from",{"type":32,"tag":84,"props":626,"children":627},{"style":91},[628],{"type":37,"value":629}," langsmith ",{"type":32,"tag":84,"props":631,"children":632},{"style":621},[633],{"type":37,"value":634},"import",{"type":32,"tag":84,"props":636,"children":637},{"style":91},[638],{"type":37,"value":639}," Client\n",{"type":32,"tag":84,"props":641,"children":642},{"class":86,"line":97},[643,648,653,658,664,668,673],{"type":32,"tag":84,"props":644,"children":645},{"style":91},[646],{"type":37,"value":647},"client ",{"type":32,"tag":84,"props":649,"children":650},{"style":621},[651],{"type":37,"value":652},"=",{"type":32,"tag":84,"props":654,"children":655},{"style":91},[656],{"type":37,"value":657}," Client(",{"type":32,"tag":84,"props":659,"children":661},{"style":660},"--shiki-default:#FFAB70",[662],{"type":37,"value":663},"api_key",{"type":32,"tag":84,"props":665,"children":666},{"style":621},[667],{"type":37,"value":652},{"type":32,"tag":84,"props":669,"children":670},{"style":121},[671],{"type":37,"value":672},"\"...\"",{"type":32,"tag":84,"props":674,"children":675},{"style":91},[676],{"type":37,"value":677},")\n",{"type":32,"tag":84,"props":679,"children":680},{"class":86,"line":106},[681],{"type":32,"tag":84,"props":682,"children":684},{"emptyLinePlaceholder":683},true,[685],{"type":37,"value":686},"\n",{"type":32,"tag":84,"props":688,"children":689},{"class":86,"line":132},[690,695,700,705,709,714],{"type":32,"tag":84,"props":691,"children":692},{"style":621},[693],{"type":37,"value":694},"with",{"type":32,"tag":84,"props":696,"children":697},{"style":91},[698],{"type":37,"value":699}," client.trace(",{"type":32,"tag":84,"props":701,"children":702},{"style":660},[703],{"type":37,"value":704},"name",{"type":32,"tag":84,"props":706,"children":707},{"style":621},[708],{"type":37,"value":652},{"type":32,"tag":84,"props":710,"children":711},{"style":121},[712],{"type":37,"value":713},"\"summarize_revenue\"",{"type":32,"tag":84,"props":715,"children":716},{"style":91},[717],{"type":37,"value":718},"):\n",{"type":32,"tag":84,"props":720,"children":721},{"class":86,"line":175},[722,727,731],{"type":32,"tag":84,"props":723,"children":724},{"style":91},[725],{"type":37,"value":726},"    result ",{"type":32,"tag":84,"props":728,"children":729},{"style":621},[730],{"type":37,"value":652},{"type":32,"tag":84,"props":732,"children":733},{"style":91},[734],{"type":37,"value":735}," llm.invoke(prompt)\n",{"type":32,"tag":84,"props":737,"children":738},{"class":86,"line":193},[739,744,749],{"type":32,"tag":84,"props":740,"children":741},{"style":91},[742],{"type":37,"value":743},"    client.log_metric(",{"type":32,"tag":84,"props":745,"children":746},{"style":121},[747],{"type":37,"value":748},"\"token_count\"",{"type":32,"tag":84,"props":750,"children":751},{"style":91},[752],{"type":37,"value":753},", result.usage.total_tokens)\n",{"type":32,"tag":33,"props":755,"children":756},{},[757],{"type":37,"value":758},"Каждый trace видно в UI LangSmith, полностью залогированы input\u002Foutput\u002Fметаданные. Если версий prompt'а несколько, можно открыть view для сравнения. В интерфейсе увидите такое: \"prompt v2 производит на 8% длиннее output, чем v1, но latency на 3% ниже\".",{"type":32,"tag":33,"props":760,"children":761},{},[762],{"type":37,"value":763},"LangSmith предоставляет playground — меняете prompt и одной кнопкой тестируете на множестве input'ов. Это даёт быструю feedback-петлю и для прототипирования, и для regression-тестов. Но помните: playground — первый фильтр, он не заменяет production A\u002FB-тест.",{"type":32,"tag":40,"props":765,"children":767},{"id":766},"второй-эффект-версионирования-promptов-rollback",[768],{"type":37,"value":769},"Второй эффект версионирования prompt'ов: rollback",{"type":32,"tag":33,"props":771,"children":772},{},[773],{"type":37,"value":774},"Когда развёртывание выходит из строя, нужен rollback. В LLM-операциях rollback — это возврат к предыдущей версии prompt'а. Чтобы это работало, нужна история версий.",{"type":32,"tag":33,"props":776,"children":777},{},[778,780,786],{"type":37,"value":779},"Простой подход: хранить каждый prompt в отдельном файле git'а (",{"type":32,"tag":80,"props":781,"children":783},{"className":782},[],[784],{"type":37,"value":785},"prompts\u002Fsummarization_v3.txt",{"type":37,"value":787},"). Скрипт развёртывания хранит текущую версию в конфиг-файле:",{"type":32,"tag":73,"props":789,"children":793},{"code":790,"language":791,"meta":16,"className":792,"style":16},"# config\u002Fproduction.yaml\nprompts:\n  summarization: v3\n  classification: v2\n","yaml","language-yaml shiki shiki-themes github-dark",[794],{"type":32,"tag":80,"props":795,"children":796},{"__ignoreMap":16},[797,806,820,837],{"type":32,"tag":84,"props":798,"children":799},{"class":86,"line":87},[800],{"type":32,"tag":84,"props":801,"children":803},{"style":802},"--shiki-default:#6A737D",[804],{"type":37,"value":805},"# config\u002Fproduction.yaml\n",{"type":32,"tag":84,"props":807,"children":808},{"class":86,"line":97},[809,815],{"type":32,"tag":84,"props":810,"children":812},{"style":811},"--shiki-default:#85E89D",[813],{"type":37,"value":814},"prompts",{"type":32,"tag":84,"props":816,"children":817},{"style":91},[818],{"type":37,"value":819},":\n",{"type":32,"tag":84,"props":821,"children":822},{"class":86,"line":106},[823,828,832],{"type":32,"tag":84,"props":824,"children":825},{"style":811},[826],{"type":37,"value":827},"  summarization",{"type":32,"tag":84,"props":829,"children":830},{"style":91},[831],{"type":37,"value":118},{"type":32,"tag":84,"props":833,"children":834},{"style":121},[835],{"type":37,"value":836},"v3\n",{"type":32,"tag":84,"props":838,"children":839},{"class":86,"line":132},[840,845,849],{"type":32,"tag":84,"props":841,"children":842},{"style":811},[843],{"type":37,"value":844},"  classification",{"type":32,"tag":84,"props":846,"children":847},{"style":91},[848],{"type":37,"value":118},{"type":32,"tag":84,"props":850,"children":851},{"style":121},[852],{"type":37,"value":853},"v2\n",{"type":32,"tag":33,"props":855,"children":856},{},[857,859,865],{"type":37,"value":858},"Для rollback'а меняете ",{"type":32,"tag":80,"props":860,"children":862},{"className":861},[],[863],{"type":37,"value":864},"summarization: v2",{"type":37,"value":866}," и запускаете развёртывание. Но это ручной процесс, при инциденте медленно. Продвинутый подход: feature flag система (LaunchDarkly, Unleash). Флаг позволяет менять версию prompt'а в runtime, без переразвёртывания кода.",{"type":32,"tag":33,"props":868,"children":869},{},[870,872,879],{"type":37,"value":871},"Здесь вступают в силу практики ",{"type":32,"tag":485,"props":873,"children":876},{"href":874,"rel":875},"https:\u002F\u002Fwww.roibase.com.tr\u002Fru\u002Ffirstparty",[489],[877],{"type":37,"value":878},"First-Party Veri & Ölçüm Mimarisi",{"type":37,"value":880}," Roibase — нужно связать изменение prompt'а с downstream event'ами (conversion, churn), чтобы решение о rollback'е было на числовой основе. Если через 6 часов после развёртывания нового prompt'а churn выросла на 4% — это сигнал откатить prompt.",{"type":32,"tag":40,"props":882,"children":884},{"id":883},"edge-case-версионирование-promptов-в-мультиязычной-среде",[885],{"type":37,"value":886},"Edge case: версионирование prompt'ов в мультиязычной среде",{"type":32,"tag":33,"props":888,"children":889},{},[890],{"type":37,"value":891},"Если ваше LLM-приложение работает на нескольких языках (TR, EN, DE), для каждого языка нужна своя версия prompt'а. Потому что prompt, хороший для английского, может не дать тот же тон на турецком.",{"type":32,"tag":33,"props":893,"children":894},{},[895],{"type":37,"value":896},"Решение: организуйте prompt-файлы по языкам:",{"type":32,"tag":73,"props":898,"children":900},{"code":899},"prompts\u002F\n  summarization\u002F\n    en_v3.txt\n    tr_v3.txt\n    de_v3.txt\n",[901],{"type":32,"tag":80,"props":902,"children":903},{"__ignoreMap":16},[904],{"type":37,"value":899},{"type":32,"tag":33,"props":906,"children":907},{},[908],{"type":37,"value":909},"Eval set тоже должен быть язык-специфичным — в турецких test case'ах ожидайте турецкий output. A\u002FB-тест запускайте отдельно по языкам, потому что поведение пользователя турецкий ≠ английский. При агрегировании метрик не забывайте language segment.",{"type":32,"tag":33,"props":911,"children":912},{},[913],{"type":37,"value":914},"Ещё один момент: в мультиязычном prompt'е длина контекста варьируется — турецкое предложение в среднем на 12% длиннее (в токенах). Это риск выхода за token limit. Добавьте в evaluation pipeline проверку token count и выдавайте warning при превышении threshold.",{"type":32,"tag":40,"props":916,"children":918},{"id":917},"на-практике-установите-первый-eval-set",[919],{"type":37,"value":920},"На практике: установите первый eval set",{"type":32,"tag":33,"props":922,"children":923},{},[924],{"type":37,"value":925},"Чтобы начать с описанной системы, первый шаг — minimal eval set из 20–30 реальных запросов пользователей. Откройте production логи, выберите самые частые запросы, для каждого определите ожидаемые свойства output'а (точность, тон, длину).",{"type":32,"tag":33,"props":927,"children":928},{},[929],{"type":37,"value":930},"Потом установите Promptfoo или LangSmith, запустите текущий prompt'ом на этом set'е, получите baseline score. Теперь сделайте маленькое изменение prompt'а (например, добавьте \"отвечай кратко и чётко\"), запустите eval снова, сравните score'ы. Если regression меньше 5% — развёртывайте.",{"type":32,"tag":33,"props":932,"children":933},{},[934],{"type":37,"value":935},"Когда цикл автоматизируется, скорость iteration по prompt'ам вырастает в 3 раза. Потому что теперь на вопрос \"хорошо ли это изменение?\" отвечаете не интуицией, а числами.",{"type":32,"tag":937,"props":938,"children":939},"style",{},[940],{"type":37,"value":941},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":16,"searchDepth":106,"depth":106,"links":943},[944,945,946,947,948,949,950],{"id":42,"depth":97,"text":45},{"id":63,"depth":97,"text":66},{"id":419,"depth":97,"text":422},{"id":593,"depth":97,"text":596},{"id":766,"depth":97,"text":769},{"id":883,"depth":97,"text":886},{"id":917,"depth":97,"text":920},"markdown","content:ru:ai:versiyonlama-a-b-test-llm-operatsii.md","content","ru\u002Fai\u002Fversiyonlama-a-b-test-llm-operatsii.md","ru\u002Fai\u002Fversiyonlama-a-b-test-llm-operatsii","md",1785967503393]