[{"data":1,"prerenderedAt":1201},["ShallowReactive",2],{"article-alternates":3,"article-\u002Fen\u002Fmarketing\u002Ffast-decision-making-with-bayesian-ab-tests":13},{"i18nKey":4,"paths":5},"marketing-002-2026-07",{"de":6,"en":7,"es":8,"fr":9,"it":10,"ru":11,"tr":12},"\u002Fde\u002Fmarketing\u002Fbayesian-ab-test-schnelle-entscheidungsfindung","\u002Fen\u002Fmarketing\u002Ffast-decision-making-with-bayesian-ab-tests","\u002Fes\u002Fmarketing\u002Fprueba-bayesiana-ab-toma-rapida-decisiones","\u002Ffr\u002Fmarketing\u002Ftest-bayesien-prise-de-decision-rapide","\u002Fit\u002Fmarketing\u002Fdecisione-veloce-test-bayesiano-ab","\u002Fru\u002Fmarketing\u002Fbayesian-ab-testirovanie-bystrye-resheniya","\u002Ftr\u002Fmarketing\u002Fbayesian-a-b-test-ile-hizli-karar-verme",{"_path":7,"_dir":14,"_draft":15,"_partial":15,"_locale":16,"title":17,"description":18,"publishedAt":19,"modifiedAt":19,"category":14,"i18nKey":4,"tags":20,"readingTime":26,"author":27,"body":28,"_type":1195,"_id":1196,"_source":1197,"_file":1198,"_stem":1199,"_extension":1200},"marketing",false,"","Fast Decision-Making with Bayesian A\u002FB Tests","Skip the waiting period of frequentist tests. Use sequential testing, posterior probability, and dynamic sample sizing to accelerate A\u002FB tests by 3x.","2026-07-25",[21,22,23,24,25],"ab-testing","bayesian-statistics","conversion-optimization","statistical-inference","growth-engineering",7,"Roibase",{"type":29,"children":30,"toc":1183},"root",[31,39,46,51,56,61,67,72,77,87,140,145,152,157,162,195,211,217,222,227,235,240,261,267,272,277,283,288,1104,1109,1115,1120,1131,1141,1151,1156,1162,1167,1172,1177],{"type":32,"tag":33,"props":34,"children":35},"element","p",{},[36],{"type":37,"value":38},"text","If you want to gain speed in performance marketing, you might be running A\u002FB tests using the wrong methodology. Classic frequentist tests operate on fixed sample size and fixed-horizon logic: you launch a test, wait 2–4 weeks, and don't touch anything until the p-value crosses your threshold. Even if the winning variant is obvious halfway through, you cannot make a decision. Bayesian approaches change this critical point: with posterior probability, you can evaluate a decision at any moment, run sequential testing, and keep sample size dynamic. Google Optimize shutting down its Bayesian engine didn't kill this method—it opened the door to integrating it into your own stack.",{"type":32,"tag":40,"props":41,"children":43},"h2",{"id":42},"the-time-trap-of-frequentist-testing",[44],{"type":37,"value":45},"The time trap of frequentist testing",{"type":32,"tag":33,"props":47,"children":48},{},[49],{"type":37,"value":50},"Classic A\u002FB testing logic operates on a core assumption: the test must continue until the p-value drops below 0.05; if you peek at intermediate results, false positive risk increases. This is theoretically sound but creates two practical problems. First: if you want to stop early, you lack a statistical guardrail against making the wrong call. Second: even if the winning variant becomes obvious early, you must wait for the fixed sample size to complete—a timeframe that typically ranges from 14 to 21 days.",{"type":32,"tag":33,"props":52,"children":53},{},[54],{"type":37,"value":55},"This approach rests on the Neyman-Pearson hypothesis testing framework: you make a decision to reject or accept the null hypothesis using a single threshold, usually α = 0.05. The problem: this threshold is tied to fixed sample size calculations, so it prevents you from making dynamic decisions during the test. For example, if variant B shows 18% conversion while control sits at 12%, and this gap appears after just 500 users, the frequentist framework says \"wait more—you haven't reached your planned 2,000 users.\"",{"type":32,"tag":33,"props":57,"children":58},{},[59],{"type":37,"value":60},"Mobile app testing sharpens this issue. For an app with 5,000 daily active users, detecting a 2% uplift requires roughly 8,000 users in your sample—about 2 weeks. But if the winning signal emerges on day 3, you're sending traffic to the losing variant for another 11 days. Those 11 days represent lost revenue—opportunity cost in its purest form.",{"type":32,"tag":40,"props":62,"children":64},{"id":63},"the-bayesian-approach-continuous-updates-with-posterior-probability",[65],{"type":37,"value":66},"The Bayesian approach: continuous updates with posterior probability",{"type":32,"tag":33,"props":68,"children":69},{},[70],{"type":37,"value":71},"Bayesian statistics asks a different question: \"What is the probability that this variant outperforms the control?\" The answer is not a p-value but a posterior probability distribution. You update your prior belief with each new data point (each new user) and recalculate the posterior. This lets you say \"variant B has a 95% probability of higher conversion than control\"—and this statement permits sequential testing.",{"type":32,"tag":33,"props":73,"children":74},{},[75],{"type":37,"value":76},"Mathematically, Bayes' theorem operates via this formula:",{"type":32,"tag":78,"props":79,"children":81},"pre",{"code":80},"P(θ|data) = P(data|θ) × P(θ) \u002F P(data)\n",[82],{"type":32,"tag":83,"props":84,"children":85},"code",{"__ignoreMap":16},[86],{"type":37,"value":80},{"type":32,"tag":33,"props":88,"children":89},{},[90,92,98,100,106,108,114,116,122,124,130,132,138],{"type":37,"value":91},"Here, ",{"type":32,"tag":83,"props":93,"children":95},{"className":94},[],[96],{"type":37,"value":97},"θ",{"type":37,"value":99}," is the conversion rate, ",{"type":32,"tag":83,"props":101,"children":103},{"className":102},[],[104],{"type":37,"value":105},"P(θ)",{"type":37,"value":107}," is your prior (initial belief), ",{"type":32,"tag":83,"props":109,"children":111},{"className":110},[],[112],{"type":37,"value":113},"P(data|θ)",{"type":37,"value":115}," is the likelihood (probability of observed data under θ), and ",{"type":32,"tag":83,"props":117,"children":119},{"className":118},[],[120],{"type":37,"value":121},"P(θ|data)",{"type":37,"value":123}," is your posterior (updated belief). For instance, if you use Beta(1,1) as your prior—a uniform distribution—each conversion increments the ",{"type":32,"tag":83,"props":125,"children":127},{"className":126},[],[128],{"type":37,"value":129},"α",{"type":37,"value":131}," parameter by 1, and each bounce increments ",{"type":32,"tag":83,"props":133,"children":135},{"className":134},[],[136],{"type":37,"value":137},"β",{"type":37,"value":139}," by 1. With 100 visitors and 18 conversions, you get Beta(19, 83). Compare this posterior to your control group's posterior to calculate \"probability that B > A.\"",{"type":32,"tag":33,"props":141,"children":142},{},[143],{"type":37,"value":144},"Chris Stucchio's 2015 VWO article was among the first case studies bringing this logic to production: running the same test via Bayesian yields roughly 40% faster results on average because early stopping risk stays controlled. Google's internal experimentation framework began using Bayesian posteriors as an intermediate metric starting in 2018 (no public documentation, but Kohavi et al. reference it in their book).",{"type":32,"tag":146,"props":147,"children":149},"h3",{"id":148},"sequential-testing-and-stopping-rules",[150],{"type":37,"value":151},"Sequential testing and stopping rules",{"type":32,"tag":33,"props":153,"children":154},{},[155],{"type":37,"value":156},"The Bayesian approach's biggest strength is enabling sequential testing. In frequentist methodology, peeking at p-values during the test inflates Type I error (multiple comparison problem). In Bayesian, posterior probability is always a valid metric because it represents a continuously updated belief state. This lets you check \"posterior probability of B > A\" every day and stop the test once it exceeds 95%.",{"type":32,"tag":33,"props":158,"children":159},{},[160],{"type":37,"value":161},"A stopping rule works like this:",{"type":32,"tag":163,"props":164,"children":165},"ol",{},[166,172,177,190],{"type":32,"tag":167,"props":168,"children":169},"li",{},[170],{"type":37,"value":171},"Define a minimum sample size (e.g., 200 users per variant—to filter out early noise)",{"type":32,"tag":167,"props":173,"children":174},{},[175],{"type":37,"value":176},"Update posteriors daily",{"type":32,"tag":167,"props":178,"children":179},{},[180,182,188],{"type":37,"value":181},"When ",{"type":32,"tag":83,"props":183,"children":185},{"className":184},[],[186],{"type":37,"value":187},"P(variant_B > control) > 0.95",{"type":37,"value":189},", stop the test",{"type":32,"tag":167,"props":191,"children":192},{},[193],{"type":37,"value":194},"If after 14 days you haven't reached 95%, mark it \"inconclusive\"",{"type":32,"tag":33,"props":196,"children":197},{},[198,200,209],{"type":37,"value":199},"We use this approach in ",{"type":32,"tag":201,"props":202,"children":206},"a",{"href":203,"rel":204},"https:\u002F\u002Fwww.roibase.com.tr\u002Fen\u002Fcro",[205],"nofollow",[207],{"type":37,"value":208},"Conversion Rate Optimization",{"type":37,"value":210}," workflows: set priors at test launch, automate posterior updates daily, align the stopping rule threshold with engineering. For example, in e-commerce checkout flow tests, we use 98% instead of 95% because the cost of a false positive is high—payment page changes directly impact transaction volume.",{"type":32,"tag":40,"props":212,"children":214},{"id":213},"dynamic-sample-size-and-expected-loss-calculation",[215],{"type":37,"value":216},"Dynamic sample size and expected loss calculation",{"type":32,"tag":33,"props":218,"children":219},{},[220],{"type":37,"value":221},"In frequentist testing, sample size is computed upfront via power analysis: you specify minimum detectable effect (MDE), statistical power (80%), and significance level (α = 0.05), then wait for that number. In Bayesian, sample size is dynamic because posterior distribution can lead you to an early conclusion. But this does not mean \"stop whenever you want\"—the concept of expected loss enters the picture.",{"type":32,"tag":33,"props":223,"children":224},{},[225],{"type":37,"value":226},"Expected loss quantifies the cost of making the wrong decision. Say your posterior shows variant B winning with 92% probability. But there is an 8% chance A is actually better, and if you choose B, you miss a potential uplift. Expected loss makes this scenario numerical:",{"type":32,"tag":78,"props":228,"children":230},{"code":229},"E[Loss_B] = ∫ max(0, θ_A - θ_B) × P(θ_A, θ_B | data) dθ\n",[231],{"type":32,"tag":83,"props":232,"children":233},{"__ignoreMap":16},[234],{"type":37,"value":229},{"type":32,"tag":33,"props":236,"children":237},{},[238],{"type":37,"value":239},"In practical terms: \"If I pick B and I'm wrong, the expected loss is 0.3 percentage points in conversion rate.\" This translates to money—say 10,000 sessions daily, 0.3 pp loss = 30 lost conversions = multiply by average order value to get daily cost.",{"type":32,"tag":33,"props":241,"children":242},{},[243,245,251,253,259],{"type":37,"value":244},"Evan Miller's Bayesian A\u002FB Testing Calculator automates this: input conversion counts and sample sizes for control and variant, get posterior + expected loss + probability of being best. This tool is good for understanding the concept but insufficient for production. In production, we use Python's ",{"type":32,"tag":83,"props":246,"children":248},{"className":247},[],[249],{"type":37,"value":250},"pymc",{"type":37,"value":252}," or R's ",{"type":32,"tag":83,"props":254,"children":256},{"className":255},[],[257],{"type":37,"value":258},"rstan",{"type":37,"value":260}," to sample from posteriors and compute expected loss via Monte Carlo.",{"type":32,"tag":146,"props":262,"children":264},{"id":263},"regret-minimization-perspective",[265],{"type":37,"value":266},"Regret minimization perspective",{"type":32,"tag":33,"props":268,"children":269},{},[270],{"type":37,"value":271},"A concept borrowed from multi-armed bandit literature is regret. In A\u002FB testing, regret is the total loss from not selecting the optimal variant. Bayesian sequential testing minimizes this because winning signals lead to fast decisions. In frequentist, regret grows linearly over test duration (you keep sending traffic to the loser), while in Bayesian it grows sublinearly—you stop early and cut losses.",{"type":32,"tag":33,"props":273,"children":274},{},[275],{"type":37,"value":276},"Regret calculation is critical for e-commerce landing page tests. Say you have a 48-hour test window for a Black Friday campaign. Frequentist planning requires 2,000 users, and daily traffic is 3,000—you might not complete the test in time. With Bayesian, if you reach 97% posterior confidence in 12 hours, you can open the winning variant to 100% traffic for the remaining 36 hours and zero out regret.",{"type":32,"tag":40,"props":278,"children":280},{"id":279},"implementation-bayesian-ab-testing-pipeline-in-python",[281],{"type":37,"value":282},"Implementation: Bayesian A\u002FB testing pipeline in Python",{"type":32,"tag":33,"props":284,"children":285},{},[286],{"type":37,"value":287},"Moving from theory to practice, here's how to get Bayesian tests into production. This code snippet fetches test data from BigQuery, calculates posteriors, and checks stopping rules:",{"type":32,"tag":78,"props":289,"children":293},{"code":290,"language":291,"meta":16,"className":292,"style":16},"import numpy as np\nfrom scipy.stats import beta\n\ndef calculate_posterior(conversions, trials, prior_alpha=1, prior_beta=1):\n    \"\"\"Calculate posterior using Beta-Binomial conjugate prior\"\"\"\n    return beta(prior_alpha + conversions, prior_beta + trials - conversions)\n\ndef prob_b_beats_a(posterior_a, posterior_b, samples=100000):\n    \"\"\"Calculate P(B > A) via Monte Carlo\"\"\"\n    samples_a = posterior_a.rvs(samples)\n    samples_b = posterior_b.rvs(samples)\n    return (samples_b > samples_a).mean()\n\ndef expected_loss(posterior_a, posterior_b, samples=100000):\n    \"\"\"Expected loss if we choose B\"\"\"\n    samples_a = posterior_a.rvs(samples)\n    samples_b = posterior_b.rvs(samples)\n    loss = np.maximum(0, samples_a - samples_b)\n    return loss.mean()\n\n# Example data: Control 1000 sessions \u002F 120 conversions, Variant 1000 \u002F 145\nposterior_control = calculate_posterior(120, 1000)\nposterior_variant = calculate_posterior(145, 1000)\n\nprob_win = prob_b_beats_a(posterior_control, posterior_variant)\nloss_variant = expected_loss(posterior_control, posterior_variant)\n\nprint(f\"P(Variant > Control): {prob_win:.3f}\")\nprint(f\"Expected loss if choose Variant: {loss_variant:.4f}\")\n\n# Stopping rule\nif prob_win > 0.95 and loss_variant \u003C 0.01:\n    print(\"SHIP VARIANT\")\nelif prob_win \u003C 0.05:\n    print(\"SHIP CONTROL\")\nelse:\n    print(\"CONTINUE TEST\")\n","python","language-python shiki shiki-themes github-dark",[294],{"type":32,"tag":83,"props":295,"children":296},{"__ignoreMap":16},[297,325,348,358,407,417,460,467,498,507,525,543,566,574,603,612,628,644,681,694,702,712,750,784,792,810,828,836,889,936,944,953,1001,1023,1049,1070,1083],{"type":32,"tag":298,"props":299,"children":302},"span",{"class":300,"line":301},"line",1,[303,309,315,320],{"type":32,"tag":298,"props":304,"children":306},{"style":305},"--shiki-default:#F97583",[307],{"type":37,"value":308},"import",{"type":32,"tag":298,"props":310,"children":312},{"style":311},"--shiki-default:#E1E4E8",[313],{"type":37,"value":314}," numpy ",{"type":32,"tag":298,"props":316,"children":317},{"style":305},[318],{"type":37,"value":319},"as",{"type":32,"tag":298,"props":321,"children":322},{"style":311},[323],{"type":37,"value":324}," np\n",{"type":32,"tag":298,"props":326,"children":328},{"class":300,"line":327},2,[329,334,339,343],{"type":32,"tag":298,"props":330,"children":331},{"style":305},[332],{"type":37,"value":333},"from",{"type":32,"tag":298,"props":335,"children":336},{"style":311},[337],{"type":37,"value":338}," scipy.stats ",{"type":32,"tag":298,"props":340,"children":341},{"style":305},[342],{"type":37,"value":308},{"type":32,"tag":298,"props":344,"children":345},{"style":311},[346],{"type":37,"value":347}," beta\n",{"type":32,"tag":298,"props":349,"children":351},{"class":300,"line":350},3,[352],{"type":32,"tag":298,"props":353,"children":355},{"emptyLinePlaceholder":354},true,[356],{"type":37,"value":357},"\n",{"type":32,"tag":298,"props":359,"children":361},{"class":300,"line":360},4,[362,367,373,378,383,389,394,398,402],{"type":32,"tag":298,"props":363,"children":364},{"style":305},[365],{"type":37,"value":366},"def",{"type":32,"tag":298,"props":368,"children":370},{"style":369},"--shiki-default:#B392F0",[371],{"type":37,"value":372}," calculate_posterior",{"type":32,"tag":298,"props":374,"children":375},{"style":311},[376],{"type":37,"value":377},"(conversions, trials, prior_alpha",{"type":32,"tag":298,"props":379,"children":380},{"style":305},[381],{"type":37,"value":382},"=",{"type":32,"tag":298,"props":384,"children":386},{"style":385},"--shiki-default:#79B8FF",[387],{"type":37,"value":388},"1",{"type":32,"tag":298,"props":390,"children":391},{"style":311},[392],{"type":37,"value":393},", prior_beta",{"type":32,"tag":298,"props":395,"children":396},{"style":305},[397],{"type":37,"value":382},{"type":32,"tag":298,"props":399,"children":400},{"style":385},[401],{"type":37,"value":388},{"type":32,"tag":298,"props":403,"children":404},{"style":311},[405],{"type":37,"value":406},"):\n",{"type":32,"tag":298,"props":408,"children":410},{"class":300,"line":409},5,[411],{"type":32,"tag":298,"props":412,"children":414},{"style":413},"--shiki-default:#9ECBFF",[415],{"type":37,"value":416},"    \"\"\"Calculate posterior using Beta-Binomial conjugate prior\"\"\"\n",{"type":32,"tag":298,"props":418,"children":420},{"class":300,"line":419},6,[421,426,431,436,441,445,450,455],{"type":32,"tag":298,"props":422,"children":423},{"style":305},[424],{"type":37,"value":425},"    return",{"type":32,"tag":298,"props":427,"children":428},{"style":311},[429],{"type":37,"value":430}," beta(prior_alpha ",{"type":32,"tag":298,"props":432,"children":433},{"style":305},[434],{"type":37,"value":435},"+",{"type":32,"tag":298,"props":437,"children":438},{"style":311},[439],{"type":37,"value":440}," conversions, prior_beta ",{"type":32,"tag":298,"props":442,"children":443},{"style":305},[444],{"type":37,"value":435},{"type":32,"tag":298,"props":446,"children":447},{"style":311},[448],{"type":37,"value":449}," trials ",{"type":32,"tag":298,"props":451,"children":452},{"style":305},[453],{"type":37,"value":454},"-",{"type":32,"tag":298,"props":456,"children":457},{"style":311},[458],{"type":37,"value":459}," conversions)\n",{"type":32,"tag":298,"props":461,"children":462},{"class":300,"line":26},[463],{"type":32,"tag":298,"props":464,"children":465},{"emptyLinePlaceholder":354},[466],{"type":37,"value":357},{"type":32,"tag":298,"props":468,"children":470},{"class":300,"line":469},8,[471,475,480,485,489,494],{"type":32,"tag":298,"props":472,"children":473},{"style":305},[474],{"type":37,"value":366},{"type":32,"tag":298,"props":476,"children":477},{"style":369},[478],{"type":37,"value":479}," prob_b_beats_a",{"type":32,"tag":298,"props":481,"children":482},{"style":311},[483],{"type":37,"value":484},"(posterior_a, posterior_b, samples",{"type":32,"tag":298,"props":486,"children":487},{"style":305},[488],{"type":37,"value":382},{"type":32,"tag":298,"props":490,"children":491},{"style":385},[492],{"type":37,"value":493},"100000",{"type":32,"tag":298,"props":495,"children":496},{"style":311},[497],{"type":37,"value":406},{"type":32,"tag":298,"props":499,"children":501},{"class":300,"line":500},9,[502],{"type":32,"tag":298,"props":503,"children":504},{"style":413},[505],{"type":37,"value":506},"    \"\"\"Calculate P(B > A) via Monte Carlo\"\"\"\n",{"type":32,"tag":298,"props":508,"children":510},{"class":300,"line":509},10,[511,516,520],{"type":32,"tag":298,"props":512,"children":513},{"style":311},[514],{"type":37,"value":515},"    samples_a ",{"type":32,"tag":298,"props":517,"children":518},{"style":305},[519],{"type":37,"value":382},{"type":32,"tag":298,"props":521,"children":522},{"style":311},[523],{"type":37,"value":524}," posterior_a.rvs(samples)\n",{"type":32,"tag":298,"props":526,"children":528},{"class":300,"line":527},11,[529,534,538],{"type":32,"tag":298,"props":530,"children":531},{"style":311},[532],{"type":37,"value":533},"    samples_b ",{"type":32,"tag":298,"props":535,"children":536},{"style":305},[537],{"type":37,"value":382},{"type":32,"tag":298,"props":539,"children":540},{"style":311},[541],{"type":37,"value":542}," posterior_b.rvs(samples)\n",{"type":32,"tag":298,"props":544,"children":546},{"class":300,"line":545},12,[547,551,556,561],{"type":32,"tag":298,"props":548,"children":549},{"style":305},[550],{"type":37,"value":425},{"type":32,"tag":298,"props":552,"children":553},{"style":311},[554],{"type":37,"value":555}," (samples_b ",{"type":32,"tag":298,"props":557,"children":558},{"style":305},[559],{"type":37,"value":560},">",{"type":32,"tag":298,"props":562,"children":563},{"style":311},[564],{"type":37,"value":565}," samples_a).mean()\n",{"type":32,"tag":298,"props":567,"children":569},{"class":300,"line":568},13,[570],{"type":32,"tag":298,"props":571,"children":572},{"emptyLinePlaceholder":354},[573],{"type":37,"value":357},{"type":32,"tag":298,"props":575,"children":577},{"class":300,"line":576},14,[578,582,587,591,595,599],{"type":32,"tag":298,"props":579,"children":580},{"style":305},[581],{"type":37,"value":366},{"type":32,"tag":298,"props":583,"children":584},{"style":369},[585],{"type":37,"value":586}," expected_loss",{"type":32,"tag":298,"props":588,"children":589},{"style":311},[590],{"type":37,"value":484},{"type":32,"tag":298,"props":592,"children":593},{"style":305},[594],{"type":37,"value":382},{"type":32,"tag":298,"props":596,"children":597},{"style":385},[598],{"type":37,"value":493},{"type":32,"tag":298,"props":600,"children":601},{"style":311},[602],{"type":37,"value":406},{"type":32,"tag":298,"props":604,"children":606},{"class":300,"line":605},15,[607],{"type":32,"tag":298,"props":608,"children":609},{"style":413},[610],{"type":37,"value":611},"    \"\"\"Expected loss if we choose B\"\"\"\n",{"type":32,"tag":298,"props":613,"children":615},{"class":300,"line":614},16,[616,620,624],{"type":32,"tag":298,"props":617,"children":618},{"style":311},[619],{"type":37,"value":515},{"type":32,"tag":298,"props":621,"children":622},{"style":305},[623],{"type":37,"value":382},{"type":32,"tag":298,"props":625,"children":626},{"style":311},[627],{"type":37,"value":524},{"type":32,"tag":298,"props":629,"children":631},{"class":300,"line":630},17,[632,636,640],{"type":32,"tag":298,"props":633,"children":634},{"style":311},[635],{"type":37,"value":533},{"type":32,"tag":298,"props":637,"children":638},{"style":305},[639],{"type":37,"value":382},{"type":32,"tag":298,"props":641,"children":642},{"style":311},[643],{"type":37,"value":542},{"type":32,"tag":298,"props":645,"children":647},{"class":300,"line":646},18,[648,653,657,662,667,672,676],{"type":32,"tag":298,"props":649,"children":650},{"style":311},[651],{"type":37,"value":652},"    loss ",{"type":32,"tag":298,"props":654,"children":655},{"style":305},[656],{"type":37,"value":382},{"type":32,"tag":298,"props":658,"children":659},{"style":311},[660],{"type":37,"value":661}," np.maximum(",{"type":32,"tag":298,"props":663,"children":664},{"style":385},[665],{"type":37,"value":666},"0",{"type":32,"tag":298,"props":668,"children":669},{"style":311},[670],{"type":37,"value":671},", samples_a ",{"type":32,"tag":298,"props":673,"children":674},{"style":305},[675],{"type":37,"value":454},{"type":32,"tag":298,"props":677,"children":678},{"style":311},[679],{"type":37,"value":680}," samples_b)\n",{"type":32,"tag":298,"props":682,"children":684},{"class":300,"line":683},19,[685,689],{"type":32,"tag":298,"props":686,"children":687},{"style":305},[688],{"type":37,"value":425},{"type":32,"tag":298,"props":690,"children":691},{"style":311},[692],{"type":37,"value":693}," loss.mean()\n",{"type":32,"tag":298,"props":695,"children":697},{"class":300,"line":696},20,[698],{"type":32,"tag":298,"props":699,"children":700},{"emptyLinePlaceholder":354},[701],{"type":37,"value":357},{"type":32,"tag":298,"props":703,"children":705},{"class":300,"line":704},21,[706],{"type":32,"tag":298,"props":707,"children":709},{"style":708},"--shiki-default:#6A737D",[710],{"type":37,"value":711},"# Example data: Control 1000 sessions \u002F 120 conversions, Variant 1000 \u002F 145\n",{"type":32,"tag":298,"props":713,"children":715},{"class":300,"line":714},22,[716,721,725,730,735,740,745],{"type":32,"tag":298,"props":717,"children":718},{"style":311},[719],{"type":37,"value":720},"posterior_control ",{"type":32,"tag":298,"props":722,"children":723},{"style":305},[724],{"type":37,"value":382},{"type":32,"tag":298,"props":726,"children":727},{"style":311},[728],{"type":37,"value":729}," calculate_posterior(",{"type":32,"tag":298,"props":731,"children":732},{"style":385},[733],{"type":37,"value":734},"120",{"type":32,"tag":298,"props":736,"children":737},{"style":311},[738],{"type":37,"value":739},", ",{"type":32,"tag":298,"props":741,"children":742},{"style":385},[743],{"type":37,"value":744},"1000",{"type":32,"tag":298,"props":746,"children":747},{"style":311},[748],{"type":37,"value":749},")\n",{"type":32,"tag":298,"props":751,"children":753},{"class":300,"line":752},23,[754,759,763,767,772,776,780],{"type":32,"tag":298,"props":755,"children":756},{"style":311},[757],{"type":37,"value":758},"posterior_variant ",{"type":32,"tag":298,"props":760,"children":761},{"style":305},[762],{"type":37,"value":382},{"type":32,"tag":298,"props":764,"children":765},{"style":311},[766],{"type":37,"value":729},{"type":32,"tag":298,"props":768,"children":769},{"style":385},[770],{"type":37,"value":771},"145",{"type":32,"tag":298,"props":773,"children":774},{"style":311},[775],{"type":37,"value":739},{"type":32,"tag":298,"props":777,"children":778},{"style":385},[779],{"type":37,"value":744},{"type":32,"tag":298,"props":781,"children":782},{"style":311},[783],{"type":37,"value":749},{"type":32,"tag":298,"props":785,"children":787},{"class":300,"line":786},24,[788],{"type":32,"tag":298,"props":789,"children":790},{"emptyLinePlaceholder":354},[791],{"type":37,"value":357},{"type":32,"tag":298,"props":793,"children":795},{"class":300,"line":794},25,[796,801,805],{"type":32,"tag":298,"props":797,"children":798},{"style":311},[799],{"type":37,"value":800},"prob_win ",{"type":32,"tag":298,"props":802,"children":803},{"style":305},[804],{"type":37,"value":382},{"type":32,"tag":298,"props":806,"children":807},{"style":311},[808],{"type":37,"value":809}," prob_b_beats_a(posterior_control, posterior_variant)\n",{"type":32,"tag":298,"props":811,"children":813},{"class":300,"line":812},26,[814,819,823],{"type":32,"tag":298,"props":815,"children":816},{"style":311},[817],{"type":37,"value":818},"loss_variant ",{"type":32,"tag":298,"props":820,"children":821},{"style":305},[822],{"type":37,"value":382},{"type":32,"tag":298,"props":824,"children":825},{"style":311},[826],{"type":37,"value":827}," expected_loss(posterior_control, posterior_variant)\n",{"type":32,"tag":298,"props":829,"children":831},{"class":300,"line":830},27,[832],{"type":32,"tag":298,"props":833,"children":834},{"emptyLinePlaceholder":354},[835],{"type":37,"value":357},{"type":32,"tag":298,"props":837,"children":839},{"class":300,"line":838},28,[840,845,850,855,860,865,870,875,880,885],{"type":32,"tag":298,"props":841,"children":842},{"style":385},[843],{"type":37,"value":844},"print",{"type":32,"tag":298,"props":846,"children":847},{"style":311},[848],{"type":37,"value":849},"(",{"type":32,"tag":298,"props":851,"children":852},{"style":305},[853],{"type":37,"value":854},"f",{"type":32,"tag":298,"props":856,"children":857},{"style":413},[858],{"type":37,"value":859},"\"P(Variant > Control): ",{"type":32,"tag":298,"props":861,"children":862},{"style":385},[863],{"type":37,"value":864},"{",{"type":32,"tag":298,"props":866,"children":867},{"style":311},[868],{"type":37,"value":869},"prob_win",{"type":32,"tag":298,"props":871,"children":872},{"style":305},[873],{"type":37,"value":874},":.3f",{"type":32,"tag":298,"props":876,"children":877},{"style":385},[878],{"type":37,"value":879},"}",{"type":32,"tag":298,"props":881,"children":882},{"style":413},[883],{"type":37,"value":884},"\"",{"type":32,"tag":298,"props":886,"children":887},{"style":311},[888],{"type":37,"value":749},{"type":32,"tag":298,"props":890,"children":892},{"class":300,"line":891},29,[893,897,901,905,910,914,919,924,928,932],{"type":32,"tag":298,"props":894,"children":895},{"style":385},[896],{"type":37,"value":844},{"type":32,"tag":298,"props":898,"children":899},{"style":311},[900],{"type":37,"value":849},{"type":32,"tag":298,"props":902,"children":903},{"style":305},[904],{"type":37,"value":854},{"type":32,"tag":298,"props":906,"children":907},{"style":413},[908],{"type":37,"value":909},"\"Expected loss if choose Variant: ",{"type":32,"tag":298,"props":911,"children":912},{"style":385},[913],{"type":37,"value":864},{"type":32,"tag":298,"props":915,"children":916},{"style":311},[917],{"type":37,"value":918},"loss_variant",{"type":32,"tag":298,"props":920,"children":921},{"style":305},[922],{"type":37,"value":923},":.4f",{"type":32,"tag":298,"props":925,"children":926},{"style":385},[927],{"type":37,"value":879},{"type":32,"tag":298,"props":929,"children":930},{"style":413},[931],{"type":37,"value":884},{"type":32,"tag":298,"props":933,"children":934},{"style":311},[935],{"type":37,"value":749},{"type":32,"tag":298,"props":937,"children":939},{"class":300,"line":938},30,[940],{"type":32,"tag":298,"props":941,"children":942},{"emptyLinePlaceholder":354},[943],{"type":37,"value":357},{"type":32,"tag":298,"props":945,"children":947},{"class":300,"line":946},31,[948],{"type":32,"tag":298,"props":949,"children":950},{"style":708},[951],{"type":37,"value":952},"# Stopping rule\n",{"type":32,"tag":298,"props":954,"children":956},{"class":300,"line":955},32,[957,962,967,971,976,981,986,991,996],{"type":32,"tag":298,"props":958,"children":959},{"style":305},[960],{"type":37,"value":961},"if",{"type":32,"tag":298,"props":963,"children":964},{"style":311},[965],{"type":37,"value":966}," prob_win ",{"type":32,"tag":298,"props":968,"children":969},{"style":305},[970],{"type":37,"value":560},{"type":32,"tag":298,"props":972,"children":973},{"style":385},[974],{"type":37,"value":975}," 0.95",{"type":32,"tag":298,"props":977,"children":978},{"style":305},[979],{"type":37,"value":980}," and",{"type":32,"tag":298,"props":982,"children":983},{"style":311},[984],{"type":37,"value":985}," loss_variant ",{"type":32,"tag":298,"props":987,"children":988},{"style":305},[989],{"type":37,"value":990},"\u003C",{"type":32,"tag":298,"props":992,"children":993},{"style":385},[994],{"type":37,"value":995}," 0.01",{"type":32,"tag":298,"props":997,"children":998},{"style":311},[999],{"type":37,"value":1000},":\n",{"type":32,"tag":298,"props":1002,"children":1004},{"class":300,"line":1003},33,[1005,1010,1014,1019],{"type":32,"tag":298,"props":1006,"children":1007},{"style":385},[1008],{"type":37,"value":1009},"    print",{"type":32,"tag":298,"props":1011,"children":1012},{"style":311},[1013],{"type":37,"value":849},{"type":32,"tag":298,"props":1015,"children":1016},{"style":413},[1017],{"type":37,"value":1018},"\"SHIP VARIANT\"",{"type":32,"tag":298,"props":1020,"children":1021},{"style":311},[1022],{"type":37,"value":749},{"type":32,"tag":298,"props":1024,"children":1026},{"class":300,"line":1025},34,[1027,1032,1036,1040,1045],{"type":32,"tag":298,"props":1028,"children":1029},{"style":305},[1030],{"type":37,"value":1031},"elif",{"type":32,"tag":298,"props":1033,"children":1034},{"style":311},[1035],{"type":37,"value":966},{"type":32,"tag":298,"props":1037,"children":1038},{"style":305},[1039],{"type":37,"value":990},{"type":32,"tag":298,"props":1041,"children":1042},{"style":385},[1043],{"type":37,"value":1044}," 0.05",{"type":32,"tag":298,"props":1046,"children":1047},{"style":311},[1048],{"type":37,"value":1000},{"type":32,"tag":298,"props":1050,"children":1052},{"class":300,"line":1051},35,[1053,1057,1061,1066],{"type":32,"tag":298,"props":1054,"children":1055},{"style":385},[1056],{"type":37,"value":1009},{"type":32,"tag":298,"props":1058,"children":1059},{"style":311},[1060],{"type":37,"value":849},{"type":32,"tag":298,"props":1062,"children":1063},{"style":413},[1064],{"type":37,"value":1065},"\"SHIP CONTROL\"",{"type":32,"tag":298,"props":1067,"children":1068},{"style":311},[1069],{"type":37,"value":749},{"type":32,"tag":298,"props":1071,"children":1073},{"class":300,"line":1072},36,[1074,1079],{"type":32,"tag":298,"props":1075,"children":1076},{"style":305},[1077],{"type":37,"value":1078},"else",{"type":32,"tag":298,"props":1080,"children":1081},{"style":311},[1082],{"type":37,"value":1000},{"type":32,"tag":298,"props":1084,"children":1086},{"class":300,"line":1085},37,[1087,1091,1095,1100],{"type":32,"tag":298,"props":1088,"children":1089},{"style":385},[1090],{"type":37,"value":1009},{"type":32,"tag":298,"props":1092,"children":1093},{"style":311},[1094],{"type":37,"value":849},{"type":32,"tag":298,"props":1096,"children":1097},{"style":413},[1098],{"type":37,"value":1099},"\"CONTINUE TEST\"",{"type":32,"tag":298,"props":1101,"children":1102},{"style":311},[1103],{"type":37,"value":749},{"type":32,"tag":33,"props":1105,"children":1106},{},[1107],{"type":37,"value":1108},"You can embed this code in a dbt model and run it on a daily schedule. If you have a BigQuery table with test_id, variant, session_count, and conversion_count, use a Python UDF to calculate posteriors and write results to a new table. Connect it to a Looker or Metabase dashboard so the product team watches posterior graphs in real time.",{"type":32,"tag":40,"props":1110,"children":1112},{"id":1111},"trade-offs-and-when-to-stay-frequentist",[1113],{"type":37,"value":1114},"Trade-offs and when to stay frequentist",{"type":32,"tag":33,"props":1116,"children":1117},{},[1118],{"type":37,"value":1119},"Bayesian approaches are not universally superior. Three scenarios warrant staying frequentist:",{"type":32,"tag":33,"props":1121,"children":1122},{},[1123,1129],{"type":32,"tag":1124,"props":1125,"children":1126},"strong",{},[1127],{"type":37,"value":1128},"1. Tests requiring regulatory compliance:",{"type":37,"value":1130}," Pharmaceutical trials, financial services, and insurance pricing models have frequentist p-values as the regulatory standard (FDA, EMA). If you use Bayesian posteriors, you need additional documentation.",{"type":32,"tag":33,"props":1132,"children":1133},{},[1134,1139],{"type":32,"tag":1124,"props":1135,"children":1136},{},[1137],{"type":37,"value":1138},"2. Very low base rates:",{"type":37,"value":1140}," If conversion rate is 0.5% in a funnel step, Bayesian prior selection becomes critical. An uninformative prior (Beta(1,1)) struggles to separate signal from noise; an informative prior introduces subjective bias. Frequentist feels safer here.",{"type":32,"tag":33,"props":1142,"children":1143},{},[1144,1149],{"type":32,"tag":1124,"props":1145,"children":1146},{},[1147],{"type":37,"value":1148},"3. One-time, high-stakes campaigns:",{"type":37,"value":1150}," Annual Black Friday landing page tests that don't repeat. If you stop early via Bayesian and you're wrong, you cannot revert because the campaign is over. Conservative frequentist + Bonferroni correction might be preferred.",{"type":32,"tag":33,"props":1152,"children":1153},{},[1154],{"type":37,"value":1155},"Outside these exceptions—especially in SaaS, e-commerce, and mobile apps where continuous iteration is the norm—Bayesian's velocity gain is clear-cut. Netflix, Booking.com, and Spotify use Bayesian internally (they mention it in public tech blogs).",{"type":32,"tag":40,"props":1157,"children":1159},{"id":1158},"accelerating-decision-velocity",[1160],{"type":37,"value":1161},"Accelerating decision velocity",{"type":32,"tag":33,"props":1163,"children":1164},{},[1165],{"type":37,"value":1166},"Bayesian A\u002FB testing is not just a statistical change; it restructures your decision process. When posterior probability becomes a daily-updated metric, your test pipeline looks like this: you launch Monday, posterior hits 92% Wednesday, reaches 96% Thursday—you decide immediately. The same test under frequentist logic would run 2 weeks. Gaining 10 days = 10 days of faster iteration = 20–30 extra tests per year.",{"type":32,"tag":33,"props":1168,"children":1169},{},[1170],{"type":37,"value":1171},"To capture this speed advantage, build your tooling Bayesian-native: BigQuery + Python UDF + Looker dashboard + Slack alerts. Align your expected loss threshold with the CFO (e.g., 0.5% of daily revenue). Use domain knowledge when selecting priors but avoid overconfidence—Beta(2,2) works for most cases. Embed sequential testing into your product roadmap: if you launch 3 tests per sprint, Bayesian lets you close 2 mid-sprint and start new ones.",{"type":32,"tag":33,"props":1173,"children":1174},{},[1175],{"type":37,"value":1176},"In performance marketing, the winner moves fast. Bayesian methods give you that speed without sacrificing statistical rigor.",{"type":32,"tag":1178,"props":1179,"children":1180},"style",{},[1181],{"type":37,"value":1182},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":16,"searchDepth":350,"depth":350,"links":1184},[1185,1186,1189,1192,1193,1194],{"id":42,"depth":327,"text":45},{"id":63,"depth":327,"text":66,"children":1187},[1188],{"id":148,"depth":350,"text":151},{"id":213,"depth":327,"text":216,"children":1190},[1191],{"id":263,"depth":350,"text":266},{"id":279,"depth":327,"text":282},{"id":1111,"depth":327,"text":1114},{"id":1158,"depth":327,"text":1161},"markdown","content:en:marketing:fast-decision-making-with-bayesian-ab-tests.md","content","en\u002Fmarketing\u002Ffast-decision-making-with-bayesian-ab-tests.md","en\u002Fmarketing\u002Ffast-decision-making-with-bayesian-ab-tests","md",1785103515001]