Blog Posts

Author

Rohan Arora

Technically Correct Bayesian A/B Testing without Monte Carlo Simulations

Why estimating conversion rates is not the same as running an A/B Test
Product
Experimentation
AB Testing
Bayes Factor
Bayesian

We use a simple and tractable mathematical model that can be solved algaebrically. We demonstrate one key omission in the IBE approach: that of the alternative hypothesis. We also look at a sample-size imputation method to help with the process of building priors.

Nov 20, 2025
Rohan Arora

Running a classical Bayesian A/A Test, and paying for our priors.

A discussion on the fault in our Bayesian Stars through an A/A Test
Product
Experimentation
AB Testing
AA Testing
Bayesian

We look at how an A/A test seems to reveal some common flaws in implementation of the IBE Bayesian model. We also look at some poor decisions made at the time of selecting a prior in A/B Tests.

Nov 12, 2025
Rohan Arora

Is H0: 𝛿= |pb - pa| = 0 the same as H0: pb = pa?

A very public displpay of my ignorance while also discussing A/A tests
Product
Experimentation
AB Testing
AA Testing
Causal Inference

Is planning an A/B test the same as running an A/B Test? We try and figure out from the perspective of decisioning rules

Nov 10, 2025
Rohan Arora

Common-sense Bayesian A/B Tests are flawed

The beta-binomial model with independent CRO ratio is flawed
Product
Experimentation
AB Testing
Bayesian
Causal Inference

This is part 1 of a multi-part series. In this post we discuss the math, and in the next post we use code and visualisations for the exposition.

Nov 8, 2025
Rohan Arora

Building a rudimentary website with AI

My baby steps into AI-assisted coding.
AI
vibe-coding

I used lovable to build a simple react-based website.

Nov 3, 2025
Rohan Arora
No matching items