Most startups don't A/B test. Not because they don't want to, but because it feels complicated. You need statistics knowledge, tracking infrastructure, experiment frameworks, and someone to interpret the results. So instead, you just ship and hope your feature is "better" than what you're replacing. I'm not claiming to know everything, but here's what I learned about the engineering side of A/B testing over the years.

![You after reading this post](https://i.imgflip.com/aj9qde.jpg)

![Graph showing conversion metrics (hopefully going up)](https://images.unsplash.com/photo-1460925895917-afdab827c52f?w=800&h=600&fit=crop)

## What A/B Testing Actually Is You have two versions of something: - **A (Control):** The current version - **B (Variant):** The new version You show A to 50% of users, B to 50% of users, and measure which performs better. **That's it. Everything else is just details.** ## Why Startups Need A/B Testing

Without A/B testing:

With A/B testing:

**You stop guessing. You know.** ## The Simple Implementation You don't need a system like LaunchDarkly, PostHog or Optimizely to get started with A/B testing. Here's the basics: ### 1. User Bucketing (Assignment) You might already have a user ID if your users log in, but if not you'll need to create a "stable ID". A key that you store (cookies, localStorage etc) for each unique user that you can use to do the bucketing against. A UUID works well for this. Give each user a consistent experiment assignment. ```typescript // Simple hash-based bucketing function getExperimentVariant(userId: string, experimentName: string): 'A' | 'B' { // Hash user ID + experiment name const hash = simpleHash(`${userId}_${experimentName}`) // Return A or B based on hash return hash % 2 === 0 ? 'A' : 'B' } function simpleHash(str: string): number { let hash = 0 for (let i = 0; i < str.length; i++) { const char = str.charCodeAt(i) hash = ((hash << 5) - hash) + char hash = hash & hash // Convert to 32-bit integer } return Math.abs(hash) } ``` **Why hash instead of random?** - Same user always gets same variant - No database needed - Works across devices/sessions - Deterministic (can replay behavior)

### 2. Feature Flags Wrap your experiment in a feature flag: This is where you add the control flow logic and actually display the different experience to the user. ```typescript // components/signup-button.tsx export function SignupButton() { const variant = getExperimentVariant(userId, 'button-size') if (variant === 'B') { // Variant B: Big button return ( ) } // Control: Normal button return ( ) } ```

Feature Flags ≠ A/B Testing

People often confuse these. Here's the difference:

Feature Flags:

Control who sees what. A deployment mechanism. Turn features on/off for specific users, teams, or percentages. No measurement, no statistics.

if (featureEnabled('new-design')) → show new design

A/B Testing:

Measure which version performs better. Includes random assignment, event tracking, and statistical analysis to determine a winner.

variant = getVariant() → track events → analyze results → ship winner

How they work together:

Feature flags implement the variants. A/B testing measures which variant wins. After you run an experiment and find a winner, you use the same feature flag to roll out the winning variant to 100% of users.

### 3. Event Tracking Track what users do in each variant: - exposure - assingment ```typescript // Track when button is shown trackEvent('button_shown', { experiment: 'button-size', variant: variant, userId: userId }) // Track when user clicks trackEvent('button_clicked', { experiment: 'button-size', variant: variant, userId: userId }) // Track when user completes signup trackEvent('signup_completed', { experiment: 'button-size', variant: variant, userId: userId }) ``` ### 4. Calculate Results ```sql -- Get conversion rates for each variant SELECT variant, COUNT(DISTINCT user_id) as users, COUNT(DISTINCT CASE WHEN event = 'signup_completed' THEN user_id END) as conversions, ROUND( COUNT(DISTINCT CASE WHEN event = 'signup_completed' THEN user_id END)::decimal / COUNT(DISTINCT user_id) * 100, 2 ) as conversion_rate FROM events WHERE experiment = 'button-size' GROUP BY variant ``` **Results:** ```plaintext Variant A: 1000 users, 45 signups (4.5%) Variant B: 1000 users, 53 signups (5.3%) Difference: +0.8 percentage points Relative lift: +17.8% ``` Is this significant? Did we move the needle? That's where the stats come in. ## The Statistics Part (Made Simple) Okay, this is where most people get scared. But honestly? You only need to understand two things: 1. **How many users do I need?** (Sample size) 2. **Are my results real or just luck?** (Statistical significance - **statsig**) That's it. Let's break it down. ### Sample Size: How Many Users Do I Need? The frustrating truth: **it depends on how big your change is.** Testing a huge improvement (like 50% lift)? You only need a few hundred users. Testing a small improvement (like 5% lift)? You might need 10,000+ users per variant. Here's why: small changes are hard to see through the noise. It's like trying to hear someone whisper in a crowded bar vs hearing them shout. The whisper needs more time (users) to be sure you heard it right. **The rough math:** - Tiny lift (5-10%): Need ~5,000 users per variant 😅 - Small lift (10-20%): Need ~1,000 users per variant - Medium lift (20-30%): Need ~400 users per variant - Big lift (30%+): Need ~200 users per variant 🎉 Most real-world tests fall in the 10-20% range. If you're seeing 50%+ lifts, either you found gold or something's broken.

### Statistical Significance: Is This Real or Just Luck? You run your test. Variant B wins by 8%. Ship it right? **Not so fast.** Maybe Variant B just got lucky. Maybe Tuesday's traffic is weird. Maybe it's a fluke. This is where the **p-value** comes in. It answers one question: > "If there was actually no difference between A and B, what's the chance I'd see results this extreme just by random luck?" **p < 0.05** means less than 5% chance it's luck → **Ship it** ✅ **p > 0.05** means it could easily be luck → **Keep testing** ⏳ Think of it like this: you flip a coin 10 times and get 7 heads. Is it a rigged coin? Probably not—that happens by chance all the time. But if you flip it 1,000 times and get 700 heads? Yeah, that coin is sus. Same idea with A/B testing. Small differences need big samples to be believable.

**Real talk:** Don't stress too much about the math. Use a calculator (like the one above) and follow these rules: - ✅ Decide your sample size upfront - ✅ Run the test for at least 2 weeks - ✅ Wait until you hit your sample size - ✅ Check if p < 0.05 - ✅ If yes, ship it. If no, keep testing or try a bigger change. **Don't:**

Warning: p-hacking

Don't stop the test when you see p < 0.05. That's p-hacking.

❌ Check every day, stop when p < 0.05

❌ Stop when variant B is winning

❌ Add more users until you get significance

✅ Decide sample size upfront, wait until you hit it

## What to A/B Test **High-impact areas:** ### 1. Pricing Page ```typescript // Test: Monthly vs Annual pricing emphasis const variant = getVariant(userId, 'pricing-display') ``` **What to measure:** Conversion to paid ### 2. Signup Flow ```typescript // Test: Single-step vs multi-step signup const variant = getVariant(userId, 'signup-flow') {variant === 'B' ? ( ) : ( )} ``` **What to measure:** Signup completion rate ### 3. Onboarding ```typescript // Test: Interactive tutorial vs skip to app const variant = getVariant(userId, 'onboarding') {variant === 'B' ? ( ) : ( )} ``` **What to measure:** Feature adoption, time to first value ### 4. Email Subject Lines ```typescript // Test: Different email subjects const variant = getVariant(userId, 'welcome-email') const subject = variant === 'B' ? "Your account is ready 🎉" : "Welcome to [Product]" ``` **What to measure:** Open rate, click rate ### 5. Call-to-Action Copy ```typescript // Test: Button text const buttonText = variant === 'B' ? "Start free trial" : "Get started" ``` **What to measure:** Click-through rate ## Real Example: Button Color **Hypothesis:** Green button will convert better than blue **Implementation:** ```tsx export function CTAButton() { const variant = getVariant(userId, 'cta-color') const color = variant === 'B' ? 'green' : 'blue' return ( ) } ``` **Results after 2 weeks:**

Variant	Users	Clicks	CTR
A (Blue)	5,243	287	5.5%
B (Green)	5,198	343	6.6%

Lift: +1.1 percentage points (+20% relative)

p-value: 0.012 → Significant! ✓

**Decision:** Ship green button to everyone. ## Common Mistakes ### 1. Testing Too Many Things at Once ```typescript // Bad: Can't tell which change caused the difference const variant = getVariant(userId, 'homepage-redesign') {variant === 'B' && ( )} ``` **Fix:** Test one change at a time. ### 2. Not Waiting Long Enough ```plaintext Day 1: Variant B is winning! Ship it! Day 7: Actually variant A is better ``` **Fix:** Run tests for at least 2 weeks or until you hit sample size. ### 3. Testing Things That Don't Matter ```typescript // Testing button corner radius const borderRadius = variant === 'B' ? '8px' : '6px' ``` Nobody cares. Test things that affect conversion, not aesthetics. ### 4. Ignoring Seasonal Effects Running tests during Black Friday, holidays, or product launches skews results. **Fix:** Avoid testing during unusual periods. ### 5. Looking at Results Too Early Every test will show noise early. Don't peek. **Fix:** Decide sample size upfront. Wait until you hit it. ## When NOT to A/B Test **Don't test:** - ❌ Things with no users yet (need traffic) - ❌ Bug fixes (just fix bugs) - ❌ Obvious improvements (don't test "fix broken link") - ❌ Brand changes (some things aren't about metrics) - ❌ Things you can't measure (subjective "feels better") **Just ship:** - Performance improvements - Accessibility fixes - Security updates - Bug fixes ## Tools **DIY (what I use):** ```typescript // Feature flag + event tracking // Store results in Postgres // Analyze with SQL // Total cost: $0 ``` **SaaS Tools:** - **PostHog** - Free tier, open source, full product analytics - **GrowthBook** - Open source, self-hostable - **Optimizely** - Enterprise, expensive, powerful - **LaunchDarkly** - Feature flags + experiments **Start with DIY. Upgrade when it hurts.** ## My A/B Testing Checklist Before starting an experiment: ```markdown - [ ] Clear hypothesis (X will increase Y by Z%) - [ ] Defined success metric - [ ] Calculated sample size needed - [ ] Set test duration (minimum 2 weeks) - [ ] Tracking events implemented - [ ] Variant code written - [ ] QA'd both variants - [ ] Documented experiment ``` After running experiment: ```markdown - [ ] Hit sample size or time limit - [ ] Calculated statistical significance - [ ] Checked for bugs in tracking - [ ] Looked at secondary metrics - [ ] Documented results - [ ] Made ship/no-ship decision ``` ## The Uncomfortable Truth Most A/B tests fail. Like 70-80% fail. **Your brilliant idea?** Probably won't work. **Your designer's mockup?** Probably won't convert better. **That thing you saw work for Stripe?** Probably won't work for you. **That's the point.** A/B testing saves you from shipping bad ideas. ## Example Framework Here's a simple experiment framework you can use: ```typescript // lib/experiments.ts export type Experiment = { name: string variants: ['A', 'B'] active: boolean traffic: number // 0-1 (percentage to include) } export const experiments: Record = { 'button-size': { name: 'Button Size Test', variants: ['A', 'B'], active: true, traffic: 1.0 // 100% of users }, 'pricing-display': { name: 'Pricing Display', variants: ['A', 'B'], active: false, traffic: 0.5 // 50% of users } } export function getVariant( userId: string, experimentKey: string ): 'A' | 'B' | null { const experiment = experiments[experimentKey] if (!experiment || !experiment.active) { return null // Experiment not running } // Check if user is in experiment (traffic %) const hash = simpleHash(userId) if ((hash % 100) / 100 > experiment.traffic) { return null // User not in experiment } // Return variant return simpleHash(`${userId}_${experimentKey}`) % 2 === 0 ? 'A' : 'B' } // Usage in components const variant = getVariant(userId, 'button-size') if (variant === 'B') { // Show variant B } ``` ## Framework-Specific Considerations One thing I've glossed over: **how rendering contexts affect A/B testing implementation.** If you're working with modern frameworks like Next.js, you'll need to think about where and when variant assignment happens. Server-side rendering, static generation, and client-side hydration each have different implications for consistency, performance, and user experience. This gets complex quickly—middleware timing, edge runtime limitations, the SSG vs SSR tradeoff—so I wrote a separate deep dive on implementing A/B tests specifically in Next.js contexts. **Read the follow-up:** [A/B Testing in Next.js: Rendering Contexts](/writing/2026-02-08-ab-testing-nextjs-rendering-contexts) For most traditional SPAs or client-rendered apps, the bucketing and tracking patterns I showed earlier are all you need. ## A/B Testing Glossary Quick reference for common terms you'll encounter: **Control (Variant A)** The current/baseline version. What users see today. **Variant (Variant B)** The new version you're testing. The challenger. **P-value** Probability that your results are due to random chance. p < 0.05 means less than 5% chance it's random. **Statistical Significance** When your p-value is low enough (usually < 0.05) to trust the results aren't random. **Sample Size** Number of users per variant needed to detect a difference. Small changes need big samples. **Conversion Rate** Percentage of users who complete your target action (signup, purchase, click, etc.). **Lift** The improvement you measured. Can be absolute (+0.8 percentage points) or relative (+17.8%). **Confidence Interval** Range where the true effect likely falls. "5.5% conversion with 95% CI of [4.8%, 6.2%]" **Type I Error (False Positive)** Saying there's a difference when there isn't. The p-value threshold controls this. **Type II Error (False Negative)** Missing a real difference because your sample was too small. **Statistical Power** Probability you'll detect a real difference. Aim for 80% power (you'll catch real effects 4 out of 5 times). **Minimum Detectable Effect (MDE)** Smallest change your test can reliably detect with your sample size. **Bucketing/Assignment** How you divide users into variants. Use consistent hashing, not random. **P-hacking** Manipulating data or stopping rules to get p < 0.05. Don't peek early and stop when it's significant. **Novelty Effect** Temporary boost from something being new. Run tests long enough (2+ weeks) to let it wear off. **Winner's Curse** Tendency to overestimate the true effect. The measured lift is usually higher than the long-term reality. **Multi-armed Bandit** Adaptive approach that sends more traffic to better-performing variants. More complex than A/B testing. **Sequential Testing** Advanced method that lets you stop early when you have strong evidence. Requires special statistical methods. ## The Bottom Line **A/B testing is:** - Simple to implement (feature flag + event tracking) - Hard to interpret (need statistics) - Worth doing (stops you shipping bad ideas) **Start small:** 1. Pick one high-impact thing to test 2. Implement basic bucketing + tracking 3. Run experiment for 2 weeks 4. Ship winner **Don't overthink it.** The best time to start A/B testing was last year. The second best time is now. --- **Resources:** - [Evan Miller's A/B Test Calculator](https://www.evanmiller.org/ab-testing/) - [GrowthBook (Open Source)](https://www.growthbook.io/) - [PostHog Experiments Docs](https://posthog.com/docs/experiments) - [Trust in Numbers by Ronny Kohavi](https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264) Ship data-driven changes, not opinions.

Everything I Know About: A/B Testing for StartupEngineers