Search Marketing FAQ

Five criteria. Specific change (not 'improve the page' - 'add money-back guarantee text below CTA'). Specific metric (checkout completion rate, not 'conversion'). Direction predicted (increase or decrease). Research-backed reasoning (not opinion). Measurable within reasonable sample size and time. Good hypotheses prevent generic tests with diffuse results.

in CRO Fundamentals

Quantitative: numbers and statistics from analytics, A/B tests, heatmaps (what's happening). Qualitative: insights from user interviews, session recordings, surveys, usability tests (why it's happening). Both required - quantitative tells you where to look; qualitative tells you what to test. Skipping qualitative leads to data-rich but insight-poor CRO.

in CRO Fundamentals

Five criteria. Sample size reached. Statistical significance achieved (p < 0.05 typically). Effect size meaningful (not just statistically significant but practically significant). Guardrail metrics not regressed. Result holds across key segments (mobile + desktop, traffic sources). 'Success' = winner that holds up to scrutiny, not just one segment positive.

in CRO Fundamentals

Ongoing process. Conversion rates are not static - user expectations evolve, traffic mix changes, seasonality shifts, competitor activity affects baselines. Mature CRO programs run 4-12 tests per month continuously. One-time CRO projects produce one-time wins; ongoing programs compound year-over-year. Treat CRO as a function, not a project.

in CRO Fundamentals

A controlled experiment comparing two variants (A: control, B: variant) of a page or element. Visitors randomly assigned to one variant. Conversion rates compared with statistical methods to determine which version wins. The gold standard for testing causal impact on conversions. Underlying randomization removes selection bias - critical for confident conclusions.

in A/B Testing

When you have a hypothesis worth testing, sufficient traffic for statistical power, and a measurable conversion outcome. Use A/B testing for: copy changes, layout variations, CTA placement, pricing tests, form length, checkout flow. Don't A/B test if the change is obvious bug fix, small sample size (under 100 conversions/variant), or directional (no winner needed).

in A/B Testing

A/B: tests one variable with two options (e.g., red CTA vs green CTA). Multivariate (MVT): tests multiple variables simultaneously (e.g., 2 CTAs × 3 headlines = 6 variants). MVT requires much more traffic to achieve power. Use A/B for clear hypotheses; MVT only for high-traffic sites testing interaction effects. Most sites should default to A/B.

in A/B Testing

A/B: two variants compared. A/B/n: more than two variants (A/B/C/D/E). More variants split traffic into smaller buckets - each needs the full sample size for the same statistical power. A/B/n useful when you have several distinct hypotheses to test simultaneously, but most projects benefit from sequential A/B tests over parallel A/B/n.

in A/B Testing

Five criteria. Specific change to a specific element. Predicted direction (increase or decrease). Specific metric. Research-backed reasoning. Reasonable expected effect size. Bad: 'redesign the page.' Good: 'changing CTA from generic Submit to specific Get My Free Quote will lift form completion by 5-15% because users prefer descriptive action language.'

in A/B Testing

Primary: the conversion you most want to improve (purchases, signups, leads). Guardrails: metrics that shouldn't regress (revenue per visitor, bounce rate, average order value, support tickets). Example: testing checkout simplification with primary 'completion rate' and guardrail 'AOV' (in case simpler checkout reduces upsells). Always pre-declare metrics before the test.

in A/B Testing

Use a sample size calculator (Evan Miller, AB Tasty, Optimizely have free ones). Inputs: current conversion rate (baseline), minimum detectable effect (MDE) you care about (5%, 10%, 20%), statistical confidence level (95%), statistical power (80%). Output: required visitors per variant. Plan tests for this minimum sample - smaller samples produce unreliable results.

in A/B Testing

Five factors. Baseline conversion rate (lower CR = larger sample needed). MDE (smaller detectable lift = larger sample). Number of variants (each split increases needed sample). Statistical confidence (95% vs 99% confidence). Statistical power (80% vs 90%). Smaller MDE and lower baseline CR are the biggest sample multipliers. Plan tests where the math is achievable.

in A/B Testing

The probability that observed differences between variants are not due to chance. Conventionally p < 0.05 (95% confidence) means there's less than 5% probability the difference is random. Statistical significance is necessary but not sufficient - small effects can be significant in huge samples without being practically meaningful. Always pair with effect size and segment analysis.

in A/B Testing

The probability of observing your test result (or more extreme) if the variants are actually identical. p < 0.05 means less than 5% chance the variant 'won' randomly. p > 0.05 means insufficient evidence to declare a winner. P-values do NOT measure 'probability the variant is better' - that's a common misinterpretation. P-values measure evidence against the null hypothesis.

in A/B Testing

95% is the industry standard - corresponds to p < 0.05. Higher confidence (99%) requires larger samples for the same MDE. Lower confidence (90%) increases false positive rate. Stick with 95% unless you have specific reason (legal, medical, financial) for higher. For low-risk tests, some teams use 90% to make decisions faster, accepting more false positives.

in A/B Testing

The probability that the test will correctly detect a real effect when one exists. Conventionally 80% (some teams use 90%). Higher power requires larger sample size. Underpowered tests miss real wins (false negatives). The combination of MDE + confidence + power determines sample size. Power is the 'sensitivity' of the test.

in A/B Testing

The smallest lift you can reliably detect given your sample size, confidence, and power. Example: baseline 2% CR, sample 10,000 visitors per variant -> can reliably detect lifts of 5%+ (so 2.10% CR or higher). Smaller MDE requires larger samples (geometric scaling). Pick an MDE that matches a meaningful business effect, not just any number.

in A/B Testing

Until two conditions are met. Sample size reached (from sample calculator). Full business cycle covered (typically one week minimum, two weeks for B2B with longer purchase consideration, full week for sites with weekday/weekend patterns). Don't stop early because results 'look significant.' Don't run beyond planned duration hoping for significance - both inflate false positive rates.

in A/B Testing

Two factors. Sample size from your sample calculator. Business cycle coverage (capture at least one weekly cycle - Monday through Sunday). Some tests need multiple cycles for confidence (B2B with longer consideration). For new feature tests, allow a 'novelty' period to wear off. Minimum: 7 days. Typical: 14-28 days. Maximum: about 6 weeks before contamination from other site changes.

in A/B Testing

Sample size is the statistically required number of visitors per variant. Duration is the calendar time needed to reach that sample. Duration = sample size ÷ daily traffic ÷ variant split. Low-traffic sites have long durations. Always check both: a 3-day test on huge traffic may statistically work but miss business cycles. Always cover at least one full business cycle.

in A/B Testing

Eight. Peeking early at results and stopping. Stopping when results 'look significant' without preset criteria. Running too small a sample. Testing too many things at once without proper setup. Not covering full business cycles. Ignoring segment differences. Confusing correlation with causation. Treating one test as definitive. Pre-commit to design, sample, duration, decision rules.

in A/B Testing

Inflates false positive rates dramatically. Stopping at the first moment of 'significance' captures random variation, not real effect. A test with planned 5% false positive rate becomes 20-30% if you peek and stop. Solution: pre-commit to sample size and duration; ignore mid-test results; use sequential testing methods if early stopping is necessary for business reasons.

in A/B Testing

Running many tests simultaneously increases false positive risk. With 95% confidence per test, 20 tests have a 64% chance of at least one false positive. Address by: limiting concurrent tests, using Bonferroni correction (lower per-test threshold), running pre-registered hypotheses, requiring stricter criteria for novel claims. Many CRO teams run too many parallel tests.

in A/B Testing

Five tactics. Pre-commit to sample size and duration. Don't peek mid-test. Use 95% confidence (or stricter for novel claims). Limit concurrent tests. Validate winners with holdout periods or repeat tests. Be especially skeptical of large effects in small samples - they're usually random. False positive control matters more than always declaring winners.

in A/B Testing