skills-for-ai
Get started
Data Skill

A/B Test Auditor & Planner

A/B test significance check — before you ship, I'll tell you the truth.

A/B testSignificanceCROA/B TestingSample Ratio MismatchBayesian A/B TestConfidence intervalStatistical PowerSample Size CalculatorSample sizeMDETest planningExperimentExperimentationConversionGrowthOptimizelyVWOGA4StatsigHypothesis testExpected LossShip or HoldStatistical SignificanceBayesian statistics
Free
0 ↓ 0 v2.6.0 Updated 22.06.2026 by skills-for-ai

SKILL.md in the open agentskills.io standard — works directly in Claude, ChatGPT/Codex, Cursor, Copilot & more.

A/B Test Auditor & Planner is an AI skill for experimentation — an A/B test significance calculator and audit that checks statistical significance, sample size and sample ratio mismatch (frequentist or Bayesian) and gives a clear ship / hold / iterate verdict.

What this skill does

  • Test planning: sample size, runtime, MDE plausibility (binary & continuous)
  • Result audit with a validity checklist (sample ratio mismatch, peeking, power, novelty, data quality)
  • Frequentist analysis (z-/Welch t-test, confidence interval, multiple-comparison correction)
  • Bayesian analysis (posterior, P(B>A), credible interval, expected loss)
  • Guardrail and (exploratory) segment reporting
  • Clear Ship / Hold / Iterate recommendation with business impact
  • Handles summary metrics OR raw CSV/event data

Description

A/B Test Auditor & Planner is a skill for statistically sound A/B tests — it plans and audits your tests with full rigor, catches hidden killers like sample ratio mismatch and gives a clear ship/hold/iterate verdict. Most A/B tests are planned or read wrong — too-small samples, peeking, missing power or broken tracking splits. A falsely shipped test costs real revenue. This skill brings analyst-level rigor to any team: it plans the required sample size and runtime up front, checks finished tests for validity (SRM, peeking, power, novelty, guardrails), computes significance + confidence/credible interval + expected loss, and turns it all into a clear roll-out decision — no stats degree needed.

Examples

SRM trap (audit)
Control 5000/410, variant 5480/500, planned 50/50, 10 days, p=0.03.
You July 17, 2026
A/B Test Auditor & Planner

Verdict: HOLD — don’t roll out.

GroupUsersConversionsRate
Control5,0004108.2%
Variant5,4805009.1%

Problem: sample ratio mismatch. Planned 50/50, actual 5,000 vs. 5,480 → χ²≈22, p≈3e-6. The assignment is broken; chance hardly explains the skew.

Consequence: the +11% lift is not reliable — biased groups, no valid comparison.

Next step: inspect and fix the split mechanism (bucketing, redirect, bot filter), then re-test cleanly.

Bayesian audit
Bayesian please: A 2000/120, B 2000/150, clean 50/50, 14 days, guardrails ok.
You July 17, 2026
A/B Test Auditor & Planner

Verdict: SHIP — variant B.

Bayesian analysis:

  • P(B > A) ≈ 97%
  • Credible interval, rel. lift ≈ [−1%, +58%]
  • Expected loss from choosing B ≈ 0

Data quality: clean 50/50, guardrails ok — no SRM.

Reading: strong evidence for B. Lower bound slightly negative → no drama; if you want near-zero risk, run three to four more days.

Test planning
Baseline 4%, want to detect +10% relative, 1000 users/day total, 2 variants.
You July 17, 2026
A/B Test Auditor & Planner

Sample-size planning:

  • Baseline 4%, target +10% relative (→ 4.4%), 2 variants, α = 0.05 / power 80%.
  • Needed: ~39,500 users/variant → at 1,000/day total ≈ 11 weeks.

Warning: that's a small effect — 11 weeks is long (seasonality, cookie drift).

Alternative: design for +20% first → only ~3 weeks. Confirm the big effect first, then refine.

1 / 3

What it does not do

  • Won't invent optimization ideas or hypotheses
  • No funnel or landing-page design
  • No ad copy or creatives
  • No industry benchmarks ('is 3% good?')

Compatibility & tech

Claude (Projects & system prompt)ChatGPT (Custom GPT & custom instructions)Any LLM with computeEngine: frequentistEngine: Bayesian
Tested (internal)
3 scenarios
Recommended runtime
With a code/analysis tool (Claude Analysis, ChatGPT Data Analysis) for exact stats; also works without, then with clearly flagged approximations.
Modes
Planning · Audit
Inputs
Summary stats · CSV/event data
Output format
Markdown report (verdict, validity checks, statistics, recommendation)
Subcategory
Experimentation / A/B testing
License
Proprietär

Security profile

Local

Runs entirely on your machine with your own AI — no external runtime, no running costs.

Instruction-only

Only instructions, templates and references — no executable scripts.

No network access

Works offline with what you provide — does not call external services on its own.

What do these badges mean? →

What you get

  • ab-test-auditor-2.6.0/4 files
    • .claude-plugin/marketplace.json
    • plugins/ab-test-auditor/skills/ab-test-auditor/4 files
      • SKILL.md
      • manifest.json
      • references/2 files
        • ab-test-auditor-checklist.md
        • methods.md
    • .agents/skills/ab-test-auditor/→ universal — same content (Codex, Cursor, Copilot, Gemini, Windsurf, Cline)
    • LICENSE.txt

Installation

After unlocking, you install with a single command — it auto-detects your AI tool.

Runs inClaude CodeGitHub CopilotGemini CLICursorCodex CLIWindsurfCline

Also works as a chat prompt

Full in chat

No AI tool? Paste it into Claude, ChatGPT or Gemini and use the method right away.

This skill ships no scripts, so the prompt carries its full method.

Installing is the full version — it triggers automatically, runs its scripts and loads references as needed. As a chat prompt you drive the method by hand.

Unlock to copy the ready-to-paste prompt — then in “My Skills”.

Reviews

No reviews yet – be the first.

Note

Decision support based on the data you provide — no guarantee of success.

Changelog

  • v2.6.022.06.2026Platform-native output: now actively uses your platform's native formatting — tables, checklists, code blocks and clickable/button options, with a numbered fallback on non-interactive platforms. Plus internal metadata that prepares the upcoming agent builder.
  • v2.5.018.06.2026Reworked to the AUTHORING v2 standard: realistic input/answer examples (incl. a calibration case), copy checklist(s) and internal evals; cross-references added.
  • v2.4.018.06.2026Added references/methods.md: deeper detail reference via progressive disclosure; SKILL.md stays lean.
  • v2.3.010.06.2026Compute reliability: use a code tool, otherwise flag the approximation.
  • v2.2.010.06.2026Graceful handoff for out-of-scope requests.
  • v2.1.010.06.2026Fixed an incorrect sample-size figure in an example.
  • v2.0.010.06.2026Planning mode, Bayesian engine, CSV handling, mode router.
  • v1.0.010.06.2026Frequentist audit with a validity checklist.

Frequently asked questions

What does A/B Test Auditor & Planner do?

A/B Test Auditor & Planner is an AI skill for experimentation — an A/B test significance calculator and audit that checks statistical significance, sample size and sample ratio mismatch (frequentist or Bayesian) and gives a clear ship / hold / iterate verdict.

When is an A/B test statistically significant?

Significance depends on sample size, effect and confidence level — and not every green winner holds up. A/B Test Auditor & Planner checks this with full statistical rigor and gives you a clear ship/hold/iterate verdict instead of gut feeling.

What is a sample ratio mismatch (SRM)?

An SRM means users weren't split across variants as expected — a silent killer that makes results worthless. A/B Test Auditor & Planner catches exactly these hidden problems before you draw wrong conclusions.

How many visitors do I need for an A/B test?

The required sample size depends on baseline rate, expected effect and confidence level. A/B Test Auditor & Planner plans this upfront — frequentist or Bayesian — so your test can actually be meaningful.

Frequentist or Bayesian — which is better for A/B tests?

Both approaches are valid and fit different situations. A/B Test Auditor & Planner handles both and applies the appropriate method with full rigor.

Which AI tools does A/B Test Auditor & Planner work with?

Claude (Projects & system prompt) · ChatGPT (Custom GPT & custom instructions) · Any LLM with compute · Engine: frequentist · Engine: Bayesian

How do I use A/B Test Auditor & Planner?

A/B Test Auditor & Planner is a SKILL.md in the open agentskills.io standard: install it with one command (npx) or download it and add it to your AI tool — Claude (Projects), ChatGPT (Custom GPT), Cursor, Copilot, Gemini CLI and more. No code needed.