> For the complete documentation index, see [llms.txt](https://docs.hyperswitch.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hyperswitch.io/integration-guide/workflows/intelligent-routing/ab-testing.md).
# A/B Testing
A/B testing runs two routing strategies side by side on a **live split** of your own traffic - a **control** arm (what you do today) and a **variant** arm (the change you're evaluating) - measures how each performs, and tells you whether the difference is **statistically significant** or just noise. A built-in **guardrail** flags the experiment the moment the variant starts hurting approvals, so you catch a bad change early.
Use it whenever you change how payments are routed - turning on Multi-Objective Routing, letting [Autopilot](/integration-guide/workflows/intelligent-routing/autopilot.md) self-tune, adjusting an auth-rate setting - to prove the change helps **before** it touches all your traffic. A change that quietly lowers your approval rate costs far more than any fee it saves.
> **Note.** An experiment compares *routing strategies*, not payment providers. Each arm is a routing configuration. The connectors are still chosen by that arm's logic per payment.
## The core idea - control vs variant
An experiment has exactly two arms.
* **Control** - your baseline strategy (typically your current live routing).
* **Variant** - the change you want to prove out. Always the **minority** arm - it takes a small, capped share of traffic so a bad change can only ever affect a fraction of payments.
You choose what share of traffic the variant sees (the **variant split**). In the dashboard builder this is a slider capped at **5-30%**. Everything else goes to control. Because the variant is always the minority, control stays the majority - the safe side.
Each arm is one of two things.
* **An Auth-Rate Routing strategy** with its own settings (auth-rate scoring, optionally with [Multi-Objective Routing](/integration-guide/workflows/intelligent-routing/routing-strategies/multi-objective-routing.md) and/or [Autopilot](/integration-guide/workflows/intelligent-routing/autopilot.md)). This is the common case.
* **An existing named routing algorithm** - a single connector, a priority list, a volume split, or an advanced rule tree you've already built.
## Setting up an experiment
You build an experiment in the dashboard's A/B test builder, with the following pieces.
| Setting | What it does |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Experiment type** | **Algorithm comparison** (pick any two strategies as arms) or **Auth-rate config tuning** (same strategy both arms, variant changes one or two settings). |
| **Control strategy** | The baseline arm. |
| **Variant strategy** | The arm you're testing. |
| **Variant split %** | Share of traffic sent to the variant (builder slider, 5-30). The rest goes to control. |
| **Sample target (minimum sample size)** | How many transactions each arm needs before results are treated as meaningful (builder presets 1,000 / 5,000 / 10,000 / 50,000, default 5,000). Until then the verdict is "collecting data". |
| **Guardrail threshold (pp)** | If the variant's approval rate falls this many percentage points below control, the experiment is flagged **guardrail breached** (default 3 pp). |
### Strategy presets
For the **Algorithm comparison** type, the builder offers ready-made auth-rate strategies so you don't have to hand-tune each arm - two independent dials (Multi-Objective Routing × self-tuning).
| Preset | What it is |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Maximize approvals · manual tuning** | Pure auth-rate routing - pick the connector most likely to approve. |
| **Maximize approvals · auto-tuned** | The same, with [Autopilot](/integration-guide/workflows/intelligent-routing/autopilot.md) self-tuning the auth-rate settings. |
| **Approvals + save on fees · manual** | Adds Multi-Objective Routing - cheapest among equally-reliable connectors, settings you control. |
| **Approvals + save on fees · auto-tuned** | Multi-Objective Routing with Autopilot self-tuning. |
You can also pick a **saved routing algorithm** (a single connector, a priority list, a volume split, or an advanced rule tree) as either arm.
> **Note.** The "save on fees" presets only appear if you have [**Multi-Objective Routing**](/integration-guide/workflows/intelligent-routing/routing-strategies/multi-objective-routing.md) enabled, and the "auto-tuned" presets only if **Autopilot / auto-calibration** is on.
A typical first experiment is **control = Maximize approvals** vs **variant = Approvals + save on fees** - i.e. "does Multi-Objective Routing save money without hurting approvals?"
## How traffic is split
Arm assignment is **deterministic per payment**. Each payment's id is hashed and taken **modulo 100** against the variant split - so a given payment always lands in the **same arm**, including on retries. This matters - a payment retried after a decline is scored consistently, never split across arms. (At the platform level the split is capped below 50%, so the variant is always the minority arm. The dashboard builder narrows that to a 5-30% slider.)
There are two places this assignment is used.
* **Real payments** - your live routing decisions route into the assigned arm. This only happens once you explicitly enable **real-payment interception** for the merchant (the `ab-test-real-payments` feature). It's **off by default**, so creating and activating an experiment does **not** touch live traffic until you opt in.
* **Simulation / Decision Explorer** - you can preview what each arm *would* do for any payment context without a real transaction, using the routing simulator. This works even before real-payment interception is on, so you can sanity-check an experiment before it ever sees a customer.
> **Info.** An experiment must also be **activated** (like any routing algorithm) to run. And an experiment can only be **edited or deleted while inactive** - deactivate it first if it's live.
## Guardrails
The variant is the risky arm, so it's watched. Once each arm has enough data, before any significance test runs, the results check the guardrail first - if the variant's **approval rate is more than the guardrail threshold** (in percentage points) below control, the verdict short-circuits to **guardrail breached** and the dashboard shows a red banner - "Variant auth rate dropped X pp below control. Consider stopping the experiment."
This is a **flag, not an automatic stop** - you decide whether to stop it (there's a **Stop** button). Combined with the capped minority split, it means a bad variant only ever affects a small share of traffic and is surfaced clearly.
## Reading the results
Once both arms have traffic, the results view runs a **statistical significance test** between them and returns a verdict. For a Multi-Objective Routing experiment it tests **net expected value** (approvals *and* fees together) with a two-sample z-test. For an approvals-only experiment it collapses to a plain **approval-rate** test.
**Per-arm metrics**
| Metric | Meaning |
| ------------------------------------------ | ----------------------------------------------------------------------------------- |
| **Transactions** | How many payments the arm handled. |
| **Outcome** | Approved vs declined counts. |
| **Net auth rate (NAR)** | Overall approval rate - any attempt succeeded. |
| **First-attempt rate (FAAR)** | Approval rate on the first try, before retries. |
| **Cost saved (TCS)** | Total fees saved versus baseline (Multi-Objective Routing arms only). |
| **Avg chosen cost / avg cost saved (bps)** | Average fee of the connector picked, and the per-txn saving. |
| **Avg latency (ms)** | Average routing-decision latency for the arm. |
| **Net EV (bps)** | Net expected value - the combined approvals-and-cost score the winner is judged on. |
For an **approvals-only** experiment (neither arm uses Multi-Objective Routing), the cost fields - cost saved, avg cost saved, and Net EV - are empty, and the winner is judged on approval rate alone.
**Overall stats**
| Field | Meaning |
| ------------------------ | ------------------------------------------------------------------------ |
| **delta\_pp** | The variant's approval-rate difference vs control, in percentage points. |
| **p\_value** | Probability the observed difference is chance. Lower = more confident. |
| **confidence\_interval** | The plausible range of the true difference. |
| **net\_delta\_bps** | The net expected-value difference between arms. |
| **verdict** | The bottom line (see below). |
### Verdicts
| Verdict | Meaning | What to do |
| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------- |
| **Collecting data** | At least one arm hasn't hit the minimum sample size yet. | Wait for more traffic. |
| **Not significant** | Enough data, but the difference is within noise. | The change is neutral. Keep or drop it on other grounds. |
| **Variant wins** | The variant significantly beats control on net expected value. | Roll the variant out. |
| **Variant loses** | The variant significantly underperforms control. | Keep control. Discard the variant. |
| **Guardrail breached** | The variant's approval rate fell too far below control. | Stop - the variant is hurting approvals. |
You can also open the **per-transaction log** to see exactly which arm each payment landed in and how it turned out - useful for spot-checking arm assignment or debugging a specific payment.
## A typical workflow
1. **Build the experiment.** Pick control (your current routing) and variant (the change), set a modest variant split (e.g. 10-20%), a sample target, and a guardrail threshold.
2. **Preview in the simulator.** Confirm each arm does what you expect on sample payments.
3. **Activate it.** Activating prompts you to **enable live-traffic testing** at the same time. If you activate without it, the experiment shows as **Not collecting** (an amber banner reminds you to turn on live traffic). Switch it on when you're ready for the variant to take its split of real payments.
4. **Watch the results.** The status reads **Active** once it's live and collecting. The verdict starts at "collecting data" and firms up as traffic accrues (statistical significance is declared at p < 0.05).
5. **Decide.** On **variant wins**, promote the variant to your live routing. On **variant loses** or **guardrail breached**, keep control (hit **Stop**). An experiment must be **inactive** to be edited or deleted - you can also **duplicate** one to iterate.
## Relationship to other features
* [**Multi-Objective Routing**](/integration-guide/workflows/intelligent-routing/routing-strategies/multi-objective-routing.md) - a variant arm can turn this on to A/B test whether it pays off for you.
* [**Autopilot**](/integration-guide/workflows/intelligent-routing/autopilot.md) - a variant arm can enable Autopilot self-tuning, scoped to just that arm, to test self-tuning against your manual settings.
* **Merchant features** - where `ab-test-real-payments` is toggled (Dashboard → merchant features).
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.hyperswitch.io/integration-guide/workflows/intelligent-routing/ab-testing.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.