A/B testing text messages when your list is small

A small business splitting 400 customers into two groups of 200 will see a difference between them. That difference is almost always noise. Understanding when a result is real is the difference between improving your messaging and randomly changing it every month.

Test big differences, not wording tweaks

With a small list you can only detect large effects. A 2% improvement from changing 'Hi' to 'Hello' is undetectable and probably does not exist. A change from 'click here to confirm' to 'reply C to confirm' can move response by a third, and that you can see.

Split deterministically

Hash the customer ID with the test name so a given person always lands in the same arm. Random assignment per send means someone gets variant A on Monday and B on Thursday, which measures nothing.

import { createHash } from "node:crypto";
 
export function variantFor(customerId: string, testName: string) {
  const hash = createHash("sha256")
    .update(`${testName}:${customerId}`)
    .digest();
 
  return hash[0] % 2 === 0 ? "a" : "b";
}
 
// Record it in utm_content so the arm follows the click all the way
// through to the conversion.
const link = tagged(url, {
  campaign: "appointment_reminder",
  content: `reply_vs_click_${variantFor(customer.id, "reply_vs_click")}`,
});

Decide the sample size before you start

Not after you look at the numbers. A rough rule for a small list: you need roughly a few hundred per arm to reliably detect a change of ten percentage points on a rate around 50%. Smaller effects need far more. If you cannot reach that, accumulate across several sends of the same campaign instead of calling it after one.

Measure the outcome, not the click

A variant that gets more clicks and fewer bookings has lost. Always evaluate against the action that pays you — confirmed appointment, invoice paid, order placed — with the variant carried through in utm_content. Tracking it end to end.

Watch the metric nobody tracks

Opt-out rate per variant. A message that converts slightly better while generating twice the unsubscribes is destroying an asset to win a quarter. Track it alongside the win, and treat a rising rate as a failed test regardless of the conversion number.