Fail over to a second text provider

How to do it

A text provider outage, or one carrier route going bad, stops every sign-in that needs a code. Register your senders with both providers ahead of time, as Register every sender before you send says, because a new registration can take weeks. Send through one interface with an adapter per provider. Count failed API calls and the time from send to the delivered callback per provider and country, as Track what happened to each text records. Switch when, for example, 5 percent of calls fail over 5 minutes, or the median delivery time passes 60 seconds. Keep one idempotency key per message, so a retry on the second provider never sends two codes. Push has no second route, since APNs and FCM are the only services for their devices, so offer another channel instead.

Examples

JavaScript example

const FAILOVER_ERROR_RATE = 0.05;

export async function sendCritical(message) {
  const [primary, backup] = providers;
  const healthy = (await health.errorRate(primary.name, 300)) < FAILOVER_ERROR_RATE;
  const first = healthy ? primary : backup;
  try {
    return await first.send(message, { idempotencyKey: message.id });
  } catch {
    return (first === primary ? backup : primary).send(message, { idempotencyKey: message.id });
  }
}

Python example

FAILOVER_ERROR_RATE = 0.05

def send_critical(message):
    primary, backup = providers
    first = primary if health.error_rate(primary.name, 300) < FAILOVER_ERROR_RATE else backup
    try:
        return first.send(message, idempotency_key=message.id)
    except ProviderError:
        other = backup if first is primary else primary
        return other.send(message, idempotency_key=message.id)

How to check

A test makes the first provider's adapter return errors. The next code text goes out through the second provider, once. A monthly drill sends one live code through the second provider to a team phone and it arrives.

Sources

• Twilio, Track the message status of outbound messages

Related rules

• NTF-22 Register every sender before you send in Notifications
• NTF-29 Track what happened to each text in Notifications