Turn it off, now.
An incident runbook: switch a feature off for everyone, know when every app has it, and put the targeting back afterwards.
Print this, or link it from your incident docs. You need flags.value.update, plus flags.protected.update if Production is protected.
1
Kill: serve OFF to everyone
- Open the flag in the console and pick the affected environment.
- Switch it off. If the flag has rules, the console asks: Serve OFF to everyone or change the default only. In an incident choose Serve OFF to everyone: it removes every rule and the split in one change, so staff and rollout buckets are off too. “Change the default only” leaves rule matches on.
- Write the incident id in the note (required where the environment asks for notes).
From a script or the MCP server it is one call:
import { Flags } from "@productcraft/flags";
const flags = new Flags(); // PRODUCTCRAFT_API_KEY + PRODUCTCRAFT_WORKSPACE_ID
declare const projectId: string;
declare const settingId: string;
declare const productionId: string;
// 1. Kill: serve the plain value (off) to everyone. clear_rules removes
// every rule and the split; the old targeting stays in History.
const current = await flags.settings.values.retrieve(settingId, productionId);
await flags.settings.values.update(settingId, productionId, {
version: current.version,
value: false,
clear_rules: true,
note: "INC-42: checkout errors, killing new_checkout",
});
// 2. Later: find the change that killed it and restore the version before it.
for await (const change of flags.projects.changes.list(projectId, { setting_id: settingId, limit: 5 })) {
console.log(change.id, change.action, change.created_at, change.note);
}
declare const killChangeId: string;
const now = await flags.settings.values.retrieve(settingId, productionId);
await flags.changes.revert(killChangeId, { version: now.version, note: "INC-42 resolved" });2
When every app has it
Worst case from Save to the last running process.
| Your setup | Worst case |
|---|---|
| Auto polling, 60 s (default) | about 71 s |
| Auto polling, 30 s (minimum) | about 38 s |
| Lazy polling (serverless) | the TTL + 5 s |
| forceRefresh() or a webhook-triggered refresh | about 5 s |
| A proxy of your own in front of delivery | add up to 15 s |
To confirm, check the revision your apps report (configChanged events carry it) against the one shown in the console's History, or ask the test evaluation endpoint what a user gets now.
3
If Flags itself is down
- Running apps keep the last config they hold; new processes start from your cache or bootstrap. Nothing changes until we are back.
- While our database is down, changes cannot be saved (
503 MANAGEMENT_UNAVAILABLE) but delivery keeps serving. - For a switch that does not depend on us at all, deploy an environment variable override. It wins over the downloaded value until you remove it:
import { FlagsClient } from "@productcraft/flags";
// Deploy with PRODUCTCRAFT_FLAGS_OVERRIDES='{"new_checkout": false}' to
// force a value even when Flags cannot be reached. Remove it to hand
// control back to the console.
export const flags = new FlagsClient({
sdkKey: process.env.PRODUCTCRAFT_FLAGS_SDK_KEY,
overrides: { env: "PRODUCTCRAFT_FLAGS_OVERRIDES", behaviour: "local_over_remote" },
});4
Restore after the incident
Open History, filter by the flag, find the kill change and choose Restore this version on the change before it. That writes the old rules and split back as a new change, with its own note; nobody loses their bucket, because buckets come from the seed, not from stored state. The second half of the script above does the same with changes.revert.