Run Trawl in production.

timeout_s is the budget you control. Set it per job at create time: 1 to 3,600 seconds, default 120. It bounds the extraction run itself — an agent that hasn't finished inside the budget ends as failed, with either extraction did not complete within the step/time budget or extraction timed out after <ms>ms depending on where the clock caught it. (There is also a server-side step budget — each agent step is roughly one model call — so timeout_s and page count are the levers you size; see 03 · Schema design for sizing guidance.)

The reaper is the safety net. A background sweep runs every 5 minutes and force-fails jobs that can no longer produce a result:

• A running job is failed once started_at is older than timeout_s plus a 60-second grace period — the grace covers a result that's in flight or a worker restart, so a job finishing right at the buzzer isn't clobbered.
• A queued job that no worker ever picked up is failed after 30 minutes.

Reaped jobs come out looking like any other failure: status failed, completed_at stamped, and a distinctive error string —

A reap sends job.failed to your webhook endpoints like any other failure — your receiver doesn't need a special case.

Worst-case time to a terminal state (useful for setting your own alerting thresholds): for a running job whose worker died, timeout_s + 60 s grace + up to 5 minutes of reaper cadence. For a queued job that was never picked up, 30 minutes + up to 5 minutes. If a job is still non-terminal well past those bounds, that's worth a support ticket — it should be impossible.

• Extraction: one run per job. A job is executed once; there are no automatic re-runs of a failed extraction. The attempts field on the job is an informational counter, not a retry knob. To try again, create a new job — ideally with the failure's lesson applied (looser schema, better seeds, bigger timeout_s).
• Transient model-provider errors: retried in place. Inside a run, transient 429/5xx errors from the model provider are retried up to 4 times with backoff. You never see these; they just consume budget.
• Webhook delivery: retried automatically for about 33 hours. A failed delivery is recorded in the event log (section 5), counts toward auto-disable, and is sent again after 1 minute, 5 minutes, 30 minutes, 2 hours, 6 hours and 24 hours (seven attempts in all, same webhook-id). If every attempt fails, nothing is lost: the result lives on the job row. Poll GET .../jobs/:id to recover.
• Messaging-layer redelivery: rare duplicates possible. Delivery is at-least-once end to end — a retry, or a crash on our side mid-dispatch, can deliver the same event twice. Dedupe on X-Trawl-Event-Id.
• Your own retries of the create call: use Idempotency-Key. Replaying POST .../jobs with the same key and body returns the original response (with header Idempotent-Replay: true) for 24 hours instead of enqueuing a second job; the same key with a different body is rejected with 409.

• 10 job creates per minute per workspace. The 11th in a fixed one-minute window gets 429 with "code": "RATE_LIMITED" and a Retry-After header (seconds until the window resets). Sleep for Retry-After and retry — that's the whole handling story.
• 25 jobs queued or running at once. Over that, creates get 429 with "code": "TOO_MANY_ACTIVE_JOBS" and no Retry-After — it clears when your jobs finish, not on a clock. Cancel or wait.

Rejected requests don't consume quota (validation runs first), and an idempotent replay short-circuits before the limiter — so retrying a create with the same Idempotency-Key is free. Reads (GET jobs, webhooks, events) are not rate limited today; poll on a sensible interval anyway.

The list is newest-first with cursor pagination (limit 1–100, default 20; pass cursor from pagination.next_cursor while pagination.has_more is true). The status filter accepts queued, running, succeeded, failed, and cancelled. Listing needs the trawl.list action; reading a single job needs trawl.read.

On a failed job, the error string is the diagnosis. The common ones — and what to do about each — are in the troubleshooting matrix below.

Events of the last 30 days, newest first (requires trawl.webhook.read). GET .../webhooks/events/:event_id lists every attempt. How to read an attempt:

• status_code is null when the request never got a response — DNS failure, TLS failure, the 15-second delivery timeout, or a URL our delivery guard refused (delivery requires an HTTPS URL resolving to a public address; private, loopback, link-local, and cloud-metadata targets are rejected). In those cases error_message says why.
• response_body_truncated holds up to the first 1,024 bytes of your endpoint's response — enough to see your own error message.
• Success is any 2xx; anything else is a recorded failure.

Auto-disable. Failures accumulate per endpoint: after 20 consecutive failures, or once a failure streak has lasted 24 hours, the endpoint is switched off (status becomes auto_disabled) and its queued retries are dropped. Any 2xx along the way resets the streak.

Recovery: fix your endpoint, switch it back on, prove it with a test, then replay what it missed:

Switching on clears the failure streak (requires trawl.webhook.manage). The test returns what your server answered. The results are also still on the job rows, so you can reconcile by listing jobs that completed during the outage instead.

• Requires the trawl.cancel action. Only queued and running jobs can be cancelled; a job that has already reached succeeded, failed, or cancelled is immutable and the cancel is rejected with 422. An unknown id (or a job in another workspace) returns 404.
• On success you get the job back with status cancelled and completed_at stamped. Treat any 2xx as success — the endpoint answers 201 today.
• Cancellation doesn't interrupt a running extraction. It marks the job cancelled; a worker already browsing keeps going until its own budget, and the late result is dropped rather than written back. Practical upshot: cancelling a running job stops you from seeing the result, not from the work happening — so cancel to clean up your own pipeline, not to save budget.
• Cancellation fires no webhook event. Only job.succeeded and job.failed exist — if your pipeline tracks cancelled jobs, track them from the API response or by polling, not from webhooks.

• Job stuck in queued. Workers pull jobs from a queue; under load, queued time is normal. A job that is never picked up is failed by the reaper after 30 minutes with job exceeded its time budget (reaped) — so “stuck forever” can't happen. If you see queued jobs regularly aging toward that ceiling, that's a platform-side capacity signal — contact support rather than re-submitting the same job repeatedly.
• Failed with extraction did not complete within the step/time budget (or extraction timed out after <ms>ms). The agent ran out of time or steps. Unreachable pages are a common cause — the agent sees a navigation error and tries other sources, which burns budget; if every source is dead or the information genuinely isn't findable, the budget runs out. Fix: verify your suggested_urls resolve, raise timeout_s for multi-page jobs, or narrow the ask.
• Failed with agent could not complete the extraction. The agent finished but judged that it couldn't actually satisfy the request — it never ships a “successful” result it doesn't stand behind. Usually the data isn't on the public web in the shape you asked for. Make hard-to-find fields nullable (03 · Schema design) so a partial answer is a legal answer.
• Blocked or bot-walled pages. Trawl doesn't do anything special to defeat anti-bot walls. A blocked page behaves like any other unreachable page: the agent typically routes around it to other sources, or the job ends in one of the budget failures above. If the only source for your data aggressively blocks automation, Trawl is the wrong tool for that source. Note also that pages behind a login are out of scope — Trawl reads the public web only.
• Failed with result failed schema validation or invalid json_schema: …. Schema problems, not operations problems: the first means the agent's output couldn't satisfy your constraints; the second means the stored schema used a construct the converter can't handle. 03 · Schema design covers both, including the keyword set to stay inside.
• “Empty” extraction — success, but fields are null. Working as designed: the agent is instructed to return null rather than invent a value it couldn't find. Lots of nulls means the description didn't point at where the data lives, or the data isn't public. Sharpen the description, add suggested_urls, and treat persistent nulls on a field as a signal to drop it.
• Webhook silent while jobs are completing. Check the endpoint first: if its status is auto_disabled, it was switched off after a failure streak. Also check its event_types include the event you expect. Fix your endpoint, PATCH { "enabled": true }, then POST .../test to confirm — see section 5.
• 403 on every call: trawl is not enabled on this workspace. The workspace doesn't have the trawl service enabled — enable it in the console under Workspace → Services → Trawl → Enable. A 403 naming a specific action (e.g. Policy denies trawl.create on *) means your key or role lacks that permission instead.
• 429 on create: RATE_LIMITED or TOO_MANY_ACTIVE_JOBS. You hit a submission limit — see section 3. Honour Retry-After when it's present; otherwise wait for in-flight jobs to finish.

• Duration scales with pages: every extra source the agent visits costs extra steps, and each step is roughly one model call. The default 120-second budget suits a job that reads one to three pages; research-style jobs that aggregate across many sources need a larger timeout_s (up to 3,600).
• Good suggested_urls shorten runs — without seeds the agent spends steps searching for sources before it extracts anything.
• Measure your own workloads: enqueued_at, started_at, and completed_at are on every job row, so queue wait and run duration are one subtraction each. Alert on the worst-case bounds from section 1, not on averages.

From here:

• API reference for every endpoint, field, and status code.
• Concepts for the mental model behind jobs, schemas, and webhooks.
• Revisit 04 · Webhooks when you harden your receiver — signature verification and event dedupe live there.