Diagnose missing pages and progress

Captures and crawls are both requests. Use capture_list or crawl_list (optionally finished=true or false, search by name or URL, or an exact host) for your organization's work, then capture_status (by request_id) or crawl_get for one request. A capture request has one url and, once the page is captured, its capture_id; failure explains a page that could not be captured. A crawl's urls holds the URLs you supplied; crawl.follow holds the page limit, depth and boundaries, or is null for a list crawl that fetches exactly its URLs. Your URLs count toward follow.page_limit, which is a ceiling, not a promise that every host was fetched. name and description (Markdown) are optional labels and never change the work; version guards edits through PATCH /captures/{id} or PATCH /crawls/{id}.

Read the view in this order:

1. status: queued (nothing started), running, paused, done or cancelled. A cancelled or time-limited request may still be finishing already-started pages.
2. For a capture request, capture_id (with failure if it failed). For a crawl, progress: captured, failed and pending counts for this request only, not corpus coverage; list failed captures to see why each failed. finish_reason explains a finished request: completed (a capture or list crawl), page_limit_reached, no_more_links, time_limit or cancelled.
3. queryable: true once the capture (for a crawl, every captured page) is in published SQL; false for a failed capture, which has no page to query. It stays null until the request is finished and while readiness cannot be confirmed. A finished request can precede SQL availability; pass wait_seconds (up to 60) to get instead of polling rapidly.
4. waiting: null, or a reason why nothing is progressing: paused, crawler_paused, stopping, website_pacing, website_paused, retrying, crawler_busy, crawler_unavailable or following_paused. until is the earliest observed eligibility, not a promised start time. Do not cancel and recreate work merely because it is waiting.

For missing host X in a crawl, call crawl_captures with host='host.example'. host matches exactly; it does not include subdomains. Redirects may move a successful page to a different effective_url. Add status='failed' for failed captures (they keep their capture_id) or status='captured'; pages not yet attempted are counted in progress, not listed. Add depth for exact traversal depth (your URLs are zero), or search for a literal URL substring. Rows include capture_id, http_status, parent_capture_id, a safe failure explanation and capture_assessment. Follow next_cursor with the same filters.

capture_assessment is the capture service's own verdict on the page, copied as reported: source "capture_service", completeness, a coarse confidence, plain-language reasons and the service's decision trail (path, tier, decision and reason_code for each step). A captured page is complete under the service's capture policy, so its completeness is "complete". A failed page reports "partial" or "empty" when the service's last assessment said so, for example incomplete_content with loading placeholders still visible; partial HTML is never stored as a capture. It is null for pending pages and for results recorded before verdicts were kept. Pass completeness=complete, partial or empty to filter by it, or completeness=unknown for rows without one. Periplus retries transient failures such as incomplete_content within each request's attempt budget.

render_incomplete is deprecated. It was a Periplus heuristic for loading placeholders, now retired: capture quality is the capture service's verdict. Captures recorded before the change keep their true, false or null value; newer captures are always null, and it is never derived from capture_assessment. The filter keeps its old meaning: render_incomplete=true lists only those historical flags, and false lists every other row, including all newer captures. Use completeness instead.

failure_code accepts access_denied, not_found, gone, rate_limited, website_error, request_rejected, paywall, parked, geo_blocked, bot_challenge (a bot check not passed; with the user's consent and the organization's allowance, offer a rerun of just those URLs with resolve_bot_challenges=true, which permits paid resolution; requesting it does not prove a resolver ran), bot_blocked (the capture service detected a bot-protection refusal for this capture), interstitial, unsupported_browser, incomplete_content, unsupported_content (not HTML, XML or JSON), excluded, redirect_loop (redirects did not end), timeout, connection, host_not_found (the domain does not exist), capture_unavailable (the capture service itself repeatedly could not process the page for about an hour; try again later) or unknown. Failure-code filtering inspects at most 100 rows per call: keep following next_cursor even after an empty page.

A failed capture does not establish access to other pages on the same host. HTTP 403 alone does not identify the refusal's cause. If resolution was requested, check what actually ran with support before another paid retry; see bot challenges.

If a crawl did not reach X, check that X was one of its urls or reachable within follow.max_depth, follow.scope, host and path rules, and follow.page_limit. Path rules filter traversal: excluding an intermediate page can hide pages linked only through it. A following_paused request found a page too large to extract links from; an operator must resume it.

Captures are made from Japan. A geo_blocked page, or a regional storefront serving Japan-specific language, prices or assortment, reflects that region.

A crawl's capture list is the only record of which captures it produced. Failed and bodyless captures never appear in the public captures relation, so empty SQL results cannot prove a host was never attempted.

A 429 crawl_queue_delay or crawl_history_pressure rejects new admission while the crawler is busy. Retry-After is when to check admission again, not a promised start time. Use backoff rather than rapid resubmission; accepted requests keep running. Stable request IDs make retrying the same intent safe; do not create duplicates.