Crawlscoverage for any list of URLs

Capture one URL for instant feedback (waits and summarizes); crawl for coverage (any list of URLs, optionally following links).

When to crawl

A crawl collects a finite set of pages for coverage. Give it 1–1,000 HTTP(S) URLs:

• Without link following it fetches exactly those URLs: a list crawl. Use it for any known list of more than one URL; a capture takes one.
• With link following it also follows links from them: a publication, directory, research section, or other starting point relevant to your question.

1. Add 1–1,000 URLs.
2. To follow links, set a page limit (the most pages you are willing to collect, including your URLs), the link depth (from 1 up to your organization's limit, 10 by default) and whether to stay within the starting sites or follow links to other sites.
3. Optionally pin every successful capture for a number of days, then start the crawl.

Fetch a list of URLs

A crawl without follow fetches each URL once and follows no links; its page limit is the number of distinct URLs, which counts against your organization's crawl page limit. It finishes completed once every URL is captured or has failed. Unlike a capture it may reuse a capture of the same URL made in the last few minutes, and it does not wait or summarize: follow it with crawl_get or GET /crawls/{id}?wait_seconds=30.

{"urls": ["https://example.com/a", "https://example.com/b", "https://example.com/c"]}

Keep exploration relevant

These boundaries go inside follow. Crawls stay within the starting sites by default (follow.scope: starting_sites): the registrable domains of your URLs, including subdomains. Following links to other sites reaches connected websites; it is not a topic filter.

Boundaries let you include or exclude hosts (allowed_hosts, excluded_hosts) and exclude URL paths (excluded_paths); include paths with link rules, for example {"path": "/research/*"} follows only paths under /research/. Hosts include their subdomains, and your URLs must satisfy host rules. Paths are case-sensitive and match the whole URL-encoded path; * is the only wildcard, not a regular expression. Query strings and fragments are ignored.

Exclusions take precedence. Your URLs may sit outside the link rules, but cannot match an excluded path. Excluding an intermediate page can also prevent reaching pages linked through it. Boundaries do not block website redirects.

Select paths and query parameters

In Advanced filters, add follow alternatives. A link needs to match any alternative (OR). Within an alternative, its path and all query conditions must match (AND). Leaving alternatives empty follows any link within your other boundaries. Skip paths and excluded hosts always win.

For example, follow filing-history pagination and XHTML documents without collecting PDF links on the same document paths:

Try a link checks path/query rules without visiting a URL. It does not check site boundaries, document types, budgets or whether the page advertises that link. Your URLs bypass alternatives. These rules restrict traversal, so links through unmatched intermediate pages may be missed. They do not block redirects.

Query names and values are decoded and case-sensitive. Percent escapes decode once as UTF-8 (invalid UTF-8 becomes the replacement character); + means a space, & separates parameters, and blank values are kept. URLs themselves are never rewritten or reordered.

• equals: the parameter exists and every repeated value equals the literal. format=xhtml matches; format=xhtml&format=pdf does not.
• does not equal: no value equals the literal; a missing parameter also matches. Add is present if it must exist.
• is present: the parameter exists, including an empty value.
• is absent: the parameter does not exist.

The API and MCP crawl_create tool accept the same typed follow.rules: up to 100 alternatives with up to 10 query conditions each. Query values are literal strings; path globs use only *. An alternative needs a path or query condition. Comparisons require value (empty string is valid); presence operators omit value. Path-only alternatives are how you include paths: [{"path": "/research/*"}, {"path": "/companies/*"}] follows links under either. Path inclusion and query conditions combine only within one alternative: a separate list of included paths that every alternative must also satisfy is not available. Older crawls created with such a restriction show it, read-only, as follow.retained_allowed_paths. Invalid rules are rejected before acceptance. Full follow SQL stays internal.

{
  "urls": ["https://example.com/company/123/filing-history"],
  "follow": {
    "max_depth": 3,
    "page_limit": 100,
    "rules": [
      {"path": "/company/*/filing-history"},
      {"path": "/document/*", "query": [
        {"name": "format", "operator": "equals", "value": "xhtml"}
      ]}
    ]
  }
}

Check actual advertised link paths before using these illustrative patterns. Document links on another registrable domain also need Include linked sites (follow.scope: linked_sites); follow.allowed_hosts can bound that outward following. Your URLs must satisfy host boundaries.

Python callers can pass a dictionary or the exported CrawlFollow, FollowRule and QueryCondition types:

from periplus_sdk import Client, CrawlFollow, FollowRule, QueryCondition

with Client() as client:
    crawl = client.crawls.create(
        urls=["https://example.com/filings"],
        follow=CrawlFollow(max_depth=3, page_limit=100, rules=[
            FollowRule(path="/filings"),
            FollowRule(path="/document/*", query=[QueryCondition(name="format", value="xhtml")]),
        ]),
    )

Page limits and results

With link following, your URLs count toward follow.page_limit. A crawl finishes when it reaches the limit, has no eligible links left, or is cancelled; a list crawl finishes when every URL is done. The limit is a ceiling, not a promise of that many results.

Crawls can share work with other requests, and may reuse a capture of the same URL made in the last few minutes instead of fetching again. Your crawl still has its own results and page limit. Requests with and without paid bot-challenge resolution never share or reuse each other's captures.

Pages that fail with bot_challenge can be retried with paid challenge resolution: with the user's consent and your organization's allowance, start a crawl without follow listing just those URLs with resolve_bot_challenges: true. It is off by default; see bot challenges for the allowance, billing and refusal. Cancelling stops remaining work; it does not delete captures already collected. SQL availability can lag capture completion. Which pages a crawl produced is available from its page list, not from SQL.

Crawl from a script

This starts a crawl. Acceptance does not mean the captures are queryable. Supply your own stable UUID as id when you need to recover a create request after losing its response. An optional name and Markdown description only label the crawl and are kept from the first create on a retry; rename it later with crawls.update(crawl, name=...) or PATCH /crawls/{id} with expected_version.

Python

from periplus_sdk import Client

with Client() as client:
    crawl = client.crawls.create(
        urls=["https://example.com/research/"],
        follow={"max_depth": 2, "page_limit": 500, "rules": [{"path": "/research/*"}]},
        name="Research archive",
    )
    listed = client.crawls.create(urls=["https://example.com/a", "https://example.com/b"])
    print(crawl.id, crawl.status, listed.crawl.follow)  # None: a list crawl follows no links

cURL

curl --fail-with-body "${PERIPLUS_API_URL:-https://api.periplus.dev}/crawls" \
  -H "Authorization: Bearer $PERIPLUS_API_KEY" \
  -X POST \
  -H 'Content-Type: application/json' \
  --data '{"urls":["https://example.com/research/"],"follow":{"max_depth":2,"page_limit":500,"rules":[{"path":"/research/*"}]}}'

Read progress and captures

Set CRAWL_ID to the crawl ID returned at creation. The crawl's progress counts captured, failed and pending pages; list failed captures with status=failed to see why each failed. Its captures, successful and failed, are listed newest first with depth and the capture they were linked from; failed captures keep their capture_id but are never queryable. URLs not yet attempted are counted, not listed. Check queryable before expecting a capture in SQL.

Python

from periplus_sdk import Client

with Client() as client:
    import os
    crawl = client.crawls.get(os.environ["CRAWL_ID"])
    print(crawl.status, crawl.progress, crawl.finish_reason)
    page = client.crawls.captures(crawl, status="failed")
    for item in page.items:
        print(item.url, item.capture_id, item.failure.title if item.failure else None)
    # Pass page.next_cursor to captures() for the next page.

cURL

curl --fail-with-body "${PERIPLUS_API_URL:-https://api.periplus.dev}/crawls/$CRAWL_ID/captures?limit=20" \
  -H "Authorization: Bearer $PERIPLUS_API_KEY"

Start a crawl

Find earlier crawls

Search the crawls list by name, URL or site: the box at the top of Crawls matches any part of the name or of a crawl URL, ignoring case, and combines with the project switcher and the active/finished filter. ⌘K also finds crawls by name or URL. A search needs at least three consecutive letters or digits. To list only crawls whose URLs are on one site, use host, an exact host where a leading www. is ignored (host=example.com also matches www.example.com). Search covers the name and the URLs you supplied, not pages found while crawling. Agents use crawl_list with search or host.

from periplus_sdk import Client

with Client() as client:
    for item in client.crawls.list(search="pricing", finished=True).items:
        print(item.id, item.name)
    by_site = client.crawls.list(host="example.com")

Projects

Every crawl belongs to exactly one project, and projects are how you organize crawls. Your organization starts with a Default Project, and a crawl created without a project lands there, so nothing changes for work that never mentions projects. Create projects for separate pieces of work, such as one per product or customer, and keep each crawl with the queries, pins and schedules around it. Projects organize your work, not your data: SQL reads every capture your organization can see, whichever project a query is in.

Choose a project with the switcher in the sidebar: Crawls lists that project's crawls, and new crawls go into it. To move a crawl, use Move… on its page, or select several and choose Move to project. Moving never changes what a crawl does. Like the name, the project is presentation: a create retry with the same ID keeps the project it was first created with. From the API, pass project_id when creating a crawl, or move it later with project_id and its current version as expected_version. Agents find project IDs with projects_list and pass project_id to crawl_create and crawl_list, or move crawls with project_move_items; there is no active project, so name it on every call.

from periplus_sdk import Client

with Client() as client:
    default_project = next(client.projects.iter())    # listed first
    project = client.projects.create(name="Product A")
    client.crawls.create(urls=["https://example.com/"], follow={"max_depth": 2, "page_limit": 200}, project_id=project)
    for crawl in client.crawls.list(project_id=default_project).items:
        client.crawls.move(crawl, project_id=project)
    print(client.projects.get(project).item_counts.crawls)

Related guides

• Captures · instant feedback on one URL
• Pins · keep captures longer
• Usage and billing

Waiting and estimates

The crawl detail shows approximate time remaining and next activity when recent progress is available. Ranges can grow as new links are found. Open the explanation for the known-work estimate and website bottleneck. Page limits are ceilings, not completion percentages. Capture completion and SQL readiness are separate.

Queue admission checks the starting websites, so a busy unrelated website does not delay your request. If a crawl is refused because those websites have a long queue (crawl_queue_delay), nothing is accepted. You can retry with the longest wait you accept before it starts: choose a longer wait in the form, or pass accepted_queue_wait_seconds in the API, SDK and MCP (by default a crawl accepts up to 1 hour; at most 30 days). Agree a longer wait with the user first. POST /access/crawl-preview with urls checks these websites without consuming admission. This is an estimate, not a reservation or a promised start time.