Extract page data

Use capture_id with all HTML helpers. node_index is local to that capture's tree. Select the intended captures by URL/host before extracting; see docs_read('sql').

For JSON endpoints, join documents by capture_id and use native ClickHouse JSON functions and ARRAY JOIN over documents.text. Check format and parse_status; see docs_read('json'). JSON fields are not CSS elements, inferred links or prose.

For repeated cards, query_selector_all(capture_id, '.card') gives node identities. Extract fields inside the appropriate subtree using the helper signatures from schema_get. Do not join nodes from different captures by node_index alone. A JOIN to nodes stays capture-bounded when a literal capture_id reaches it through plain INNER joins (WHERE l.capture_id = ? with ON n.capture_id = l.capture_id), or when nodes is filtered by n.capture_id IN (SELECT capture_id FROM captures WHERE url LIKE ?). For other shapes, or after a memory error, call node_text/node_html on the explicit capture_id/node_index pairs instead of joining nodes. To get the container of a match (its row, card or section), use closest(capture_id, node_index, selector) rather than repeated parent_index joins.

CSS adjacent (+) and following (~) sibling combinators work on elements, ignoring intervening whitespace text nodes:

SELECT capture_id,
       node_text(capture_id, query_selector(capture_id, 'p.title + p')) AS value
FROM captures WHERE url = ? ORDER BY observed_at DESC LIMIT 1

The selector must identify the intended label by structure or attributes. It does not match label text automatically. For many labels, retrieve label and value nodes, then pivot after extraction. Never guess node_index + 2 or + 3: preorder positions change with page markup. sibling_index counts all parsed siblings.

Metadata can be selected with title, meta[name="description"], meta[property="article:published_time"], link[rel="canonical"], and script[type="application/ld+json"]. Read attributes from nodes at the selected capture_id/node_index. JSON-LD can be an object, array or @graph, and malformed JSON needs an isValidJSON guard. Preserve the source of any date: publication, modification, capture time and a date guessed from body text are different facts. There is no authoritative publication date merely because a date was found.

Before reporting a listing as complete, compare its stated total ("240 results") with the cards rendered in the capture. A shortfall is a discovery gap: Periplus does not paginate or click "Load more". Some sites expose paging in URL parameters (?page=2, ?offset=40) that can be captured explicitly. Lazy-loaded tabs/panels and client-side search results may be empty in the stored HTML; record them as gaps, not as absence.

To compare a section over time, select two captures of the same page_id ordered by observed_at and compare the chosen section. node_text/documents.text preserve parsed tree text, including script/style content; they are not visible prose. A changed document hash or text hash does not prove the human-visible section changed. Search uses a separate conservative prose representation. A dedicated node-prose helper is not currently available.