Search and phrase counts

search('artificial intelligence') ranks token matches and returns at most 100 distinct URLs, using each URL's latest published complete HTML capture. Latest means observed_at descending (NULL last), then capture_id descending to break ties. Selection happens before matching, so a match only in an older capture is omitted. search_matches returns all matching captures within query/result budgets. Both match ANY non-stopword token: artificial OR intelligence, not an exact phrase and not an AND. Phrase bonuses affect ranking, not mandatory matching.

Both helpers accept two optional named options. host = 'example.com' keeps only captures whose URL host equals that string exactly (the same value as pages.host; no scheme, path or subdomain matching). phrase = true additionally requires the whole query to appear, case-insensitively, in the title or the searchable text. Phrase mode still needs at least one non-stopword token and does not change scores. Options apply before the top-100 cap. Literals and ? parameters are accepted.

SELECT s.score, c.url, s.snippet
FROM search('artificial intelligence', host = 'example.com', phrase = true) s
JOIN captures c USING (capture_id)
ORDER BY s.score DESC

SELECT domain(c.url) AS host, count() AS captures_with_either_word
FROM search_matches('artificial intelligence') s
JOIN captures c ON c.capture_id = s.capture_id
GROUP BY host

Without phrase = true, never label this count as exact phrase mentions. Count distinct page_id for pages, or distinct document_id for exact bytes, if that is the question. Repeated captures count separately. Do not use the top-100 ranked search for corpus-wide coverage counts.

documents.visible_text exposes the prose used by search snippets. It excludes head, script, style, template, comments, nav and role="navigation" subtrees, plus hidden and aria-hidden="true" subtrees, and normalizes whitespace. Roles are case-insensitive and trimmed. Ordinary prose and pricing text remain. Empty prose is an empty string; search snippets are at most 240 characters and NULL for empty prose. Titles also participate in matching and ranking.

SELECT c.url, d.visible_text
FROM captures c JOIN documents d USING (capture_id)
WHERE c.url = 'https://example.com/pricing'
ORDER BY c.observed_at DESC NULLS LAST, c.capture_id DESC
LIMIT 1

This is static structural visibility: CSS, browser layout and unmarked navigation are not interpreted. documents.text and node_text retain full tree text, including script/JSON content. Phrase checks on visible_text describe normalized prose, not browser rendering; token intersection alone is not adjacent phrase matching.