Website crawling
Point Relay at a URL, and keep the index tracking your site as it changes.
A crawl starts at a URL and follows links from there, extracting readable text from each page.
Starting one
Add the site as a knowledge source and start the crawl. It runs as a background job: a large site takes minutes, and the bot keeps answering from whatever is already indexed while it runs.
Page limits
Crawls are capped at a configurable page count. The cap exists because most sites have a long tail of pages that answer nothing (archives, tag pages, paginated lists), and indexing them dilutes retrieval.
Point the crawl at the section that answers questions rather than the domain root when the domain is large.
Re-crawling
Scheduled re-crawls are what keep answers current. A bot trained once against a site that has since changed will quote last quarter's pricing with complete confidence.
What does not get crawled
Content behind a login, rendered only by client-side JavaScript after
interaction, or excluded by robots.txt. If an answer is missing and the page
exists, check which of those three applies before assuming the crawl failed.