Adding a website and how crawling works

Your workspace has a single assistant. The first website you add becomes its primary domain and carries the chat widget. Any additional domains you add feed the same shared knowledge base but don’t get a widget of their own, they’re knowledge sources, not separate bots. This keeps one assistant consistent across every property you own instead of fragmenting into several.

SiteMind fetches your pages, reads the rendered HTML, and splits the text into chunks small enough to search accurately. Each chunk gets embedded so the assistant can find the right passage for a given question later. Crawling respects your site’s pace: requests go out with a short delay between them rather than all at once.

Each plan caps how many pages a single crawl can index: 100 pages on Starter, 500 on Growth, 2,000 on Pro, 5,000 on Enterprise. If your site is larger than your limit, the crawl still completes, it just stops at the cap rather than failing outright. See Billing and plans for the full plan comparison.

You can trigger a manual recrawl from a website’s detail page any time your content changes. There’s also an optional automatic schedule: daily, weekly, or monthly. It’s off by default, since re-embedding changed pages uses AI quota, so turning it on is a deliberate choice rather than something switched on for you.

SiteMind’s crawler reads the HTML your server sends, not what JavaScript builds afterward. If your site is a single-page app that renders its content client-side, the crawler may see little more than an empty page shell, and you’ll get a clear warning about it rather than a silently thin knowledge base. Until headless-browser crawling ships, the most reliable fix is to add key content as static pages, uploaded documents, or manual entries (see Knowledge sources).