The crawler fetches pages over HTTP like a search engine would. Use it for content outside WordPress, a subdomain, or an old site you have not migrated yet.
Starting a crawl
- Go to Knowledge → Add & Sources → Add Knowledge → Website URL or Sitemap.
- Paste the starting URL. A sitemap URL is usually better: it does not follow links, so nothing unexpected gets pulled in.
- Set a page limit and start. Progress is shown live and can be paused.
What it skips
- Tag, category, author and date archives, which are duplicate content.
- Shop filter URLs with query strings, which are effectively infinite.
- Anything on another domain, and anything robots.txt disallows.
- Non-HTML files.
Excluding a page
In the crawled-pages table, Remove deletes a page from the knowledge base and excludes it from future crawls. It can be re-included later. This is the right tool for a page that keeps producing bad answers.
Re-crawling is incremental: pages that have not changed cost nothing, and pages that now return 404 are removed from the knowledge base.