The Recrawl option allows you to instantly crawl new pages after an update. This ranges from content revamp across pages to new product additions. It immediately reflects in the updated results when a relevant search is made.
While the scheduled crawling detects the new content during their routine process, the recrawl option allows you to manually trigger an instant crawling to detect and index fresh additions with no time gap.
Recrawl Window Options
The Keyspider Recrawl Window includes the following options:
Recrawl Window
The following table describes the Recrawl parameters.
| Field | Description |
|---|---|
| Add new data source domain | The option is ideal when there are changes across the defined data source(s). A full-fledged crawling will be conducted to ensure that all the additions are reflected. Since the entire data source is undergoing crawling, this process would take a while to reflect in the search results. |
| Remove the data source | The option allows you to remove data source(s) that you’d like to remove from indexing. |
| Add specific URLs to index | When changes are made to specific URL(s), the option allows you to add the URL(s) for instant crawling. You will witness instant updation and reflection of results in the searches. |
| Remove specific URLs | Similarly, the option allows you to remove specific URLs that you’d like to remove from indexing. |
| Re-crawl selected data source(s) | The option allows you to choose and select the data sources to crawl, refresh, and update the existing record. |
| Recrawl all | The ‘Re-crawl all’ option allows you to crawl all the data sources in the collection after updating the existing page. |
Note: The option is available only for the existing pages.