Monitoring crawl progress

To see the progress of any crawl:

From the main menu select Data Sources
Select the data source for the crawl you want to investigate
Select the Performance tab

What is it showing me?

Ingesting items into the platform is a complex process that falls into the three phases you can see in the screenshot.

Let’s look at them in more detail.

Discovered

The platform sends out a crawler to find items to ingest. It looks for ‘containers’.

What’s a container?

Containers are the grouping and organisation of items. For example, when you’re crawling Exchange365, a container can be a mailbox or a calendar. Archives such as zips, are not considered to be containers.

For a file share crawl, a container is a folder
For an entire database crawl, a container is a table
For Enterprise Vault crawl, a container is an Enterprise Vault Archive

The platform adds the containers it finds to a queue. Then it checks if any of them hold any items to ingest.

Ultimately, the platform only ingests items it finds. The container counts in the screenshot indicate how the process of discovering those items is succeeding.

Crawled

When we’ve discovered an item, the process of ingesting it into the platform begins. The item goes through processing. This means the platform extracts the metadata and text of the item, as well as any further processing that’s needed, such as PII extraction.

This section might include an error count. There are many potential reasons why NOW Privacy may not be able to analyse an item or container.

For example, the platform might not have the correct permissions to access the item, or there might be technical difficulties in processing the item.

Click the Errors tab to find more information.

If you can’t see an Errors tab yet, the crawl is still in progress. Wait for it to finish.

Analysed

The final phase of ingestion is analysing and indexing the data. After that, you’ll be able to access the data using searches, dashboards, and workflows.