What is Fusion CrawlDB and what can it be used for? – Lucidworks

Environment:

Fusion

Description:

In Fusion, the concept of a "CrawlDB" refers to a specialised database or data store that is used to manage information related to web crawling and indexing activities. The CrawlDB keeps track of various details about URLs, their crawling status, metadata, and other relevant information. This concept is similar to what is commonly referred to as a "Crawl Database" in the context of web crawling and scraping.

Here's what the CrawlDB is and how it's used in Lucidworks Fusion:

Crawling Management: The CrawlDB in Lucidworks Fusion is responsible for managing the crawling process. It keeps track of which URLs have been visited, which ones are pending, and which ones have been successfully crawled. This helps prevent duplicate crawling and ensures that all relevant content is properly indexed.
URL Tracking: The CrawlDB maintains a record of URLs encountered during the crawling process. This includes information such as the URL itself, its status (crawled, pending, etc.), last crawl time, and any associated metadata.
Deduplication: Duplicate content is a common challenge in web crawling. The CrawlDB helps identify and manage duplicate URLs, ensuring that the same content isn't indexed multiple times.
Crawl Scheduling: The CrawlDB can be used to manage the scheduling of URLs for crawling. It helps prioritize which URLs to crawl next based on factors like recency, importance, or relevance.
Crawl Metrics and Monitoring: The CrawlDB stores information about the crawling process, such as the number of URLs crawled, the success rate, and any errors encountered. This allows administrators to monitor the health and effectiveness of the crawling activities.
Integration with Indexing: The information stored in the CrawlDB is crucial for feeding data into the indexing process. It helps determine what content needs to be indexed, what changes have occurred since the last crawl, and which URLs should be updated or reindexed.
Crawl Policies and Rules: The CrawlDB can be used to enforce crawl policies and rules. For example, it can define how frequently certain websites or pages should be crawled, or whether certain URLs should be excluded from crawling altogether.
Data Enrichment: The CrawlDB can also store additional metadata or information about each URL, such as the content type, language, and any custom attributes that might be useful for indexing and searching.

In summary, the CrawlDB in Fusion is a core component for managing and tracking the web crawling process. It helps ensure efficient and effective content discovery, indexing, and maintenance within a search and data application built using Fusion.

Working with the crawl database:

Some of the connectors use a crawl database to track documents that have been seen by prior crawls and are able to use this information to understand which documents are new or have been updated or removed and take appropriate action in the index. The /api/connectors/datasources/{id}/db endpoints allow looking into the crawl database and dropping tables or clearing the database.

The connectors that support the crawl database are currently lucid.fs and lucid.solrxml. The lucid.anda connector also uses a crawl database, but it is not the same database, and does not have a REST API or other interface to access it.

Examining a crawlDB

The output from a GET request to /api/connectors/datasources/<id>/db will include several sections detailing the database structure:

counters: The counters section reports the document counts of database activities, such as table inserts.
ops: The ops section reports on database operations that have occurred, such as initiating tables, retrieving items, processing items and table drops.
tables: The tables section lists the tables in the database with a count of the number of items in each table. Inspecting the items is described in the next section.

Drop tables from a crawlDB

The output from a DELETE request to /api/connectors/datasources/<id>/db/<table> will be empty. When dropping the database, note that no documents will be removed from the index. However, the crawl database will be empty, so on the next datasource run, all documents will be treated as though they were never seen by the connectors.

When dropping tables, be aware that the items table does not delete documents from the index, but instead changes the database so database considers them new documents. When dropping other tables, such as the errors table, it will merely clear out old error messages.

Get or delete items from a crawlDB

A GET request to /api/connectors/datasources/<id>/db/items/<item> retrieves information about an item or items.

A DELETE request removes the information from the Crawl Database only. Note that this does not affect the Solr Index.

Examples

Get datasources assigned to the "demo" collection:

REQUEST

curl -u USERNAME:PASSWORD https://FUSION_HOST:8764/api/connectors/datasources?collection=demo

RESPONSE

[ {
  "id" : "database",
  "created" : "2014-05-04T19:47:22.867Z",
  "modified" : "2014-05-04T19:47:22.867Z",
  "connector" : "lucid.jdbc",
  "type" : "jdbc",
  "description" : null,
  "pipeline" : "conn_solr",
  "properties" : {
    "db" : null,
    "commit_on_finish" : true,
    "verify_access" : true,
    "sql_select_statement" : "select CONTENTID as id from CONTENT;",
    "debug" : false,
    "collection" : "demo",
    "password" : "password",
    "url" : "jdbc:postgresql://FUSION_HOST:5432/db",
    "nested_queries" : null,
    "clean_in_full_import_mode" : true,
    "username" : "user",
    "delta_sql_query" : null,
    "commit_within" : 900000,
    "primary_key" : null,
    "driver" : "org.postgresql.Driver",
    "max_docs" : -1
  }
} ]

Cause: