What is Fusion CrawlDB and what can it be used for? – Lucidworks

Issue:

I read about Fusion crawlDB. What is it and how can I leverage it?

Environment:

Fusion

Resolution:

CrawlDB is a meta data store which is created per datasource for Anda Crawlers. It stores meta data about all the URLs crawled so that the Anda crawler can optimize the consecutive crawls. It can monitor the details like lastModified, lastFetched, depth etc. so that it can avoid re-crawling all the links on the next delta crawls.

CrawlDB has the status in a field called map_s (ERRORS_MAP or, FINISHED_MAP).

- Also, it has a field called error_ss which will have the cause of error which you find in the connector log. You can query this log to get links which failed and to get the cause of the error

- http://139.59.72.222:8983/solr/#/crawldb_<collectionname>_shard1_replica1/query

Link which couldn't be crawled (itemID_s)

{

"id": "ERRORS_MAP|http://139.59.72.222/byebye.html",

"crawlCycle_ti": 1,

"map_s": "ERRORS_MAP",

"itemID_s": "http://139.59.72.222/byebye.html",

"discarded_b": false,

"depth_ti": 1,

"fetchedDate_tdt": "2017-03-24T14:55:17.696Z",

"signature_tl": 0,

"lastModified_tl": 0,

"batchID_tl": 1,

"error_ss": [

"crawler.common.CrawlItemException: phase=FETCH; This item failed during fetch()",

"\tat crawler.Crawler$FetchCallable.call(Crawler.java:976)",

"\tat crawler.Crawler$FetchCallable.call(Crawler.java:944)",

"\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)",

"\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)",

"\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)",

"\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)",

"\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)",

"\tat java.lang.Thread.run(Thread.java:745)",

"Caused by: java.io.IOException: Non-OK HTTP status: 404",

"\tat crawler.fetch.impl.http.WebFetcher.fetch(WebFetcher.java:372)",

"\tat crawler.Crawler$FetchCallable.call(Crawler.java:968)",

"\t... 7 more"

"parentID_s": "http://139.59.72.222/",

"linked_b": true,

"_version_": 1562763401784983600

Link which was crawled successfully (itemID_s)

{ "id": "FINISHED_MAP|http://139.59.72.222/about.html",

"crawlCycle_ti": 1,

"map_s": "FINISHED_MAP",

"itemID_s": "http://139.59.72.222/about.html",

"discarded_b": false,

"depth_ti": 1,

"fetchedDate_tdt": "2017-03-24T14:55:17.682Z",

"signature_tl": 0,

"contentSignature_s": "\"37-54b7b2a7ef756\"",

"emittedDate_tdt": "2017-03-24T14:55:18.048Z",

"lastModified_tl": 1490367249000,

"batchID_tl": 1,

"parentID_s": "http://139.59.72.222/",

"linked_b": true,

"_version_": 1562763401784983600

}

CrawlDB is a file structure as part of Fusion, basically by enabling this link we are pushing the records from the CrawlDB file to Solr (Select Datasource --> Advanced --> Crawl History --> Index Crawl Datasource to Solr). It can be leveraged for easy querying and even creating banana dashboards.

Cause:

Related articles