Issue:
I read about Fusion crawlDB. What is it and how can I leverage it?
Environment:
Fusion
Resolution:
CrawlDB is a meta data store which is created per datasource for Anda Crawlers. It stores meta data about all the URLs crawled so that the Anda crawler can optimize the consecutive crawls. It can monitor the details like lastModified, lastFetched, depth etc. so that it can avoid re-crawling all the links on the next delta crawls.
Cause:
Comments
0 comments
Please sign in to leave a comment.