What kind of crawler is this?
It's a crawl crawler.
What's a crawl crawler?
A crawl crawler is a search engine that utilizes indices that may be enriched with more data, in this case,
textual web content, by crawling already crawled data or by fetching it from WWW.
Whatever, can I do keyword based search here?
Will you honor my robots.txt?
Yes, we will, to the best of our ability. Some of y'all don't know how to write a proper robots file, though.
Where can I read a god damn manual?