Added

• Support obeyRobotsTxt for crawler.queue()'s options.
• Support persist for RedisCache's constructing options.

changed

• Make cache to be required for HCCrawler.connect() and HCCrawler.launch()'s options.
• Provide skipDuplicates to remember and skip duplicate URLs, instead of passing null to cache option.
• Modify BaseCache interface.