Home

Roadmap

Future features

• Full specification support (wiki)
  • X-Robots-Tag HTTP header
  • <META> tags
  • <noindex> tags
  • Attribute rel="nofollow" of the <a> tag
• Tools for 3rd party
  • Extensive Clean-Param multi-URL parser, for maximum support
• Robots.txt fetcher
  • Automatically download the correct robots.txt
  • Follow any redirects in case of different hosts
  • Enhanced encoding detector and converter
• Caching system
  • Keep the robots.txt files for up to 24 hours, possibly longer in case of HTTP 5xx