Rules with trailing comments are not being applied correctly. Example: https://ask.fmcsa.dot.gov/robots.txt ``` User-agent: * # ADDED BY HMS Disallow: / # ADDED BY HMS ``` The above rule should disallow everything on the site. Instead: ```ruby > robotex = Robotex.new('testing') > robotex.allowed?('https://ask.fmcsa.dot.gov/anything') true ``` PR: https://github.com/chriskite/robotex/pull/3 Reference: https://developers.google.com/search/reference/robots_txt#file-format
Rules with trailing comments are not being applied correctly.
Example:
https://ask.fmcsa.dot.gov/robots.txt
The above rule should disallow everything on the site. Instead:
PR: #3
Reference: https://developers.google.com/search/reference/robots_txt#file-format