v3.0.1

robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

robots.txt parser user-agent scraper spider bot robots-exclusion-standard

Website Repository

Click the AI button to generate exact installation commands and usage examples for your environment.

Dependencies (0)

Zero dependencies

Dev Dependencies (3)

chai ^4.3.6
mocha ^10.0.0
nyc ^15.1.0

Package Info

License

MIT

Last Publish

February 21, 2023

Unpacked Size

45.17 KB

Total Files

10

Issue Tracker

Maintainers (1)

samclarke

samclarke sam@samclarke.com