robotstxt
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
v0.7.15
·
Aug 29, 2024
·
MIT + file LICENSE
Description
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Downloads
1.9K
Last 30 days
2872nd
1.9K
Last 90 days
1.9K
Last year
CRAN Check Status
14
OK
Show all 14 flavors
| Flavor | Status |
|---|---|
| r-devel-linux-x86_64-debian-clang | OK |
| r-devel-linux-x86_64-debian-gcc | OK |
| r-devel-linux-x86_64-fedora-clang | OK |
| r-devel-linux-x86_64-fedora-gcc | OK |
| r-devel-macos-arm64 | OK |
| r-devel-windows-x86_64 | OK |
| r-oldrel-macos-arm64 | OK |
| r-oldrel-macos-x86_64 | OK |
| r-oldrel-windows-x86_64 | OK |
| r-patched-linux-x86_64 | OK |
| r-release-linux-x86_64 | OK |
| r-release-macos-arm64 | OK |
| r-release-macos-x86_64 | OK |
| r-release-windows-x86_64 | OK |
Check details (14 non-OK)
OK
r-devel-linux-x86_64-debian-clang
*
OK
r-devel-linux-x86_64-debian-gcc
*
OK
r-devel-linux-x86_64-fedora-clang
*
OK
r-devel-linux-x86_64-fedora-gcc
*
OK
r-devel-macos-arm64
*
OK
r-devel-windows-x86_64
*
OK
r-oldrel-macos-arm64
*
OK
r-oldrel-macos-x86_64
*
OK
r-oldrel-windows-x86_64
*
OK
r-patched-linux-x86_64
*
OK
r-release-linux-x86_64
*
OK
r-release-macos-arm64
*
OK
r-release-macos-x86_64
*
OK
r-release-windows-x86_64
*
Check History
OK 14 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 9, 2026
Reverse Dependencies (8)
suggests
Dependency Network
Version History
new
0.7.15
Mar 9, 2026