robotstxt

0.7.15

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

7packages depend

37Kdownloads / year

86.2%test coverage

12/13checks pass

Overview

About

Maintained by Pedro BaltazarFirst published 2016-02-0812 releasesCRAN page ↗GitHub ↗

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("robotstxt", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("ropensci/robotstxt")

Development version from ropensci/robotstxt. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-robotstxt

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-robotstxt

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-robotstxt

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed).

Health

CRAN checks

1NOTE12OK

Failing flavors

NOTE r-devel-linux-x86_64-debian-gcc

Slowest check: 1.4 min · r-oldrel-windows-x86_64

Code health

Yes

Tests · ratio 1.31

86.2%

Coverage · measured lines

95%

Documentation · exports

Dependencies · direct

Check history

NOTE2026-08-01
12 OK · 1 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
OK2026-03-10
14 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE

Documentation

READMENoVignettesYes · dynamicpkgdown siteNoNEWSYes · 67% structuredCode of conductNoContributing guideNo

Examples that run: 0%
Documented parameters: 98%
Return-value docs: 44%
References docs: 0%

Downloads

37K

CRAN downloads in the past year

Rank #3,239 · ~101/day · ~3.1K/mo

Releases R versions

Daily download trend is not available in this view yet.

1.4K30 days

3.7K90 days

37K1 year

Compare downloads with other packages →

Also on292 r2u3 autocran926 conda_forge68 c2d4u

Repository

GitHub ↗ ropensci/robotstxt

69Stars

11Forks

1Open issues

0Open PRs

7Releases

315Commits

11Contributors

robotstxtcrawlerwebscrapingspiderscraperrrstatsr-package

315 commits · Last activity 2024-11-15 · +1.5% stars, 30d

Stars over time

2025-03-27 · 672026-07-07 · 69

Repository practices

Upstream repositoryBeta

6 development-tooling and community-health practices detected across 5 families in the upstream repository

Checks run against github.com/ropensci/robotstxt on 2026-07-31.

Continuous integration (1)

GitHub Actions

Reproducibility and dev environment (1)

data-raw/

CRAN release process (2)

cran-comments.mdCRAN-SUBMISSION

Docs source (1)

README.Rmd

Show all practices

Lint, format, editor (1)

RStudio project

How this is detected·Detection ruleset v1 (2026-07-18)

Dependencies

Declared dependencies

11 external dependencies (excludes base and recommended)

Depends (1)

R >= 3.0.0

Imports (6)

stringr httr spiderbar future.apply magrittrutils

LinkingTo (0)

none

Suggests (6)

knitr rmarkdown dplyr testthat covr curl

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

7direct

2indirect

BAwiR polite ralger newsanchor spiderbar vosonSML webchem

Code & Tests

People & History

People (5)

Maintainer (1)

Pedro Baltazar

Author, Maintainer · added in 0.7.15

Authors (3)

Pedro Baltazar

Author, Maintainer · added in 0.7.15

Peter Meissner

Author

Kun Ren

Author, Copyright holder · added in 0.7.4

Contributors (2)

Rich Fitz John

Contributor · added in 0.3.2

Oliver Keys

Contributor

Kun Ren

Author, Copyright holder · added in 0.7.4

Package Timeline

12 releases. Pick two to compare their code metrics. R releases are shown for context.

R
R 4.6.0 released · 2026-04-24
R
R 4.5.0 released · 2025-04-11
0.7.15Latest
2024-08-29 · current release · diff ↗
R
R 4.4.0 released · 2024-04-24
R
R 4.3.0 released · 2023-04-21
R
R 4.2.0 released · 2022-04-22
R
R 4.1.0 released · 2021-05-18
0.7.13
2020-09-03 · diff ↗
0.7.8
2020-07-25 · diff ↗
0.7.7
2020-06-27 · diff ↗
0.7.4
2020-05-31 · diff ↗
R
R 4.0.0 released · 2020-04-24
R
R 3.6.0 released · 2019-04-26
0.6.2
2018-07-18 · diff ↗
R
R 3.5.0 released · 2018-04-23
0.6.0
2018-02-11 · diff ↗

Show 8 earlier events

0.5.2
2017-11-12 · diff ↗
0.4.1
2017-09-01 · diff ↗
0.4.0
2017-07-16 · diff ↗
R
R 3.4.0 released · 2017-04-21
0.3.2
2016-12-05 · diff ↗
R
R 3.3.0 released · 2016-05-03
0.1.2
2016-02-08
R
R 3.2.0 released · 2015-04-16

Package metadata

First published: 2016-02-08
Total releases: 12 / 10 yrs
License: MIT + file LICENSE OSI
Minimum R: ≥ 3.0.0
Download size: 131 KB
Installed size: not tracked yet
With dependencies: not tracked yet

Appears in task views

Web Technologies and Services

Report a problem with this page →