spiderbar

0.2.5

Parse and Test Robots Exclusion Protocol Files and Rules

1packages depend

36Kdownloads / year

23.6%test coverage

11/13checks pass

Overview

About

Maintained by Bob RudisFirst published 2017-09-256 releasesCRAN page ↗GitHub ↗

The 'Robots Exclusion Protocol' https://www.robotstxt.org/orig.html documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' https://github.com/seomoz/rep-cpp C++ library for processing these 'robots.txt' files.

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("spiderbar", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("hrbrmstr/spiderbar")

Development version from hrbrmstr/spiderbar. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-spiderbar

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-spiderbar

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-spiderbar

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed).

Health

CRAN checks

2NOTE11OK

Failing flavors

NOTE r-devel-linux-x86_64-debian-clang
NOTE r-devel-linux-x86_64-debian-gcc

Slowest check: 1.8 min · r-oldrel-windows-x86_64

Code health

Yes

Tests · ratio 0.02

23.6%

Coverage · measured lines

100%

Documentation · exports

Dependencies · direct

Check history

NOTE2026-04-22
12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-04-18
11 OK · 2 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE
NOTE2026-03-10
12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE

Documentation

READMENoVignettesNopkgdown siteNoNEWSNoCode of conductNoContributing guideNo

Examples that run: 100%
Documented parameters: 100%
Return-value docs: 100%
References docs: 0%

Downloads

36K

CRAN downloads in the past year

Rank #3,311 · ~99/day · ~3K/mo

Releases R versions

Daily download trend is not available in this view yet.

1.1K30 days

3.7K90 days

36K1 year

Compare downloads with other packages →

Also on569 r2u19 autocran1.6K conda_forge72 c2d4u

Repository

GitHub ↗ hrbrmstr/spiderbar

9Stars

2Forks

0Open issues

1Open PRs

1Releases

24Commits

3Contributors

rrstatsrobots-exclusion-protocolrobots-txtr-cyber

24 commits · Last activity 2024-07-26

Stars over time

2019-09-23 · 92026-07-07 · 9

Repository practices

Upstream repositoryBeta

6 development-tooling and community-health practices detected across 5 families in the upstream repository

Checks run against github.com/hrbrmstr/spiderbar on 2026-07-19.

Continuous integration (2)

Travis CIAppVeyor

CRAN release process (1)

cran-comments.md

Docs source (1)

README.Rmd

Lint, format, editor (1)

RStudio project

Show all practices

Coverage (1)

Codecov

How this is detected·Detection ruleset v1 (2026-07-18)

Dependencies

Declared dependencies

5 external dependencies (excludes base and recommended)

Depends (1)

R >= 3.2.0

Imports (1)

Rcpp

LinkingTo (1)

Rcpp

Suggests (3)

covr robotstxt tinytest

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

1direct

5indirect

robotstxt

Code & Tests

People & History

People (0)

Author records are not tracked yet for this package.

Listed in earlier versions (3)

Bob Rudis (bob@rud.is)

no longer listed · 0.2.0 to 0.2.5

Inc

no longer listed · 0.2.0 to 0.2.5

SEOmoz

no longer listed · 0.2.0 to 0.2.5

Package Timeline

6 releases. Pick two to compare their code metrics. R releases are shown for context.

R
R 4.6.0 released · 2026-04-24
R
R 4.5.0 released · 2025-04-11
R
R 4.4.0 released · 2024-04-24
R
R 4.3.0 released · 2023-04-21
0.2.5Latest
2023-02-11 · current release · diff ↗
R
R 4.2.0 released · 2022-04-22
R
R 4.1.0 released · 2021-05-18
0.2.4
2021-05-16 · diff ↗
0.2.3
2020-05-30 · diff ↗
R
R 4.0.0 released · 2020-04-24
0.2.2
2019-08-19 · diff ↗
R
R 3.6.0 released · 2019-04-26
R
R 3.5.0 released · 2018-04-23
0.2.1
2017-11-17 · diff ↗
0.2.0
2017-09-25
R
R 3.4.0 released · 2017-04-21

Package metadata

First published: 2017-09-25
Total releases: 6 / 9 yrs
License: MIT + file LICENSE OSI
Minimum R: ≥ 3.2.0
Download size: 31 KB
Installed size: not tracked yet
With dependencies: not tracked yet

Appears in task views

Web Technologies and Services

Report a problem with this page →