crawlee

0.1.0

Tidy Interface for Reproducible Web Crawling

0packages depend

143downloads / year

–test coverage

13/13checks pass

Overview

About

Maintained by Andre LeiteFirst published 2026-07-031 releasesCRAN page ↗GitHub ↗

A tidy, pipe-friendly toolkit for reproducible web crawling and structured data collection, inspired by the architecture of the 'Crawlee' library. Provides a unified crawler with a deduplicating, resumable request queue, content-type aware handlers, structured storage backends and rich console logging via 'cli'. Supports crawling HTML pages, sitemaps, RSS and Atom feeds and PDF documents, with optional headless-browser rendering and helpers for retrieval-augmented generation.

Install

R (CRAN)pak (dev)Fedora (COPR)openSUSE (OBS)

install.packages("crawlee", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("StrategicProjects/crawlee")

Development version from StrategicProjects/crawlee. Installs pak first if it is not already available.

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-crawlee

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-crawlee

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed).

Health

CRAN checks

13OK

Slowest check: 3.0 min · r-devel-linux-x86_64-fedora-clang

Code health

Yes

Tests · ratio 0.24

not tracked

Coverage

100%

Documentation · exports

Yes

Vignettes

Yes

README

Dependencies · direct

Check history

OK2026-07-04
7 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE

Downloads

143

CRAN downloads in the past year

Rank #24,297 · ~0/day · ~12/mo

crawlee Releases R versions

Daily download trend is not available in this view yet.

14330 days

14390 days

1431 year

Compare downloads with other packages →

Repository

GitHub ↗ strategicprojects/crawlee

1Stars

0Forks

0Open issues

0Open PRs

0Releases

23Commits

2Contributors

crawleecrawlergovtechhttr2open-datarrstatsrvest

23 commits · Last activity 2026-06-29

Dependencies

Declared dependencies

23 external dependencies (excludes base and recommended)

Depends (1)

R >= 4.1.0

Imports (8)

cli httr2 R6 rlang rvest tibble vctrs xml2

LinkingTo (0)

none

Suggests (15)

arrow chromote DBI dplyr duckdb httptest2 jsonlite knitr later nanoparquet pdftools polite promises rmarkdown testthat

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

0direct

0indirect

Nothing depends on this yet.

Code & Tests

Code Composition

R 2,942 (59%)Rd 1,482 (30%)Vignettes 598 (12%)

Code characteristics

Object systems

S31R64

Cyclomatic complexity: 2.0 median / 15 max
Test cases: 56 / 0.24 per code line
Documented parameters: 93%

Test coverage

Line coverage

–

Expression

–

Tests / Examples

–

Functions

50 21 exported

Complexity

3.8 avg / 15 max

Call network

50 nodes / 32 edges

Test coverage has not been measured for this package yet; nodes fall back to a neutral fill.

Call graph

Open call graph →

Loading call graph…

Lowest coverage

Per-function coverage is not measured for this package yet.

People & History

People (5)

Maintainer (1)

Andre Leite

Author, Maintainer

Authors (5)

Author, Maintainer

Author

Author

Author

Author

Package Timeline

1 release. R releases are shown for context.

0.1.0Latest
2026-07-03 · current release
R
R 4.6.0 released · 2026-04-24

Package metadata

First published: 2026-07-03
Total releases: 1 / 1 yrs
License: MIT + file LICENSE OSI
Minimum R: ≥ 4.1.0
Download size: not tracked yet
Installed size: not tracked yet
With dependencies: not tracked yet