wordpiece

R Implementation of Wordpiece Tokenization

v2.1.3 · Mar 3, 2022 · Apache License (>= 2)

Description

Apply 'Wordpiece' (<arXiv:1609.08144>) tokenization to input text, given an appropriate vocabulary. The 'BERT' (<arXiv:1810.04805>) tokenization conventions are used by default.

Downloads

256

Last 30 days

12458th

256

Last 90 days

256

Last year

CRAN Check Status

14 NOTE

Show all 14 flavors

Flavor	Status	Time
r-devel-linux-x86_64-debian-clang	NOTE	38.8s
r-devel-linux-x86_64-debian-gcc	NOTE	29.3s
r-devel-linux-x86_64-fedora-clang	NOTE	58.1s
r-devel-linux-x86_64-fedora-gcc	NOTE	67.4s
r-devel-macos-arm64	NOTE	18s
r-devel-windows-x86_64	NOTE	65s
r-oldrel-macos-arm64	NOTE
r-oldrel-macos-x86_64	NOTE	37s
r-oldrel-windows-x86_64	NOTE	71s
r-patched-linux-x86_64	NOTE	34.6s
r-release-linux-x86_64	NOTE	34.5s
r-release-macos-arm64	NOTE
r-release-macos-x86_64	NOTE	48s
r-release-windows-x86_64	NOTE	66s

Check details (16 non-OK)

NOTE r-devel-linux-x86_64-debian-clang

CRAN incoming feasibility

Maintainer: ‘Jonathan Bratt <jonathan.bratt@macmillan.com>’

The Description field contains
  Apply 'Wordpiece' (<arXiv:1609.08144>) tokenization to input text,
  given an appropriate vocabulary. The 'BERT' (<arXiv:1810.04805>)
Please refer to arXiv e-prints via their arXiv DOI <doi:10.48550/arXiv.YYMM.NNNNN>.

NOTE r-devel-linux-x86_64-debian-clang

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-linux-x86_64-debian-gcc

CRAN incoming feasibility

Maintainer: ‘Jonathan Bratt <jonathan.bratt@macmillan.com>’

The Description field contains
  Apply 'Wordpiece' (<arXiv:1609.08144>) tokenization to input text,
  given an appropriate vocabulary. The 'BERT' (<arXiv:1810.04805>)
Please refer to arXiv e-prints via their arXiv DOI <doi:10.48550/arXiv.YYMM.NNNNN>.

NOTE r-devel-linux-x86_64-debian-gcc

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-linux-x86_64-fedora-clang

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-linux-x86_64-fedora-gcc

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-macos-arm64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-windows-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-oldrel-macos-arm64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-oldrel-macos-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-oldrel-windows-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-patched-linux-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-linux-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-macos-arm64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-macos-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-windows-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

Check History

NOTE 0 OK · 14 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 9, 2026

NOTE r-devel-linux-x86_64-debian-clang

CRAN incoming feasibility

Maintainer: ‘Jonathan Bratt <jonathan.bratt@macmillan.com>’

The Description field contains
  Apply 'Wordpiece' (<arXiv:1609.08144>) tokenization to input text,
  given an appropriate vocabulary. The 'BERT' (<arXiv:1810.04805>)
Please refer to arXiv e-prints via their arXiv DOI <doi:10.48550/arXiv.YYMM.NNNNN>.

NOTE r-devel-linux-x86_64-debian-gcc

CRAN incoming feasibility

Maintainer: ‘Jonathan Bratt <jonathan.bratt@macmillan.com>’

The Description field contains
  Apply 'Wordpiece' (<arXiv:1609.08144>) tokenization to input text,
  given an appropriate vocabulary. The 'BERT' (<arXiv:1810.04805>)
Please refer to arXiv e-prints via their arXiv DOI <doi:10.48550/arXiv.YYMM.NNNNN>.

NOTE r-devel-linux-x86_64-fedora-clang

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-linux-x86_64-fedora-gcc

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-macos-arm64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-devel-windows-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-patched-linux-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-linux-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-macos-arm64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-macos-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-release-windows-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-oldrel-macos-arm64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-oldrel-macos-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTE r-oldrel-windows-x86_64

Rd files

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

Reverse Dependencies (1)

suggests

textrecipes

Dependency Network

Version History

new 2.1.3 Mar 9, 2026

Maintainer

Jonathan Bratt

Dependencies

Depends

R (>= 3.3.0)

Imports

dlr (>= 1.0.0) fastmatch (>= 1.1) memoise (>= 2.0.0) piecemaker (>= 1.0.0) rlang stringi (>= 1.0) wordpiece.data (>= 1.0.2)

Suggests

covr knitr rmarkdown testthat (>= 3.0.0)

Compilation

No compilation needed

First Published

Mar 9, 2026

RSS Feed

CRAN Checks

View on CRAN →