Description
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
Downloads
13.8K
Last 30 days
1013th
13.8K
Last 90 days
13.8K
Last year
CRAN Check Status
2
NOTE
12
OK
Show all 14 flavors
| Flavor | Status |
|---|---|
| r-devel-linux-x86_64-debian-clang | OK |
| r-devel-linux-x86_64-debian-gcc | OK |
| r-devel-linux-x86_64-fedora-clang | OK |
| r-devel-linux-x86_64-fedora-gcc | OK |
| r-devel-macos-arm64 | OK |
| r-devel-windows-x86_64 | OK |
| r-oldrel-macos-arm64 | NOTE |
| r-oldrel-macos-x86_64 | NOTE |
| r-oldrel-windows-x86_64 | OK |
| r-patched-linux-x86_64 | OK |
| r-release-linux-x86_64 | OK |
| r-release-macos-arm64 | OK |
| r-release-macos-x86_64 | OK |
| r-release-windows-x86_64 | OK |
Check details (14 non-OK)
OK
r-devel-linux-x86_64-debian-clang
*
OK
r-devel-linux-x86_64-debian-gcc
*
OK
r-devel-linux-x86_64-fedora-clang
*
OK
r-devel-linux-x86_64-fedora-gcc
*
OK
r-devel-macos-arm64
*
OK
r-devel-windows-x86_64
*
NOTE
r-oldrel-macos-arm64
installed package size
installed size is 6.5Mb
sub-directories of 1Mb or more:
libs 5.7Mb
NOTE
r-oldrel-macos-x86_64
installed package size
installed size is 6.6Mb
sub-directories of 1Mb or more:
libs 5.9Mb
OK
r-oldrel-windows-x86_64
*
OK
r-patched-linux-x86_64
*
OK
r-release-linux-x86_64
*
OK
r-release-macos-arm64
*
OK
r-release-macos-x86_64
*
OK
r-release-windows-x86_64
*
Check History
NOTE 12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 9, 2026
NOTE
r-oldrel-macos-arm64
installed package size
installed size is 6.5Mb
sub-directories of 1Mb or more:
libs 5.7Mb
NOTE
r-oldrel-macos-x86_64
installed package size
installed size is 6.6Mb
sub-directories of 1Mb or more:
libs 5.9Mb
Dependency Network
Version History
new
0.2.1
Mar 9, 2026