rtiktoken
A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
v0.0.7
·
Apr 14, 2025
·
MIT + file LICENSE
Description
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
Downloads
163
Last 30 days
20034th
163
Last 90 days
163
Last year
CRAN Check Status
3
NOTE
11
OK
Show all 14 flavors
| Flavor | Status |
|---|---|
| r-devel-linux-x86_64-debian-clang | OK |
| r-devel-linux-x86_64-debian-gcc | OK |
| r-devel-linux-x86_64-fedora-clang | OK |
| r-devel-linux-x86_64-fedora-gcc | OK |
| r-devel-macos-arm64 | OK |
| r-devel-windows-x86_64 | OK |
| r-oldrel-macos-arm64 | NOTE |
| r-oldrel-macos-x86_64 | NOTE |
| r-oldrel-windows-x86_64 | NOTE |
| r-patched-linux-x86_64 | OK |
| r-release-linux-x86_64 | OK |
| r-release-macos-arm64 | OK |
| r-release-macos-x86_64 | OK |
| r-release-windows-x86_64 | OK |
Check details (14 non-OK)
OK
r-devel-linux-x86_64-debian-clang
*
OK
r-devel-linux-x86_64-debian-gcc
*
OK
r-devel-linux-x86_64-fedora-clang
*
OK
r-devel-linux-x86_64-fedora-gcc
*
OK
r-devel-macos-arm64
*
OK
r-devel-windows-x86_64
*
NOTE
r-oldrel-macos-arm64
installed package size
installed size is 12.7Mb
sub-directories of 1Mb or more:
libs 12.6Mb
NOTE
r-oldrel-macos-x86_64
installed package size
installed size is 12.5Mb
sub-directories of 1Mb or more:
libs 12.4Mb
NOTE
r-oldrel-windows-x86_64
installed package size
installed size is 9.8Mb
sub-directories of 1Mb or more:
libs 9.7Mb
OK
r-patched-linux-x86_64
*
OK
r-release-linux-x86_64
*
OK
r-release-macos-arm64
*
OK
r-release-macos-x86_64
*
OK
r-release-windows-x86_64
*
Check History
NOTE 11 OK · 3 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 9, 2026
NOTE
r-oldrel-macos-arm64
installed package size
installed size is 12.7Mb
sub-directories of 1Mb or more:
libs 12.6Mb
NOTE
r-oldrel-macos-x86_64
installed package size
installed size is 12.5Mb
sub-directories of 1Mb or more:
libs 12.4Mb
NOTE
r-oldrel-windows-x86_64
installed package size
installed size is 9.8Mb
sub-directories of 1Mb or more:
libs 9.7Mb
Version History
new
0.0.7
Mar 10, 2026