sentencepiece

Description

Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling. Wraps the 'sentencepiece' library <https://github.com/google/sentencepiece> which provides a language independent tokenizer to split text in words and smaller subword units. The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) <doi:10.18653/v1/D18-2012>. Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec', as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) <http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf>.

Downloads

CRAN Check Status

  installed size is 21.6Mb
  sub-directories of 1Mb or more:
    libs    19.9Mb
    models   1.6Mb

  installed size is 23.1Mb
  sub-directories of 1Mb or more:
    libs    21.3Mb
    models   1.6Mb

  installed size is  5.0Mb
  sub-directories of 1Mb or more:
    libs     3.3Mb
    models   1.6Mb

Check History

  installed size is 21.6Mb
  sub-directories of 1Mb or more:
    libs    19.9Mb
    models   1.6Mb

  installed size is 23.1Mb
  sub-directories of 1Mb or more:
    libs    21.3Mb
    models   1.6Mb

  installed size is  5.0Mb
  sub-directories of 1Mb or more:
    libs     3.3Mb
    models   1.6Mb

Reverse Dependencies (1)

Dependency Network

Version History