Zum Inhalt springen
Kitbay
Skill

huggingface-tokenizers

MITAktualisiert am 11. Okt. 202614.431GitHub-Sterne

Orchestra Research's skill for Hugging Face's tokenizers library, the Rust-backed tokeniser package used with Transformers.

Was der Skill vermittelt

Orchestra Research's skill for Hugging Face's tokenizers library, the Rust-backed tokeniser package used with Transformers. It explains when to choose it over alternatives such as SentencePiece or tiktoken, how to load pretrained tokenisers, and how to train custom vocabularies with BPE, WordPiece or Unigram models. It also covers normalisers and pre-tokenisers, padding and truncation, batch encoding, and offset tracking that maps tokens back to the original text. Reference files go into algorithms, the processing pipeline, training and integration with Transformers and Datasets. It assumes the tokenizers, transformers and datasets Python packages.

Funktionen

  • Train custom BPE, WordPiece or Unigram tokenisers
  • Configure padding, truncation and batch encoding
  • Track token offsets back to the source text