本文へスキップ
Kitbay
スキル

huggingface-tokenizers

MIT2026年10月11日 更新14,431GitHubスター

Orchestra Research's skill for Hugging Face's tokenizers library, the Rust-backed tokeniser package used with Transformers.

スキルの内容

Orchestra Research's skill for Hugging Face's tokenizers library, the Rust-backed tokeniser package used with Transformers. It explains when to choose it over alternatives such as SentencePiece or tiktoken, how to load pretrained tokenisers, and how to train custom vocabularies with BPE, WordPiece or Unigram models. It also covers normalisers and pre-tokenisers, padding and truncation, batch encoding, and offset tracking that maps tokens back to the original text. Reference files go into algorithms, the processing pipeline, training and integration with Transformers and Datasets. It assumes the tokenizers, transformers and datasets Python packages.

できること

  • Train custom BPE, WordPiece or Unigram tokenisers
  • Configure padding, truncation and batch encoding
  • Track token offsets back to the source text