huggingface-tokenizers
Orchestra Research's skill for Hugging Face's tokenizers library, the Rust-backed tokeniser package used with Transformers.
What the skill teaches
Orchestra Research's skill for Hugging Face's tokenizers library, the Rust-backed tokeniser package used with Transformers. It explains when to choose it over alternatives such as SentencePiece or tiktoken, how to load pretrained tokenisers, and how to train custom vocabularies with BPE, WordPiece or Unigram models. It also covers normalisers and pre-tokenisers, padding and truncation, batch encoding, and offset tracking that maps tokens back to the original text. Reference files go into algorithms, the processing pipeline, training and integration with Transformers and Datasets. It assumes the tokenizers, transformers and datasets Python packages.
Capabilities
- Train custom BPE, WordPiece or Unigram tokenisers
- Configure padding, truncation and batch encoding
- Track token offsets back to the source text
About
- Publisher
- Orchestra Research
- Tool
- Hugging Face
Install
The project doesn't document install steps that we could confirm. See the project's own instructions
What's inside
- references/algorithms.md
- references/integration.md
- references/pipeline.md
- references/training.md
MCP servers and skills can run code on your computer and act on your data. Review the source before installing.
Details
| Fact | Value | Source |
|---|---|---|
| Type | Skill | |
| Tool | Hugging Face | |
| Publisher | Orchestra Research | source |
| Official status | Community (not published by the tool's vendor) | source |
| Needs an API key or sign-in | Not stated in its documentation | |
| Includes scripts | No | source |
| Platforms | Not stated in its documentation | |
| AI clients | Not stated in its documentation | |
| Licence | MIT | source |
| Pricing | Not stated in its documentation | |
| Activity | Last commit Oct 11, 202614,431 GitHub starsDownload data not available (not on npm or PyPI) | source |
| Links | Repository Website |
More from Orchestra Research
4 skills
All 4 from Orchestra Research →Sources
- Skill file (SKILL.md)github.com/orchestra-research/ai-research-skillsNov 12, 2026
- Licence file (LICENSE)github.com/orchestra-research/ai-research-skills · Backs: LicenceNov 12, 2026
- GitHub repositorygithub.com/orchestra-research/ai-research-skills · Backs: Official status, Publisher, Activity, LicenceNov 11, 2026
- Folder in the repository (huggingface-tokenizers)github.com/orchestra-research/ai-research-skills · Backs: Includes scriptsNov 12, 2026
- Skill file (SKILL.md)github.com/Orchestra-Research/AI-Research-SKILLsNov 10, 2026
- Folder in the repository (huggingface-tokenizers)github.com/Orchestra-Research/AI-Research-SKILLs · Backs: Includes scriptsNov 10, 2026