dev-python/tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
Screenshots
-
tokenizers-0.22.2~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View 
Download 
Browse License: Apache-2.0 Apache-2.0 MIT Unicode-3.0 Overlay: bennypowers
ChangeLog
commit 6724062f578a3ed7e2c42eb9f19e1a7dc73cd2b0
Author: Benny Powers <web@bennypowers.com>
Date: Sun Mar 8 16:42:06 2026 +0200
new: tokenizers 0.22.2
Rust/maturin Python bindings for HuggingFace tokenizers.
Dependency of comfyui and transformers.
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Author: Benny Powers <web@bennypowers.com>
Date: Sun Mar 8 16:42:06 2026 +0200
new: tokenizers 0.22.2
Rust/maturin Python bindings for HuggingFace tokenizers.
Dependency of comfyui and transformers.
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>