dev-python/tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
Screenshots
-
tokenizers-0.23.1~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View 
Download 
Browse License: Apache-2.0 Apache-2.0 MIT Unicode-3.0 Overlay: bennypowers -
tokenizers-0.22.2~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View Download Browse License: Apache-2.0 Apache-2.0 MIT Unicode-3.0 Overlay: bennypowers
ChangeLog
commit 5333d0d2f1227cc05afafe7ded5085ea0e00326f
Author: Benny Powers <web@bennypowers.com>
Date: Thu Apr 30 12:51:20 2026 +0300
bump(various): version updates 2026-04-30
comfy-aimdo 0.3.0, comfyui 0.20.1, comfyui-workflow-templates 0.9.63,
comfyui-workflow-templates-core 0.3.216,
comfyui-workflow-templates-media-other 0.3.182,
comfyui-workflow-templates-media-video 0.3.81,
huggingface-hub 1.12.2, tokenizers 0.23.1, transformers 5.7.0,
mappa 0.0.4
Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
commit 6724062f578a3ed7e2c42eb9f19e1a7dc73cd2b0
Author: Benny Powers <web@bennypowers.com>
Date: Sun Mar 8 16:42:06 2026 +0200
new: tokenizers 0.22.2
Rust/maturin Python bindings for HuggingFace tokenizers.
Dependency of comfyui and transformers.
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
Author: Benny Powers <web@bennypowers.com>
Date: Thu Apr 30 12:51:20 2026 +0300
bump(various): version updates 2026-04-30
comfy-aimdo 0.3.0, comfyui 0.20.1, comfyui-workflow-templates 0.9.63,
comfyui-workflow-templates-core 0.3.216,
comfyui-workflow-templates-media-other 0.3.182,
comfyui-workflow-templates-media-video 0.3.81,
huggingface-hub 1.12.2, tokenizers 0.23.1, transformers 5.7.0,
mappa 0.0.4
Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
commit 6724062f578a3ed7e2c42eb9f19e1a7dc73cd2b0
Author: Benny Powers <web@bennypowers.com>
Date: Sun Mar 8 16:42:06 2026 +0200
new: tokenizers 0.22.2
Rust/maturin Python bindings for HuggingFace tokenizers.
Dependency of comfyui and transformers.
Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>