dev-python/tokenizers
Fast, efficient, Rust-backed tokenizers for Python (HuggingFace)
Screenshots
-
tokenizers-0.22.1~amd64 ~arm64 ~x86onig hf-hub indicatif fancy-regex test debug python_targets_python3_12 python_targets_python3_13 python_targets_python3_14
View 
Download 
Browse License: Apache-2.0 Overlay: ha-bleending-edge
ChangeLog
commit 7f04638837c0c276880c662b064e5b9eee6aee6b
Author: Fernando Cunha <tabascoz@gmail.com>
Date: Fri Jan 30 19:08:41 2026 +0000
update manifests & repo info
Signed-off-by: Fernando Cunha <tabascoz@gmail.com>
commit a030c1261fb6cfd5fc8b897158264a820640bf6c
Author: Fernando Cunha <tabascoz@gmail.com>
Date: Thu Dec 18 14:34:51 2025 +0000
update manifests
Signed-off-by: Fernando Cunha <tabascoz@gmail.com>
commit 30d2a8ca73d478090eb4c81e9228357d65d83bdf
Author: Fernando Cunha <tabascoz@gmail.com>
Date: Thu Dec 18 11:45:36 2025 +0000
update to 2025.12.3 and switch to python3_14
Signed-off-by: Fernando Cunha <tabascoz@gmail.com>
Author: Fernando Cunha <tabascoz@gmail.com>
Date: Fri Jan 30 19:08:41 2026 +0000
update manifests & repo info
Signed-off-by: Fernando Cunha <tabascoz@gmail.com>
commit a030c1261fb6cfd5fc8b897158264a820640bf6c
Author: Fernando Cunha <tabascoz@gmail.com>
Date: Thu Dec 18 14:34:51 2025 +0000
update manifests
Signed-off-by: Fernando Cunha <tabascoz@gmail.com>
commit 30d2a8ca73d478090eb4c81e9228357d65d83bdf
Author: Fernando Cunha <tabascoz@gmail.com>
Date: Thu Dec 18 11:45:36 2025 +0000
update to 2025.12.3 and switch to python3_14
Signed-off-by: Fernando Cunha <tabascoz@gmail.com>