dev-python/tokenizers
Fast, efficient, Rust-backed tokenizers for Python (HuggingFace)
-
tokenizers-0.22.1~amd64 ~arm64 ~x86onig hf-hub indicatif fancy-regex test debug python_targets_python3_12 python_targets_python3_13 python_targets_python3_14
View
Download
Browse License: Apache-2.0 Overlay: ha-bleending-edge
Runtime Dependencies
tokenizers-0.22.1
hf-hub?
( dev-python/huggingface_hub[python_targets_python3_12
(-)
?
,python_targets_python3_13
(-)
?
,python_targets_python3_14
(-)
?
] )
python_targets_python3_12?
( dev-lang/python:3.12 )
python_targets_python3_13?
( dev-lang/python:3.13 )
python_targets_python3_14?
( dev-lang/python:3.14 )

