dev-python/tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
Screenshots
-
tokenizers-0.22.2~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View 
Download 
Browse License: Apache-2.0 Apache-2.0 MIT Unicode-3.0 Overlay: bennypowers
Runtime Dependencies
tokenizers-0.22.2
>=dev-python/huggingface-hub-0.16.4[python_targets_python3_11
(-)
?
,python_targets_python3_12
(-)
?
,python_targets_python3_13
(-)
?
]
python_targets_python3_11?
( dev-lang/python:3.11 )
python_targets_python3_12?
( dev-lang/python:3.12 )
python_targets_python3_13?
( dev-lang/python:3.13 )