dev-python/tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
Screenshots
-
tokenizers-0.22.2~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View 
Download 
Browse License: Apache-2.0 Apache-2.0 MIT Unicode-3.0 Overlay: bennypowers