dev-python/tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
Screenshots
-
tokenizers-0.22.2~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View 
Download 
Browse License: Apache-2.0 Apache-2.0 MIT Unicode-3.0 Overlay: bennypowers
USE Flags
debug
Global: Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml
python_targets_python3_11
* This flag is undocumented *
python_targets_python3_12
* This flag is undocumented *
python_targets_python3_13
* This flag is undocumented *