dev-python/tokenizers
Fast, efficient, Rust-backed tokenizers for Python (HuggingFace)
Screenshots
-
tokenizers-0.22.1~amd64 ~arm64 ~x86onig hf-hub indicatif fancy-regex test debug python_targets_python3_12 python_targets_python3_13 python_targets_python3_14
View 
Download 
Browse License: Apache-2.0 Overlay: ha-bleending-edge