Gentoo Portage Overlays - dev-python/tokenizers

dev-python/tokenizers

Fast tokenizers for research and production

Screenshots

https://github.com/huggingface/tokenizers
https://pypi.org/project/tokenizers/

tokenizers-0.22.2

~amd64

python_targets_python3_12 python_targets_python3_13 python_targets_python3_14 debug

View

Download

Browse License: Apache-2.0 MIT Unicode-DFS-2016

Overlay: local-ai

ChangeLog USE Flags Dependencies Reverse Deps Related Bugs

Runtime Dependencies

tokenizers-0.22.2

python_targets_python3_12?
      ( dev-lang/python:3.12 )

python_targets_python3_13?
      ( dev-lang/python:3.13 )

python_targets_python3_14?
      ( dev-lang/python:3.14 )