Gentoo Portage Overlays - dev-python/tokenizers

dev-python/tokenizers

Fast, efficient, Rust-backed tokenizers for Python (HuggingFace)

Screenshots

https://github.com/huggingface/tokenizers
https://pypi.org/project/tokenizers/

tokenizers-0.22.1

~amd64 ~arm64 ~x86

onig hf-hub indicatif fancy-regex test debug python_targets_python3_12 python_targets_python3_13 python_targets_python3_14

View

Download

Browse License: Apache-2.0

Overlay: ha-bleending-edge

ChangeLog USE Flags Dependencies Reverse Deps Related Bugs

Runtime Dependencies

tokenizers-0.22.1

hf-hub?
      ( dev-python/huggingface_hub[python_targets_python3_12
            (-)
      ?
,python_targets_python3_13
            (-)
      ?
,python_targets_python3_14
            (-)
      ?
] )

python_targets_python3_12?
      ( dev-lang/python:3.12 )

python_targets_python3_13?
      ( dev-lang/python:3.13 )

python_targets_python3_14?
      ( dev-lang/python:3.14 )