gpo.zugaina.org

Search Portage & Overlays:

dev-python/tokenizers

Fast, efficient, Rust-backed tokenizers for Python (HuggingFace)

Screenshots

  • tokenizers-0.22.1
    ~amd64 ~arm64 ~x86
    onig hf-hub indicatif fancy-regex test debug python_targets_python3_12 python_targets_python3_13 python_targets_python3_14

    View      Download      Browse     License: Apache-2.0   

USE Flags

onig
* This flag is undocumented *
hf-hub
* This flag is undocumented *
indicatif
* This flag is undocumented *
fancy-regex
* This flag is undocumented *
test
Global: Workaround to pull in packages needed to run with FEATURES=test. Portage-2.1.2 handles this internally, so don't set it in make.conf/package.use anymore
debug
Global: Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see http://www.gentoo.org/proj/en/qa/backtraces.xml
python_targets_python3_12
* This flag is undocumented *
python_targets_python3_13
* This flag is undocumented *
python_targets_python3_14
* This flag is undocumented *