gpo.zugaina.org

Search Portage & Overlays:

dev-python/tokenizers

Fast tokenizers for research and production

Screenshots

  • tokenizers-0.22.2
    ~amd64
    python_targets_python3_12 python_targets_python3_13 python_targets_python3_14 debug

    View      Download      Browse     License: Apache-2.0 MIT Unicode-DFS-2016   
    Overlay: local-ai

ChangeLog

commit a570f010425b445129d4cf383b866ca2c397cafe
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:40:56 2026 +0000

safetensors, tokenizers: DISTUTILS_EXT and test restriction, QA notices

Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>

commit fa4a64056bd35730a4b99fa14496352d60cc206a
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:31:52 2026 +0000

tokenizers: Manifest for the rust.eclass ebuild

Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>

commit 76b29c66517df5ee2e5253b764c6639b4674f4b7
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:31:16 2026 +0000

safetensors, tokenizers: rust.eclass, virtual/rust is gone

Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>

commit 3eb8ce871773b8775136dccd740491218d17fc68
Author: gitea-actions <gitea-actions@noreply.ipnmod.org>
Date: Mon Sep 7 12:26:35 2026 +0000

Manifests for tokenizers-distfiles-v0.22.2

commit 8cf710e9a3a9baae8f8cbce6a374d1a37b7e0837
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:20:21 2026 +0000

dev-python/safetensors + tokenizers: Rust ML libs built from vendored crates

Shared gen-rust-crates.sh vendoring infrastructure; versions follow
the LocalAI release's transformers/diffusers pins.

Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>