dev-python/tokenizers
Fast tokenizers for research and production
ChangeLog
commit a570f010425b445129d4cf383b866ca2c397cafe
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:40:56 2026 +0000
safetensors, tokenizers: DISTUTILS_EXT and test restriction, QA notices
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>
commit fa4a64056bd35730a4b99fa14496352d60cc206a
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:31:52 2026 +0000
tokenizers: Manifest for the rust.eclass ebuild
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>
commit 76b29c66517df5ee2e5253b764c6639b4674f4b7
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:31:16 2026 +0000
safetensors, tokenizers: rust.eclass, virtual/rust is gone
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>
commit 3eb8ce871773b8775136dccd740491218d17fc68
Author: gitea-actions <gitea-actions@noreply.ipnmod.org>
Date: Mon Sep 7 12:26:35 2026 +0000
Manifests for tokenizers-distfiles-v0.22.2
commit 8cf710e9a3a9baae8f8cbce6a374d1a37b7e0837
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:20:21 2026 +0000
dev-python/safetensors + tokenizers: Rust ML libs built from vendored crates
Shared gen-rust-crates.sh vendoring infrastructure; versions follow
the LocalAI release's transformers/diffusers pins.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:40:56 2026 +0000
safetensors, tokenizers: DISTUTILS_EXT and test restriction, QA notices
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>
commit fa4a64056bd35730a4b99fa14496352d60cc206a
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:31:52 2026 +0000
tokenizers: Manifest for the rust.eclass ebuild
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>
commit 76b29c66517df5ee2e5253b764c6639b4674f4b7
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:31:16 2026 +0000
safetensors, tokenizers: rust.eclass, virtual/rust is gone
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>
commit 3eb8ce871773b8775136dccd740491218d17fc68
Author: gitea-actions <gitea-actions@noreply.ipnmod.org>
Date: Mon Sep 7 12:26:35 2026 +0000
Manifests for tokenizers-distfiles-v0.22.2
commit 8cf710e9a3a9baae8f8cbce6a374d1a37b7e0837
Author: Plamen K. Kosseff <plamen@ipnmod.org>
Date: Mon Sep 7 12:20:21 2026 +0000
dev-python/safetensors + tokenizers: Rust ML libs built from vendored crates
Shared gen-rust-crates.sh vendoring infrastructure; versions follow
the LocalAI release's transformers/diffusers pins.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <plamen@ipnmod.org>


View
Download
Browse