gpo.zugaina.org

Search Portage & Overlays:

sci-ml/tokenizers

Implementation of today's most used tokenizers

Screenshots

  • tokenizers-0.23.2
    ~amd64 ~arm64
    test debug python_single_target_python3_12 python_single_target_python3_13 python_single_target_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD BSD-2 CDLA-Permissive-2.0 ISC MIT MPL-2.0 Unicode-3.0 ZLIB   
    Overlay: stuff
  • tokenizers-0.23.2
    ~amd64
    test debug python_single_target_python3_12 python_single_target_python3_13 python_single_target_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: gentoo
  • tokenizers-0.23.2
    ~amd64
    test debug python_targets_python3_12 python_targets_python3_13 python_targets_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: pentoo
  • tokenizers-0.23.1
    ~amd64 ~arm64
    test debug python_single_target_python3_12 python_single_target_python3_13 python_single_target_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: stuff
  • tokenizers-0.23.1
    ~amd64
    test debug python_single_target_python3_12 python_single_target_python3_13 python_single_target_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: gentoo
  • tokenizers-0.22.2
    ~amd64
    test debug python_single_target_python3_12 python_single_target_python3_13 python_single_target_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: gentoo
  • tokenizers-0.22.2
    ~amd64 ~arm64
    test debug python_single_target_python3_12 python_single_target_python3_13 python_single_target_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: stuff
  • tokenizers-0.22.1
    ~amd64
    test debug python_single_target_python3_12 python_single_target_python3_13 python_single_target_python3_14 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: gentoo
  • tokenizers-0.21.1-r88881
    ~amd64
    test debug python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: anonch
  • tokenizers-0.21.1-r1
    ~amd64
    test debug python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: anonch
  • tokenizers-0.21.0
    ~amd64
    test debug python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug

    View      Download      Browse     License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016   
    Overlay: anonch

ChangeLog

commit 62bb7dfd56cccffd0955e349bfc3e4346dfb4d62
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Oct 4 20:44:54 2026 +0200

sci-ml/tokenizers: raise RUST_MIN_VER to what the crates declare

The cargo eclass QA notice flags both: 0.22.2's crate tarball holds
crates declaring rust-version 1.87.0 against RUST_MIN_VER 1.82.0, and
0.23.2 vendors icu_properties 2.3.0 and its siblings, which declare
1.88, against 1.87.0. 0.23.1 already matches its crates.

commit f6050ddb3e140c800f3c642c455fedafe70388dc
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Oct 4 13:12:32 2026 +0200

sci-ml/tokenizers: add missing || die

The devmanual asks for || die after every command that can fail; these
cd, mv, cp, mkdir and rm calls could fail silently. No installed files
change, so no revision bump.

commit 94c9b08eb48d812d4dc7b320d3f58b1cca170037
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sat Oct 3 23:19:32 2026 +0200

*/*: indent the remaining metadata.xml files with spaces

Twenty-six files imported from ::gentoo in August and September kept
its tab indentation, while the other 705 use two spaces, as Gentoo's
skel.metadata.xml does. Only leading whitespace changes; each file
parses to the same content.

commit b6928c829de033b43e42ebefd4115a804e769cc9
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sat Oct 3 21:27:41 2026 +0200

sci-ml/tokenizers: drop 0.22.1

Retention keeps 0.22.2, 0.23.1 and 0.23.2; nothing pins 0.22.1.

commit 2e4e67c4553c178d45d5f4bb8dd040880fe38ff7
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sat Oct 3 18:05:26 2026 +0200

sci-ml/tokenizers: restore dependent crate licenses marker

pycargoebuild -i needs the '# Dependent crate licenses' line, which the
0.22.1, 0.22.2 and 0.23.1 test-setup cleanups removed. ::gentoo's copies
keep it. Comment only.

commit 101cf7b50e8c442aa8424f203ab04c788453b13e
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 07:01:30 2026 +0200

sci-ml/tokenizers: remove dead 0.23.1 test setup

commit 9e85011c1143184ca4489acc91ce57e073cd3be7
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 07:01:30 2026 +0200

sci-ml/tokenizers: remove dead 0.22.2 test setup

commit ef598f330bdba1cd569d5f2e1a62c13b11d9bdb3
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 07:01:30 2026 +0200

sci-ml/tokenizers: remove dead 0.22.1 test setup

commit 1a012d9c7e29bf7176d830e1f85d562d93049b05
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Thu Sep 10 05:01:29 2026 +0200

sci-ml/tokenizers: trim 0.23.2 comment prose

commit 04194ec48ca1bbc4605bb415c18e4ec2ba40074c
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Wed Sep 9 13:24:59 2026 +0200

sci-ml/tokenizers: add 0.23.2

Unblocks a bump that had been waiting on a third party. The ::gentoo
ebuild this was forked from carries an empty CRATES and pulls two
pre-bundled crate archives from dev.gentoo.org/~tupone; neither was ever
published for 0.23.2, so both 404 and the version was unbuildable.

CRATES is regenerated instead, so the crates come from crates.io like every
other Rust package here. The ebuild compiles two graphs -- the tokenizers/
core and bindings/python/ -- and only the latter ships a lock, so the core
one is generated first and the two sets unioned; resolving from the python
lock alone leaves out fancy-regex and cargo fails offline. The dependent
crate license list is regenerated from the same union.

commit 6852f6daacd0d162bf6f16b2f0cc3e1521d7ce72
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 01:54:34 2026 +0200

sci-ml/tokenizers: Keyword 0.23.1 for ~arm64

The Rust library and Python bindings support aarch64.

commit d655646b8c22148f6957c69a19329e1988c7ded4
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 01:54:34 2026 +0200

sci-ml/tokenizers: Keyword 0.22.2 for ~arm64

The Rust library and Python bindings support aarch64.

commit a8c0f2589e4888fb9bd450a65e9591b1682dbc1d
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 01:54:34 2026 +0200

sci-ml/tokenizers: Keyword 0.22.1 for ~arm64

The Rust library and Python bindings support aarch64.

commit dfd04b579e0634ddcec14068517bf535ac9a6b91
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sun Sep 6 01:54:09 2026 +0200

sci-ml/tokenizers: add 0.22.1, 0.22.2, 0.23.1

Add the current Gentoo package to provide the tokenizer bindings needed by overlay consumers.