Gentoo Portage Overlays - dev-python/mosestokenizer

dev-python/mosestokenizer

Wrappers for the Moses tokenizer/detokenizer/punctuation-normaliser

Screenshots

https://github.com/luismsgomes/mosestokenizer
https://pypi.org/project/mosestokenizer/

mosestokenizer-1.2.1

~amd64 ~arm ~arm64 ~x86

python_targets_python3_12 python_targets_python3_13 python_targets_python3_14

View

Download

Browse License: LGPL-2+

Overlay: stuff

ChangeLog USE Flags Dependencies Reverse Deps Related Bugs

ChangeLog

commit 38da2e9b97e80aaca7c698fff72e0b55a227748c
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Fri Sep 25 09:25:39 2026 +0200

dev-python/mosestokenizer: Keyword 1.2.1 for ~arm

Pure Python, with no compiled code or architecture-specific paths.

commit 3d96cb05fb7dbb919615f7d6ad631b0cf0d71bfa
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sat May 9 21:54:08 2026 +0200

dev-python/mosestokenizer: new package, bigcode-eval dep