Gentoo Portage Overlays - dev-python/mosestokenizer

dev-python/mosestokenizer

Wrappers for the Moses tokenizer/detokenizer/punctuation-normaliser

Screenshots

https://github.com/luismsgomes/mosestokenizer
https://pypi.org/project/mosestokenizer/

mosestokenizer-1.2.1

~amd64 ~arm64 ~x86

python_targets_python3_12 python_targets_python3_13 python_targets_python3_14

View

Download

Browse License: LGPL-2+

Overlay: stuff

ChangeLog USE Flags Dependencies Reverse Deps Related Bugs

ChangeLog

commit 3d96cb05fb7dbb919615f7d6ad631b0cf0d71bfa
Author: Ivan S. Titov <iohann.s.titov@gmail.com>
Date: Sat May 9 21:54:08 2026 +0200

dev-python/mosestokenizer: new package, bigcode-eval dep