sci-ml/tokenizers
Implementation of today's most used tokenizers
Screenshots
-
tokenizers-0.21.1-r88881~amd64test debug python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug
View 
Download 
Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016 Overlay: anonch -
tokenizers-0.21.1-r1~amd64test debug python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug
View Download Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016 Overlay: anonch -
tokenizers-0.21.1-r1~amd64test debug python_single_target_python3_10 python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug
View Download Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016 Overlay: gentoo -
tokenizers-0.21.0~amd64test debug python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug
View Download Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016 Overlay: anonch -
tokenizers-0.21.0~amd64test debug python_single_target_python3_10 python_single_target_python3_11 python_single_target_python3_12 python_single_target_python3_13 debug
View Download Browse License: Apache-2.0 Apache-2.0 Apache-2.0-with-LLVM-exceptions BSD-2 BSD ISC MIT MPL-2.0 Unicode-DFS-2016 Overlay: gentoo
ChangeLog
commit a95a5e4aeeffd681131126b48a12dd87c270fd35
Author: (none) <>
Date: Sat May 24 10:32:33 2025 +0000
comments cleaning
commit 62d94768c35c1ccd9c7ee54a6aa25cd281ed3c68
Author: (none) <>
Date: Sat May 24 09:25:50 2025 +0000
add sci-ml/tokenizers
Author: (none) <>
Date: Sat May 24 10:32:33 2025 +0000
comments cleaning
commit 62d94768c35c1ccd9c7ee54a6aa25cd281ed3c68
Author: (none) <>
Date: Sat May 24 09:25:50 2025 +0000
add sci-ml/tokenizers