dev-python/tiktoken
Fast BPE tokeniser for use with OpenAI's models.
Screenshots
-
tiktoken-0.12.0~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View 
Download 
Browse License: MIT Apache-2.0 Unicode-3.0 Overlay: tatsh-overlay
ChangeLog
commit e73631b484b5f1264bcc96e99b408e9f37813a96
Author: Andrew Udvare <audvare@gmail.com>
Date: Sat Oct 18 22:16:10 2025 -0400
dev-python/tiktoken: update DESCRIPTION, LICENSE, SRC_URI
Signed-off-by: Andrew Udvare <audvare@gmail.com>
commit 090a84107e678af527a93fdc7b3e6401d0c6740a
Author: Andrew Udvare <audvare@gmail.com>
Date: Sat Oct 18 18:44:38 2025 -0400
dev-python/tiktoken: new package, add 0.8.0
Signed-off-by: Andrew Udvare <audvare@gmail.com>
Author: Andrew Udvare <audvare@gmail.com>
Date: Sat Oct 18 22:16:10 2025 -0400
dev-python/tiktoken: update DESCRIPTION, LICENSE, SRC_URI
Signed-off-by: Andrew Udvare <audvare@gmail.com>
commit 090a84107e678af527a93fdc7b3e6401d0c6740a
Author: Andrew Udvare <audvare@gmail.com>
Date: Sat Oct 18 18:44:38 2025 -0400
dev-python/tiktoken: new package, add 0.8.0
Signed-off-by: Andrew Udvare <audvare@gmail.com>