dev-python/tiktoken
Fast BPE tokeniser for use with OpenAI's models
Screenshots
-
tiktoken-0.8.0~amd64python_targets_python3_11 python_targets_python3_12 python_targets_python3_13
View 
Download 
Browse License: MIT Overlay: tatsh-overlay
ChangeLog
commit 090a84107e678af527a93fdc7b3e6401d0c6740a
Author: Andrew Udvare <audvare@gmail.com>
Date: Sat Oct 18 18:44:38 2025 -0400
dev-python/tiktoken: new package, add 0.8.0
Signed-off-by: Andrew Udvare <audvare@gmail.com>
Author: Andrew Udvare <audvare@gmail.com>
Date: Sat Oct 18 18:44:38 2025 -0400
dev-python/tiktoken: new package, add 0.8.0
Signed-off-by: Andrew Udvare <audvare@gmail.com>