dev-python/tiktoken
Fast BPE tokeniser for use with OpenAI's models.
Screenshots
-
tiktoken-0.12.0~amd64debug python_targets_python3_11 python_targets_python3_12 python_targets_python3_13 debug
View 
Download 
Browse License: MIT Apache-2.0 Unicode-3.0 Overlay: tatsh-overlay