This is a Java implementation of a GPT3/4 tokenizer, loosely ported from Tiktoken with the help of ChatGPT. ...that all 3.5-turbo models released after 0613 now have tokenization counts for messages ...
├── java │ ├── objects │ │ ├── ApiConfigure.java //tf-serving参数 │ │ ├── InputExample.java //input中间形式 ...