Ability to count tokens before sending

[AndrewMohawk]

Reading the Open AI docs on Chat completion they specifically call out counting tokens on input (as you are billed on these as well as the amount of tokens on output). They use their TikToken python library, it would be awesome if there was something in this lib to do the same

[sashabaranov]

Related: #62

[sashabaranov]

I think this library should solve this, right? https://github.com/pkoukk/tiktoken-go

[allesan]

Can you give example how one would use this library that you have linked?
Because ChatCompletionMessage.Content accepts string, not []uint

[sashabaranov]

@allesan tiktoken-go library might be used to count tokens, but not to pass them to ChatCompletionMessage

[allesan]

Is there some library or a way to send tokens with go-openai?

[sashabaranov]

@allesan what do you mean? You are sending tokens, just packed in the string representation (i.e. just a prompt string).

[allesan]

But if I understand correctly, with tiktoken lib you are sending tokens that are shortend and optimized for openai models?

[sashabaranov]

@allesan OpenAI API does not allow you to send tokenized inputs to it, see the API documentation: https://platform.openai.com/docs/api-reference/introduction

tiktoken lets you tokenize your inputs to count the number of tokens or use these tokens to train your own models.

[qhenkart]

@allesan tiktoken-go does resolve this problem for me. Now I can effectively count the tokens in the request by setting. Now you can properly adjust the MaxToken parameter

where count tokens is just a copy and paste of the example in tiktoken

func countTokens(prompt string) int {
	encoding := "cl100k_base"

	tke, err := tiktoken.GetEncoding(encoding)
	if err != nil {
		panic(err)
	}

	return len(tke.Encode(prompt, nil, nil))
}

[vvatanabe]

If there's more to discuss, please post it in issue #62.