Class TokenizerGpt3

Namespace

OpenAI.Tokenizer.GPT3

Assembly

AntRunnerLib.dll

GPT3 Tokenizer.

public static class TokenizerGpt3

Inheritance

System.Object

TokenizerGpt3

Inherited Members

System.Object.Equals(System.Object)

System.Object.Equals(System.Object, System.Object)

System.Object.GetHashCode()

System.Object.GetType()

System.Object.MemberwiseClone()

System.Object.ReferenceEquals(System.Object, System.Object)

System.Object.ToString()

Methods

Encode(String, Boolean)

Encode This method use LF style EOL, if you use CR LF style EOL you need to set cleanUpWindowsEOL to true

public static IEnumerable<int> Encode(string text, bool cleanUpCREOL = false)

Parameters

text System.String

cleanUpCREOL System.Boolean

set it true to get rid of CR

Returns

IEnumerable<System.Int32>

TokenCount(String, Boolean)

Get token count. This method use LF style EOL, if you use CR LF style EOL you need to set cleanUpWindowsEOL to true

public static int TokenCount(string text, bool cleanUpCREOL = false)

Parameters

text System.String

cleanUpCREOL System.Boolean

set it true to get rid of CR

Returns

System.Int32