Token

The definition of a token, simply put, is the "smallest unit" that large models understand and process language. Unlike humans, who can read based on "characters" or "words," computers fundamentally can only recognize numbers. To enable machines to process text, the text must first be segmented and converted into a string of numbers.