Tokenization

The infographic shows the right big shape: an LLM receives text, turns it into tokens, maps those tokens into vectors, runs transformer blocks, projects the fin…

A model does not see your prompt as words or characters. It sees token ids produced by a tokenizer. That is why a short-looking string can be expensive, a long …