The word “strawberry” contains three "r"s. This simple question has highlighted a limitation in large language models (LLMs), such as GPT-4 and Claude, which often incorrectly count the number of "r"s as two. The error stems from the way these models process text through a process called tokenization, where text is broken down into smaller units called tokens. These tokens do not always correspond directly to individual letters, leading to errors in counting specific letters within words.
I asked mistral/brave AI and got this response:
Yes, at some point the meme becomes the training data and the LLM doesn’t need to answer because it sees the answer all over the damn place.