Foundations
How does an AI chatbot actually work?
AI chatbots feel like they understand you. Under the hood, something simpler and stranger is happening, they predict text one small piece at a time. Here is how that turns into a helpful assistant, and where it goes wrong.
When you chat with an AI like ChatGPT, it feels like talking to something that understands you. What is really happening is a bit simpler, and honestly a bit stranger.
The AI is a giant text-prediction machine. It has read an enormous amount of writing, so much that it has learned the patterns of how language fits together. When you type a message, it answers by guessing what word should come next, then the next, then the next, until it has built a full reply. That is it. There is no little person inside, and it is not looking things up in a database like a search engine. It is writing what most likely follows.
The reason it is so good is the sheer scale of what it has read, and a second training step where humans showed it examples of helpful, polite answers so it learned to be a useful assistant rather than just an autocomplete.
This also explains its biggest weakness. Because it writes what sounds right rather than what it has checked to be true, it can state a wrong fact with total confidence. So the golden rule is simple: an AI chatbot is a brilliant helper, but you are still the one who checks anything that matters.
An AI chatbot is powered by a large language model, a type of AI trained to do one core thing extremely well: predict the next piece of text.
Step 1: your words become tokens
The model does not read whole words the way we do. Your message is first chopped into tokens, small chunks that are often a word or part of a word. Everything from here on is the model working with tokens.
Step 2: prediction, one token at a time
Given the tokens so far, the model calculates a probability for every possible next token, and picks one. Then it adds that token to the text and repeats, over and over. String enough of these predictions together and you get fluent sentences and full answers. This is why a chatbot writes its reply gradually, left to right.
Step 3: why it seems to understand context
The engine that makes this work is the transformer, an AI design from 2017. Its key trick, called attention, lets the model weigh how every token relates to every other token, so it captures context and meaning rather than reading in a rigid line. Everything in the current conversation has to fit inside its context window, its working memory for that chat.
Step 4: from raw predictor to helpful assistant
A freshly trained model is just a powerful text-predictor. To become a polite, instruction-following assistant, it goes through extra training where humans provide examples of good answers and rank responses. This alignment step is what turns raw capability into something that actually helps.
Why it makes mistakes
Because the whole system optimises for plausible text, not true text, it can produce a hallucination, a confident, well-written answer that is simply wrong. The better you frame your prompt, and the more you verify important facts, the more reliably you get good results.
An AI chatbot is a deployed large language model: an autoregressive, transformer-based neural network that models a probability distribution over a token vocabulary and generates text by repeatedly sampling the next token conditioned on the preceding sequence.
The generation loop
Input text is tokenised (typically byte-pair encoding), each token mapped to a learned embedding, and the sequence processed by stacked self-attention and feed-forward layers. The final layer produces a distribution over the vocabulary for the next position; a decoding strategy (greedy, or sampling controlled by temperature and top-p) selects a token, which is appended and fed back in. Self-attention gives constant path length between any two positions at O(n²) cost in sequence length, the practical ceiling on the context window (Vaswani et al. 2017).
Two training phases
First, pretraining: self-supervised next-token prediction over a very large corpus, where linguistic competence and world knowledge are acquired; capability scales predictably with parameters, data and compute. In-context learning, the ability to adapt to a task from the prompt alone, emerges here (Brown et al. 2020). Second, alignment: supervised fine-tuning on demonstrations followed by preference optimisation (RLHF or successors such as DPO), which produces instruction-following behaviour and shapes tone and refusals (Ouyang et al. 2022).
Consequences and limits
Three properties follow directly from this design. The objective is likelihood, not truth, so hallucination is intrinsic and mitigated, not solved, by grounding (retrieval, tools) and verification. The model is stateless across calls: any "memory" is the application re-supplying context. And knowledge is frozen at training time, which is why retrieval-augmented and agentic systems are layered on top to inject current, private or tool-derived information. Understanding the chatbot as a conditional next-token sampler, rather than a knowledge base or a reasoning oracle, is the mental model that predicts both its strengths and its failure modes.