What a Large Language Model Actually Is
A large language model is a neural network trained on vast amounts of text to predict and generate human language. The 'large' refers to the number of parameters — the numerical weights that encode learned patterns. Modern LLMs have hundreds of billions of parameters. The 'language model' part means the core task it was trained on: predicting the next word (or token) in a sequence, given all the words before it. Train this prediction task on enough diverse text — articles, books, code, conversations — and what emerges is a model that can write, reason, translate, code, and answer questions across almost any domain.
How LLMs Are Trained: The Two Phases
LLM training happens in two main phases. Pre-training: the model is exposed to enormous text datasets (often hundreds of billions of words) and learns to predict the next token. This is computationally intensive — it costs millions of dollars and takes months on thousands of specialized chips. The result is a model with broad knowledge and language capabilities, but that isn't yet aligned to be helpful, safe, or specific. Fine-tuning with RLHF (Reinforcement Learning from Human Feedback): human raters score model responses, and the model is trained to produce responses that humans rate highly. This phase transforms the raw pre-trained model into a helpful, conversational assistant aligned to human preferences.
The Transformer Architecture: Why It Works
The underlying architecture that made modern LLMs possible is the Transformer, introduced in the 2017 paper 'Attention Is All You Need.' The key innovation is the attention mechanism, which allows the model to weigh the relevance of every other token in the sequence when processing any given token — rather than reading text linearly. This means a word at the beginning of a paragraph can directly influence how a word at the end is processed, which enables the model to capture long-range dependencies in language. Virtually all modern LLMs — GPT, Claude, Gemini, LLaMA — are transformer-based architectures scaled up with more data and more parameters.
What LLMs Can and Cannot Do
LLMs excel at language tasks: writing, editing, summarizing, translating, explaining, coding, classifying, and answering questions across a huge range of domains. They can engage in multi-turn conversations, follow complex instructions, and generate structured outputs. What they fundamentally cannot do: access real-time information (without retrieval tools), maintain persistent memory across conversations, perform precise mathematical calculations reliably (though they can approximate), reason about the physical world, or take actions in the real world on their own. The most important misunderstanding about LLMs is that they 'know' things the way humans do — they have statistical patterns, not verified knowledge.
Open Source vs. Closed LLMs
LLMs come in two categories: closed (proprietary) models like GPT, Claude, and Gemini, accessible only via API or product; and open-source models like Meta's LLaMA, Mistral, and Falcon, whose weights are publicly available for download and local deployment. Closed models are generally more capable at the frontier and receive continuous improvements, but you pay per use and have no control over the underlying model. Open source models can be run locally (private, free after hardware cost) and can be fine-tuned on your own data, but require technical setup and generally lag behind closed models on the hardest tasks. The gap between open and closed models has been narrowing significantly.
Emergent Capabilities: What Makes LLMs Surprising
One of the most remarkable aspects of LLMs is emergent capabilities — abilities that appear suddenly as models are scaled up, without being explicitly trained for. Larger models can follow complex multi-step reasoning chains, engage in analogical reasoning, write code in languages barely represented in training data, and solve novel problems. These capabilities weren't engineered in; they emerged from scale. This makes LLM development partially unpredictable: researchers regularly discover new capabilities in large models that weren't anticipated. It's also why 'just make the model bigger' was such a productive research direction for so long — and why it's now running into fundamental limits that researchers are actively working around.