Key takeaways
- LLMs predict the next token sequentially; they don't search or reason like humans, and each API call is stateless unless you include prior context.
- Transformer architecture with multi-head attention lets models weight relevant context heavily; GPT-4o has 160 layers and 1.3 trillion parameters.
- Training frontier models costs $50M–$100M+; inference costs $0.03–$0.06 per million tokens for GPT-4o, making Brazilian deployment via cloud APIs practical for startups.
- LLMs hallucinate confidently and cannot access real-time information; they excel at pattern matching (summarization, code generation) but fail at novel reasoning.
A large language model is not magic — it’s a neural network trained to predict the next word in a sequence. When you ask ChatGPT a question, the model doesn’t search or reason in the way humans do. It processes your input as tokens (typically 4-character chunks), computes probabilities over the next token, samples one, and repeats. Each token generation takes milliseconds to seconds depending on model size and hardware. OpenAI’s GPT-4o supports context windows up to 128,000 tokens. Anthropic’s Claude now extends this to 200,000 tokens, meaning the model can consider that much history before generating the next word.
Tokens: How LLMs See Language
A token is not a word — it’s a subword unit. The phrase “machine learning” might tokenize as [“machine”, “learning”], or [“mach”, “ine”, ” learning”] depending on the model’s vocabulary. Crucially, an LLM with a 100,000-token vocabulary can only “see” 100,000 distinct units. This explains why LLMs misspell rare proper nouns or struggle with new terminology. Pricing structures also reflect tokenization: OpenAI charges per token consumed, not per query. As of August 2026, GPT-4o costs approximately $0.03 per million input tokens and $0.06 per million output tokens. A 1,000-word response at ~300 tokens per 200 words costs roughly $0.002 in output alone.
The Transformer Architecture and Attention
All modern LLMs use a transformer architecture introduced in 2017. The core mechanism is attention: for each token the model generates, it computes a relevance score against every previous token, then weights predictions based on those scores.
Multi-Head Attention
Modern models use parallel attention heads, each learning to track different relationships. One head might track pronoun references (“he” should attend to the most recent person), another tracks topic continuity, a third tracks syntax. A model like GPT-4o has 160 layers, each with multiple attention heads and feed-forward networks. GPT-4o’s parameter count is estimated at 1.3 trillion, though Anthropic hasn’t published Claude’s exact figure.
How Attention Works in Practice
When generating text after “The CEO of Apple, Tim Cook, announced…”, attention allows the model to weight “Apple” and “Tim Cook” more heavily than distant context, enabling coherent follow-ups. This weighting is learned during training — not hand-coded. Layer normalization and residual connections between blocks stabilize training across 160 layers.
Training: Scale and Computational Cost
LLMs are trained on massive corpora — billions to trillions of tokens from the internet, books, and code repositories. Training is self-supervised: the model predicts the next token, its error is backpropagated through all 160 layers, and billions of parameters are adjusted. Scaling laws show performance improves predictably with more data and compute — roughly following a power law. Doubling training compute yields linear improvements in loss until data limits are reached.
The Cost of Frontier Models
Training GPT-4o or Claude 3 Opus costs tens of millions to over $100 million in GPU/TPU compute alone. Nvidia H100 GPUs, essential for large-scale training, cost $15,000–$40,000 each and face severe supply constraints. Electricity and operational overhead compound costs. Anthropic uses Constitutional AI: training Claude to follow principles (a “constitution”), then fine-tuning via human feedback. This is expensive but aims for reliability.
Fine-Tuning on Custom Data
Domain-specific fine-tuning costs $1,000–$50,000 depending on model size and dataset. A bank fine-tuning on customer complaints, or a legal firm on case summaries, produces more accurate domain outputs than base models with examples.
Inference: Running the Model
Once trained, inference runs the model to generate output. Your tokenized query flows through 160 layers, producing a probability distribution over the vocabulary. The top token is sampled (or selected greedily), added to context, and the process repeats. Temperature controls randomness: high temperature (0.9) makes output creative but incoherent; low temperature (0.1) makes it deterministic. Latency is critical: Claude via the Anthropic API takes 2–5 seconds for 100 tokens on cloud infrastructure. This is acceptable for reporting and analysis but too slow for autocomplete or real-time chat on a browser without streaming.
In Brazil, most organizations access LLMs via cloud APIs because running 1.3-trillion-parameter models locally is impractical. Smaller models like Llama 2 7B can run on consumer hardware (a MacBook Pro) but generate 100 tokens in 30–60 seconds — an order of magnitude slower than cloud APIs.
Capabilities, Limitations, and Hallucination
LLMs are sophisticated autocomplete systems. They excel at pattern-matching tasks: summarization, translation, code generation within learned patterns, and creative writing. They’re weak at novel reasoning, multi-step math with very large numbers, and tasks requiring ground truth unavailable in training data.
A critical limitation: LLMs hallucinate confidently. They generate false information, especially about recent events, obscure facts, or numerical claims. A model trained through April 2024 cannot accurately answer “What happened in July 2024?” without external information. This is architectural, not a bug. Each API call is also stateless — the model won’t remember your name from Monday unless you include that context in Tuesday’s query.
Economics of Deployment
For a Brazilian SaaS startup: 10,000 monthly users × 5 queries each × 100 output tokens = 5 million output tokens monthly. At Claude Haiku’s $4 per million output tokens, that’s R$100 monthly in LLM costs (~R$20 USD equivalent). Passing this to users as a premium feature at R$50 monthly leaves margin for infrastructure. GPT-4o costs 10–15× more per token, making it economical only for high-value use cases.
The frontier is shifting toward multimodal models (text, image, audio, video), longer context windows, and reasoning-focused variants. OpenAI’s o1 spends more compute before answering, useful for complex math but slower and costlier. Inference optimization — quantization, distillation, speculative decoding — is reducing latency and cost by 2–10×, unlocking real-time applications.
Frequently Asked Questions
What exactly is a token, and why does it matter?
A token is a subword unit (roughly 4 characters) that the model processes. The phrase 'machine learning' might tokenize as two tokens or three depending on the vocabulary. Token count determines how much history the model can see (128,000 for GPT-4o, 200,000 for Claude), and pricing is per token, so understanding tokenization helps you predict API costs.
How does attention let LLMs understand context?
Attention computes relevance scores between the current token and all previous tokens, then weights predictions based on those scores. Multiple attention heads run in parallel, each learning different relationships (pronouns, topics, syntax). This allows the model to weight 'Apple' and 'Tim Cook' from earlier in a sentence more heavily when generating follow-up text.
What can I realistically use an LLM for in Brazil, and what are the costs?
LLMs excel at summarization, translation, Q&A over documents, and code generation. For 10,000 users with 5 queries monthly (100 tokens output each), Claude Haiku costs roughly R$100 monthly, making a premium feature at R$50 monthly viable. GPT-4o costs 10–15× more but suits high-value use cases where accuracy is critical.