What Leaders Should Know About How LLMs Work
You do not need the mathematics. You do need five properties, because each one explains a failure you will otherwise be surprised by.
Erin Moore
Fractional Chief AI Officer
Five properties explain almost everything a large language model will do to your business: it predicts plausible text rather than retrieving facts, it has no memory between conversations unless one is built, its knowledge has a cut-off, it is non-deterministic, and it is far more sensitive to how you frame a request than most people expect. You do not need the mathematics. You do need these, because each one predicts a specific failure.
Specific model capabilities and pricing change quickly; the properties below are structural and durable. Check current specifics against vendor documentation when you evaluate.
One: it predicts plausible text, not true text
The system generates what is likely to follow, given everything it has seen. Truth is not a separate check the model performs — accuracy is a by-product of plausible text usually being accurate.
This is why a model invents a citation with the correct format and a fabricated title: a plausible-looking reference is the right shape of answer. It is also why confident wrongness is the characteristic failure. Errors do not arrive flagged as uncertain; they arrive indistinguishable from correct answers.
What this changes: wherever being wrong is expensive, you need verification that does not rely on the model's tone. And for anything involving your own facts, you want retrieval with citations, per RAG explained for executives.
Two: it has no memory
Each conversation starts empty. Anything the system appears to remember was re-supplied behind the scenes. Nothing said in one session carries into another unless a system was built to carry it.
What this changes: "it will learn our business over time" is false unless someone is engineering that, and it is a real project when they are. It is also mildly reassuring on confidentiality — your input is not absorbed into the model — though it says nothing about logging and retention, which are separate and contractual.
Three: knowledge has a cut-off
Training data ends at a date. Ask about anything after it and the model will either decline or produce something plausible and wrong, and which of those you get is not reliably predictable.
What this changes: anything time-sensitive — pricing, regulation, current events, your own recent decisions — must be supplied at question time rather than assumed. This is the failure mode behind most embarrassing internal AI answers.
Four: it is non-deterministic
The same question can produce different answers. Not wildly different, usually, but different enough to matter for audit, testing and reproducibility.
What this changes: you cannot validate by checking one output. You need an evaluation set run repeatedly, with tolerances — the discipline in the AI vendor evaluation framework, and the specific gap that makes AI governance in regulated settings harder than traditional model validation, per AI governance for regulated industries.
Five: framing does most of the work
The difference between a useless output and a good one is usually the request, not the model. Context, constraints, examples and an explicit statement of what the output is for move quality more than switching vendors does.
What this changes: before concluding a tool is inadequate, check whether anyone framed the task properly. A surprising share of "the AI isn't good enough" conclusions are untested requests — which is why training on real tasks beats buying a better licence.
What this means for how you buy
You are buying a capable, confident, forgetful, slightly unpredictable generalist that is excellent at language and has no independent access to truth. Structure the work accordingly: give it the facts, check what matters, and keep a person on anything irreversible.
Capability at the frontier moves quickly, and Stanford HAI's AI Index is the most disciplined public record of how. None of these five properties has been engineered away, and your operating decisions should assume they hold.
Frequently asked questions
Do executives need to understand how LLMs work technically? No mathematics required. Five structural properties are enough: plausible-not-true generation, no memory between sessions, a knowledge cut-off, non-determinism, and high sensitivity to how a request is framed.
Why do LLMs make things up? Because they generate text that is likely to follow, not text verified as true. A fabricated citation has the right shape for the answer, which is exactly why it gets produced — and why errors arrive sounding as confident as correct answers.
Will the model learn our business over time? Not by itself. There is no memory between conversations. Anything that appears remembered is being re-supplied by a system someone built, and building that is a real project.
Why do we get different answers to the same question? Large language models are non-deterministic. This is why validation requires an evaluation set run repeatedly with tolerances, rather than checking a single output once.
Further reading
Tagged with:
Ready to Automate Your Business?
Let's discuss how AI automation can deliver measurable ROI for your organization in 90 days or sooner.