A cheat sheet for the terms that come up most often in ISD Insights. In our articles, the first use of a few key terms is marked with a dotted underline: hover over it, tap it or move to it with the keyboard to see the short definition, with a link back here.
Agentic AI · Context window · Inference · Isolated analysis environment (AI data clean room) · LLM (large language model) · Mixture of experts (active parameters) · Open-weight model · Parameters · Quantisation · RAG (retrieval-augmented generation) · Runtime (for example llama.cpp) · Tokens · Unified memory · VRAM
- Agentic AI
AI set up to take actions (for example opening files or calling other systems), not just answer questions.
Because an agent can act, the access it is given matters. Regulators such as Hong Kong's PCPD advise giving agentic AI the minimum access rights it needs.
- Context window
How much text a model can take into account at once, measured in tokens.
A larger context window lets a model work across long documents or many files in one go. Using the full window needs more memory and is slower, so it is not always practical on smaller machines.
- Inference
Running a trained model to get answers, as opposed to training it.
Almost all business use of AI is inference. Its cost depends on the model size, the hardware and how many requests you send. Training a model from scratch is a separate, far more expensive activity that few businesses need.
- Isolated analysis environment (AI data clean room)
A separate, controlled environment where selected data is analysed, away from live business systems.
Data is exported into the environment on purpose, analysed there, and results come back through controlled, read-only views. AI tools in the environment never connect to the systems the business runs on. See our service page.
- LLM (large language model)
An AI model trained on large amounts of text to read, summarise and write language.
Most of the AI assistants and chat tools in the news are built on an LLM. The model itself is a large file of learned numbers (its weights) plus the software that runs it. It predicts text one piece at a time, which is why it can draft and summarise well but can also state things that are wrong with confidence.
- Mixture of experts (active parameters)
A design where only part of the model (the active parameters) works on each piece of text.
A mixture-of-experts model might have hundreds of billions of parameters in total but use only a small share of them for each token. That makes it faster than its total size suggests, but the whole model still has to fit in memory.
- Open-weight model
A model whose weights the lab publishes, so you can download it and run it on your own hardware.
Open-weight is not the same as open source: the training data and code are often not published, and each model comes with its own licence. Running an open-weight model in-house means prompts and data do not go to the lab, but the licence terms still apply. Closed models, by contrast, are only available through the lab's own service.
- Parameters
The learned numbers inside a model. More parameters usually means a more capable but larger model.
Parameter counts are written in billions (B) or trillions (T), for example 27B. The count is a rough guide to capability and to the memory needed to run the model; it is not a measure of quality on its own.
- Quantisation
Storing a model's numbers at lower precision (for example 4-bit) so it needs less memory.
A 4-bit build needs far less memory than the original 16-bit model and runs on smaller machines, usually with a modest loss of accuracy; actual file sizes vary by model and method. The download sizes in our tracker are for quantised builds.
- RAG (retrieval-augmented generation)
Giving a model relevant passages from your own documents at question time, so its answers draw on them.
RAG lets a model answer from your policies, contracts or manuals without retraining it. The quality of the answers depends heavily on how the documents are prepared, indexed and kept up to date, and on who is allowed to see which documents.
- Runtime (for example llama.cpp)
The software that loads a model and runs it on your hardware.
llama.cpp, Ollama, LM Studio and vLLM are common runtimes. A model can only be run locally once a runtime supports it, which is why support in a widely used runtime is news.
- Tokens
The small chunks of text a model reads and writes, often a word or part of a word.
Hosted AI services usually charge per token, and a model's speed is measured in tokens per second. Longer prompts, documents and answers use more tokens, so they cost more and take longer.
- Unified memory
One pool of memory shared by the processor and the graphics chip.
Machines with large unified memory can hold bigger models than a single graphics card can, although they generate text more slowly than high-end graphics cards with fast dedicated memory.
- VRAM
The memory on a graphics card (GPU). Models run fastest when they fit in it.
As a rule of thumb, a model needs roughly its file size in combined VRAM and system RAM to run at usable speed. Some machines share one pool of memory between the processor and graphics (unified memory), which lets them hold larger models.