AI hardware for on-premises use

Executive summary

Running AI on your own premises is a business decision before it is a technical one. Two questions are worth asking first:

• Is your IP in the cloud? If staff paste research notes, contract drafts or client files into hosted AI services, that material leaves your control, however good the provider's terms.
• Are you overspending on routine queries? Much day-to-day AI use is drafting, summarising and searching. Paying premium per-seat or per-tokenTokens: The small chunks of text a model reads and writes, often a word or part of a word. Glossary prices for all of it may not be necessary.

Many businesses end up with a mix: a model they control for routine and confidential work, and a hosted frontier service for the hardest tasks, with clear rules on what may go where.

Sizing rule of thumb

As a rule of thumb, a model needs roughly its file size in combined RAM and VRAMVRAM: The memory on a graphics card (GPU). Models run fastest when they fit in it. Glossary (graphics card memory) to run at usable speed. A 500 GB model needs roughly 500 GB. Download sizes are from the Open-weight model tracker (Hugging Face listings, 29 Oct 2026); the starting tier is our own grouping, explained below.

Small, medium and large starting points

These tiers are our own plain-English guidance, not a specification or a quotation. They describe where to start, not where you must end up. Not sure which fits? Try the self-check: is on-prem right for you?

Small: one workstation, a few users

• Suits: a team trying AI on confidential documents, or cutting routine spend, with a handful of users.
• Runs: compact open-weightOpen-weight model: A model whose weights the lab publishes, so you can download it and run it on your own hardware. Glossary models whose 4-bitQuantisation: Storing a model's numbers at lower precision (for example 4-bit) so it needs less memory. Glossary downloads are roughly 18-25 GB (for example Qwen3.8-27B, Gemma 4 31B, Recursive Flint).
• Takes: a single workstation-class machine with enough graphics memory or unified memory to hold the model, set up with sensible access controls. Lowest cost and the easiest way to test value before committing further.

Medium: a shared machine for one or more teams

• Suits: regular use across one or more teams on confidential material, often with document search over your own files.
• Runs: mid-sized models in roughly the 50-90 GB range at 4-bit (for example Nemotron 3.5 Bolt or Super VL), or several smaller models at once.
• Takes: a high-memory workstation or a small server, shared over your network, with user accounts, logging and backups set up from the start.

Large: server-class, governed like any core system

• Suits: high volumes, many users, the strictest data rules, or a need for the largest open models.
• Runs: large models in the 300-500 GB range at 4-bit (for example Reflection Beam 1.0 or DeepSeek V4.1-Flash-1017).
• Takes: multi-GPU or large-memory servers, proper power, cooling and space, and ongoing support, managed as part of your wider IT estate.

Security and compliance for on-premises AI

Security: residency, access and audit, and third-party logs

• Residency and sovereignty: prompts, documents and outputs stay on equipment you own, in a location you choose. That makes it easier to show clients and regulators where their data is, and to meet contract clauses that restrict data location or onward transfer.
• Access and audit: you decide who can use the model, which document collections each person can search, and how long logs are kept. Give each AI tool, and each person, only the minimum access they need. This principle of least privilege appears in most information security guidance, including the international standard ISO/IEC 27001. Keep an audit trail of who asked what, and review it regularly.
• Third-party logs: a model running locally sends no prompts or data to the lab that built it or to a hosted service. Nothing is retained in someone else's logs, used for someone else's training or exposed by someone else's breach. Licence terms still apply to the model itself.

More on securing AI use: see Security in ISD Insights.

Compliance: data privacy, data protection and sector rules

Many countries and regions have their own data privacy and data protection laws, and most apply to personal data used with AI wherever the model runs. The details differ, but the same core ideas appear again and again. UN Trade and Development (UNCTAD) keeps a worldwide tracker of data protection and privacy legislation showing how widely these laws now apply.

• Data privacy: people have rights over how their personal data is used. Common examples are collecting data only for a clear, stated purpose, telling people how it will be used, and letting them see and correct what is held about them. The OECD Privacy Guidelines are a long-standing international reference for these principles.
• Data protection: the organisation holding the data must keep it safe. Common examples are appropriate security controls, limits on who can access it and how long it is kept, and reporting serious breaches. ISO/IEC 27701 is the international standard for managing privacy information alongside information security.
• Sector rules: financial services, healthcare, legal and public-sector work often add their own expectations on outsourcing, confidentiality and record-keeping. Check those, and your client contracts, alongside general data protection law.

Check which laws apply in each place you operate and wherever your data subjects are. On-premises does not remove these obligations; it can make them easier to evidence.

The honest flip side: you own the risk

• Patching and updates: the operating system, drivers, runtime software and models all need keeping up to date.
• Physical security: the equipment needs a secure, cooled location, with controlled access.
• Backups and recovery: configurations, document indexes and logs need backing up and a tested recovery plan.
• Supply chain: download models only from trusted publishers, and avoid re-packaged versions of unknown origin.
• Monitoring: someone needs to notice when something goes wrong.

This is where ISD's support comes in: we help businesses plan, set up and look after on-premises systems, so the control you gain does not become a maintenance burden.

Respected frameworks to build on

The most widely respected guidance does not tell organisations to avoid AI. It sets out how to use it safely. It falls into a few broad types, and each has a recognised place to browse a wider list. These are starting points, not a complete catalogue:

• AI risk management frameworks: voluntary frameworks for identifying, measuring and managing AI risks across an organisation, such as the NIST AI Risk Management Framework. For a wider list, the OECD keeps a Catalogue of Tools and Metrics for Trustworthy AI.
• AI application security: the common weaknesses in applications built on language models, such as prompt injection and sensitive information disclosure, and how to defend against them. The OWASP Top 10 for LLM Applications is the best-known list; the OWASP GenAI Security Project and MITRE ATLAS, a knowledge base of real-world attack techniques against AI systems, go further.
• International management standards: certifiable standards for running AI responsibly, such as ISO/IEC 42001 for an AI management system. Related AI standards are published by the joint ISO and IEC committee on artificial intelligence, ISO/IEC JTC 1/SC 42, in the ISO standards catalogue.
• Government and regulator guidance: many national cyber security agencies and data protection regulators publish practical guidance on secure AI development and use, often jointly with international partners. The OECD.AI Policy Observatory tracks AI policies and guidance across many countries, so you can find what applies where you operate.

A middle path: AI never touches live systems

For businesses that want the benefits of AI analysis without connecting AI to the systems they run on, there is a safer middle path:

• The business exports selected data, on purpose, into an isolated analysis environment (sometimes called an AI data clean room or AI warehouse).
• The data is analysed there, away from live systems.
• Results come back through controlled, read-only views.

AI tools in the environment never connect to live systems, and nothing flows back automatically. Read more on our isolated analysis environment page.

Further reading: independent testing

We link to independent testers rather than reproduce their charts: the figures and charts belong to their publishers. Each takes a different approach, so compare like with like (same model, same quantisation, same software).

• Artificial Analysis: hardware benchmarks: independent benchmarking firm testing datacenter systems, laptops and workstations, and phones.
• MLCommons: MLPerf Inference (datacenter), edge and MLPerf Client: industry-consortium benchmarks with published rules; results are submitted by vendors and reviewed.
• llama.cpp: performance on Apple Silicon: community-maintained results for the same small model across Mac chips, from the llama.cpp project.
• Puget Systems Labs: detailed workstation GPU testing with published methods. Puget also sells workstations.
• ServeTheHome: in-depth reviews of servers and compact AI systems, often of vendor-supplied review units.
• llm-speed open dataset: speed results from its own and community runs, published under an open (CC BY 4.0) licence.
• OpenBenchmarking.org: llama.cpp: crowd-sourced results from the Phoronix Test Suite across many CPUs and GPUs.
• Alex Ziskind on YouTube: hands-on video testing of local AI on Macs, PCs and compact AI systems. Many videos are sponsored; check each video's disclosures.

Benchmark speeds depend heavily on software versions and settings, and change often. Treat any single number as a snapshot.

Power, cooling and space considerations:

Data-centre GPUs draw 700 watts or more each, with the newest above 1,000; workstation cards such as the RTX 5090 draw around 450 to 600, so plan for PSU headroom, circuit limits and UPS sizing. Cooling scales with that heat, so airflow and ambient temperature decide whether air or liquid cooling works. Space here means rack footprint and weight, not storage — VRAM affects how many cards you need, while disk only holds the model weights. Contact ISD at [email protected] for further information.

Buy, lease or rent considerations:

Buying ties up a lot of cash upfront but you own the asset and can depreciate it. Leasing spreads the cost monthly and often includes maintenance, but you pay a premium over the life of the contract. Renting is the most flexible for short-term or burst needs, yet the priciest per month if you hold it long. GPU generations move fast, so a three-year lease can leave you stuck with outdated silicon while a competitor upgrades. If the hardware sits idle most of the time, renting beats owning. Contact ISD at [email protected] for further information.