Running an LLM on premise in industry: hardware, small models, operations

Running an LLM on premise means you are not buying answers, you are buying a piece of plant - with sizing, a maintenance plan, a backup regime and a named person who gets the call when it stops. Three questions decide whether it is worth it: may the data leave the building, how large does the model really have to be, and who operates the system once the project is over? This article gives you the calculation from user count to GPU, the case for small models, and the situations in which an EU cloud is simply the saner choice.

Why run an LLM on premise at all - and when the argument does not hold

Three reasons justify running a language model yourself: data sovereignty, availability without an internet connection, and cost control at a high sustained volume. Three others are cited often and do not survive scrutiny: a general discomfort with the cloud, an assumed saving at low volume, and a blanket rejection of external services. Tab. 01 sets the two sides side by side.

The Bundesamt für Sicherheit in der Informationstechnik (BSI, the German Federal Office for Information Security) lists the lack of confidentiality of submitted data as risk R2 in "Generative KI-Modelle: Chancen und Risiken für Industrie und Behörden" (version 2.0, January 2025): where the model is an internet service, the operator can access the data and use it for training. Running the model yourself eliminates exactly this risk - and only this one. Of the 28 risks the BSI lists, most persist inside your own network.

Local operation shifts responsibility, it does not remove it. Anyone who wants to self-host a language model takes on patching, testing, backup, access control and on-call duty. Local AI in the enterprise is therefore more of an organisational decision than a technical one.

Small models instead of large ones: the real precondition

A general-purpose model with several hundred billion parameters is oversized for the typical tasks in a plant. Sorting fault reports into twenty categories, pulling nominal diameter and material out of a data sheet, normalising part numbers - these are narrow, recurring tasks with little variation. For precisely this case, NVIDIA Research (Belcak et al., 2025) argues that small language models are capable enough - and, in their words:

For self-hosting this is the decisive fork in the road, because the memory needed for the weights scales with parameter count times bytes per parameter. According to the Hugging Face Transformers documentation, weights are stored in fp32 at 4 bytes or in fp16/bf16 at 2 bytes; quantisation reduces them to int8 or int4 while trying to preserve as much accuracy as possible. Tab. 02 shows this as a worked example.

The trade-off is real: quantisation costs accuracy, and how much depends on the model, the method and the task. An int4 model may still classify fault reports cleanly and visibly degrade on tolerances read out of tables. That is why a test on your own documents has to come before every quantisation step. The Mittelstand-Digital Zentrum Chemnitz (a publicly funded SME digitalisation centre) notes that thanks to Retrieval Augmented Generation (RAG) smaller language models are often sufficient and can be run on premise, which removes the security-critical transfer of data to third parties.

A second argument for the Small Language Model in industry is proximity to the machine. A model with a few billion parameters runs on a workstation GPU in the control room and stays available when the WAN link fails - more important for a fault assistant at the line than the last nuance of language quality. Which tasks in technical documentation are suited to a language model at all is covered in AI for technical documentation.

GPU sizing: from user count to hardware

GPU sizing for an LLM starts with five quantities: concurrent users, requests per hour, average context length, average response length, and the acceptable wait until the first token. From these follow memory demand (weights plus KV cache) and throughput. The KV cache is the item almost every plan underestimates.

A worked example with the figures NVIDIA quotes for Llama 2 7B: 32 layers, hidden size 4096, fp16. Per token that is 2 × 32 × 4096 × 2 bytes, roughly 0.5 MB; at 4,096 tokens NVIDIA puts this at about 2 GB. With a 16,000-token context for document analysis that is roughly 8 GB per request; eight concurrent requests occupy about 64 GB - on top of 14 GB of weights. This is why context length is the main driver, not user count. Newer architectures need less cache per token; the figures are in the model card.

There is a throughput dimension as well. The decode phase, in which the model generates token by token, is memory-bandwidth-bound according to NVIDIA - latency depends on how fast weights and cache can be read out of GPU memory. Memory size therefore limits concurrency, memory bandwidth limits response time.

1. 01

   Record the load profile

   Concurrent users in the peak hour, requests per hour, tokens per prompt including the retrieved document passages, tokens per response.

2. 02

   Fix the model class

   Use 50 to 200 of your own documents to determine the smallest model class that solves the task reliably - in fp16 and quantised.

3. 03

   Calculate the memory

   Weights per Tab. 02 plus KV cache for peak load plus headroom for the embedding model and the runtime gives you the memory class of the GPU.

4. 04

   Check throughput against wait time

   A batch extraction job may take minutes, an assistant at the line may not; the acceptable time to first token decides.

5. 05

   Test under load before you buy

   Measure on rented hardware with a real load profile, and only then place the order - with a date stamp and a review reminder on file.

What else is needed besides the GPU

An AI server on your own hardware is more than the graphics card; inference is rarely the bottleneck, the ingest path almost always is. Every component has to be installed, monitored and backed up.

Embedding model

Turns text passages into vectors. If it is swapped out, the index has to be rebuilt.

Vector index

Stores vectors, text passages and metadata such as source, version and permission. A second body of data with its own protection requirement: backup, permission filter, deletion path.

Ingest pipeline

OCR, layout recognition, table extraction, chunking, re-indexing on change. This is where most errors arise, and it often consumes more compute than inference does.

Reverse proxy with authentication

Connection to the directory service, roles, per-user request limits, separation of network zones.

Monitoring

Latency, error rate, GPU utilisation, aborted requests, requests per user - without these figures you can neither assess a model change nor detect an attack.

Infrastructure

Rack space, power and cooling for several hundred watts per GPU, network connectivity, a UPS for an orderly shutdown.

Operations: the part nobody writes about

The most important question is not which GPU, but who will be running the system in eighteen months. According to Bitkom Research, Germany is short of roughly 79,000 IT specialists in 2026 (survey weeks 18 to 25 of 2026); the most sought-after group, at 26%, are specialists for IT administration and systems operations - exactly the profile a local language model needs. A system built by an external party and handed over to an already saturated IT department with no hours budget is the most common pattern of quiet decay.

The second underestimated item is the model change. A new model or a new quantisation shifts answers unevenly: what was extracted correctly yesterday can be wrong today. In measure M11 the BSI recommends repeating tests periodically and in response to specific events, and looking at the whole system with all its interfaces when doing so.

Security inside your own network: on premise does not mean secure

Self-hosting takes exactly one risk off the table - an external operator having access to your inputs. Every other attack surface stays your responsibility. The OWASP GenAI Security Project lists them in the Top 10 for LLM Applications 2025, the BSI in its system of measures. Tab. 04 maps the points that matter most for a document system in a plant.

Two points deserve emphasis. First, indirect prompt injection: in R28 the BSI describes how attackers can hide instructions in documents that the model processes - down to characters that are processed but not displayed. A supplier data sheet in the index is therefore an input from outside, even if the server sits in your own basement. Second, the permission concept: in M14 the BSI explicitly advises against implementing permissions and roles through textual instructions. Authorisation has to take effect as a filter before retrieval, not afterwards as a polite request in the prompt.

On resource limits, OWASP names strict validation of input size, rate limits, user quotas and timeouts under LLM10. In the cloud such an attack produces unexpected invoices - OWASP calls it denial of wallet - while on your own hardware it produces a system at a standstill. On logging, the BSI points out legal requirements in M19: purpose, pseudonymisation and retention period belong before the first line of log.

The cost structure, broken down honestly

The cost of a local language model consists of several blocks, of which the hardware is the most visible but rarely the largest. There is deliberately no price list: hardware, electricity and cloud prices change faster than this article. Tab. 05 shows where the blocks sit for self-hosting and for a usage-based EU cloud.

The volume at which the arithmetic tips can only be worked out with your own figures, but the logic is always the same: annual fixed cost from hardware, electricity, staff and on-call against the usage-based price times the actual annual volume. A GPU processing delivery notes around the clock pays off differently from one waiting for twenty questions a day. How to estimate setup and integration realistically is covered in What custom industrial software costs.

When an EU cloud is the saner choice

An EU cloud is the saner choice when at least two of these apply: irregular or unknown volume; no team to run the system; a need for models that will not run on your own hardware; a pilot with an open outcome; data with no special protection requirement. Running AI without the cloud is then not a value in itself, only a cost block without a return. Tab. 06 condenses the decision into eight criteria.

The hybrid middle road - index and documents local, language model external - makes technical sense, because the laborious ingest stays in house and the model becomes interchangeable. In data protection terms, however, it is not local operation: with every request the retrieved document passages go to the provider. The Mittelstand-Digital Zentrum Chemnitz credits local operation with eliminating the transfer of data to third parties; that benefit does not apply here, and the hybrid route needs the same review as a cloud scenario.

Conclusion: shrink the model first, then size the hardware

An LLM on premise is worth it when data must not leave the building, when the application has to run without the internet, or when there is a steady high volume - and when somebody is there to operate the plant. The economics come not from the GPU but from the smallest model that solves the task reliably. Calculate weights and KV cache for your actual context length, test under load before you buy, and build the regression set before you go live.

If two or more criteria from Tab. 06 argue against self-hosting, an EU cloud or a hybrid setup is the more honest answer. If you want to work out which model class and which operating route suit your own document base, that is a typical question for ralwo's AI solutions.