Definitions · AI & GPU compute
What is AI inference?
AI inference explained: what the term means, which hardware and constraints matter, and how inference capacity is listed, metered and settled.
AI inference is running a trained machine-learning model so that it produces output from new input - a classification, a prediction, a generated token, a detected object. Training is what built the model; inference is what the model does every time it is asked a question, which in a deployed system is continuously.
The two stages share hardware but differ in shape. Training is one long parallel job that a cluster absorbs for days and judges by throughput per run. Inference is a service judged by how quickly it answers, how many concurrent requests it absorbs, and how much capacity sits idle between them. Most practical decisions about inference hardware follow from that asymmetry.
Inference versus training
A team that has trained a model already knows part of the answer: the same accelerator classes serve both workloads. What changes is the profile. Training wants large batches, deep parallelism and a scheduler that keeps hardware busy for hours; it is judged once at the end and re-run when the data or the objective changes. Inference wants predictable response times under a load curve nobody chose, and it runs for as long as the model stays in service. The other half of the pair is described in AI training.
That divergence shows up in capacity planning. Training demand arrives in peaks and is scheduled around them; inference demand is sized against a baseline plus the headroom an endpoint tolerates. Some operators run both on one cluster, queueing batch training behind interactive traffic. Others separate them, because the failure modes and the buying patterns are different enough to deserve separate budgets.
What an inference workload needs
Choosing machines for an endpoint comes down to constraints that no spec sheet states directly:
- Model residency - enough accelerator memory to hold the weights, plus the cache and activation headroom the serving stack needs for the context you accept.
- Batching behaviour - whether throughput comes from queueing many requests together or from answering one request immediately.
- Latency budget - interactive endpoints are dominated by time to first result; offline scoring jobs are dominated by total throughput.
- Host and fabric paths - how quickly tensors move between accelerator, system memory and network.
- Concurrency - how many simultaneous sessions an endpoint holds before queues grow and answers slow down.
None of these appear in a single GPU model number. They are properties of the model, the serving runtime and the machine taken together, which is why a result measured on one configuration says little about another.
The cost shape of a serving endpoint
Because inference runs continuously, its cost is dominated by what idles between requests. A model that answers in bursts still pays for the memory holding the weights resident, so teams size for residency first and throughput second - and some workloads suit a smaller accelerator that stays warm rather than a larger one that is mostly empty. Capacity that can be released when the workload stops is the practical answer to that, and it is a leasing decision before it is a hardware one.
Which is why most teams rent rather than buy: model demand moves, a rewrite halves the memory footprint, a new release changes the batching behaviour. Renting converts that volatility into a parameter. The spectrum runs from a virtual machine with an accelerator attached, through a managed platform that takes a container and a port, to a hosted endpoint where the provider runs the serving stack too. Each step further along gives up control of the runtime for less work on your side - guidance for sizing the whole workload is in AI and ML workloads.
Inference capacity on the open market
On VirtEngine, inference capacity is not a phone call with an account manager. A provider describes what it can supply - accelerator model and count, memory, fabric, host resources, the region the machines sit in - and publishes that description as a listing. A buyer either takes a listed offering at its published price or posts an order saying what is needed and lets providers bid against it. The mechanics of that exchange are covered in how the marketplace works, and the accelerator category itself is GPU as a Service.
Once matched, the agreement becomes a lease backed by funded escrow: the budget is provably committed before the first model loads, so the provider serves no stranger's promise, and the tenant prepays nothing that settlement rules have not authorised. Usage is metered while the endpoint runs, submitted in signed batches, held open for a dispute window, and only then drawn down from escrow. Both sides read the same record from meter to payment, which is what allows either of them to buy time on hardware they have never stood in front of.
In practice
A team serving a model orders a GPU listing whose memory and fabric match its runtime; the lease meters while the endpoint answers and closes when the endpoint is torn down.
Questions
Asked about what is ai inference
How does AI inference differ from model training?
Training builds the model and consumes one long, highly parallel run; inference executes the finished model on new input and repeats for as long as the service is live. They share accelerator hardware but are measured differently - throughput per run against response time under load.
What hardware is suited to AI inference?
It depends on the model rather than the marketing category: accelerator memory must hold the weights plus serving headroom, and the host and fabric paths must move tensors fast enough for the latency budget. Small models can serve well on modest hardware; large models need memory first and everything else second.
Why is inference billed differently from training?
Training is a finite job with a start and an end, so it is easy to treat as a batch of capacity. Inference is a long-lived service whose load varies by hour, which makes metering and the ability to release capacity the deciding factors in what it costs.
How is AI inference bought on VirtEngine?
As a listing or an order. Providers publish accelerator capacity with its specifications, tenants either buy at the listed price or let providers bid on a posted order, and the resulting lease is funded by escrow and settled from metered, signed usage records.
Related
Keep reading
- What is AI training? What is AI training? Fitting a model to data on accelerators: what a run needs, how it differs from inference, and where the capacity comes from.
- What is GPU as a Service? GPU as a Service rents accelerator capacity instead of buying servers. What the model covers, what a listing must state, and how usage settles.
- What is a compute marketplace? A compute marketplace matches demand for capacity with providers who have it. How orders, escrow and signed usage become settled payment.
- What is HPC (high-performance computing)? What is HPC? High-performance computing joins many processors to finish large jobs faster: what clusters need, and how compute is bought per job.
- All definitions Every cloud and marketplace term, defined in one place.
Longer explainers live in Learn; the full vocabulary is in the marketplace glossary.