Choose an AI Accelerator by Workload, Not Peak Compute

An AI accelerator cannot be ranked responsibly by one peak compute number. A chip reaches useful performance only when the model, numerical format, memory system, interconnect, compiler, runtime, and serving pattern work together. A device that is efficient for large batched matrix multiplication may be a poor fit for a memory constrained model, an irregular recommendation workload, or an interactive service with small batches. The practical unit of comparison is therefore a complete system running the intended workload under the required accuracy, latency, reliability, and cost constraints. CPUs, GPUs, and application specific accelerators expose different balances of flexibility and specialization. None is automatically the right answer for every training or inference job. Start with the workload envelope Describe the work before looking at product tables. Record whether the system will train, fine tune, or serve a model; the model family and parameter footprint; input and output shapes; supported numerical formats; expected request rate; latency objective; batch size limits; context or sequence lengths; and the required quality threshold. Include preprocessing, retrieval, postprocessing, and data transfer because accelerator time is only one part of the user visible path. Separate steady workloads from bursty ones. A continuously utilized training cluster presents a different economic problem from an inference endpoint that sits idle between traffic spikes. Identify whether the model must fit on one device, can be partitioned across devices, or needs frequent movement between host and accelerator memory. For distributed training, document the parallelism strategy and communication pattern rather than treating device count as a simple multiplier. This envelope prevents a common error: buying for a theoretical operation rate that the actual graph cannot sustain. It also establishes the tests that every candidate must pass without quietly changing model quality or service behavior. Compare the execution model, not just the processor label A CPU is a general purpose processor with mature control flow support and broad software compatibility. It can be appropriate for small models, irregular operations, preprocessing, orchestration, or services whose volume does not justify a dedicated accelerator. A GPU exposes large amounts of parallel execution and a memory hierarchy suited to many tensor operations, while retaining programmability for kernels beyond matrix multiplication. An AI ASIC narrows the design around selected machine learning operations and data movement, potentially gaining efficiency when the workload maps well to that design. The labels still hide important differences. Google documents Cloud TPU as a matrix oriented ASIC whose tensor cores combine matrix multiply, vector, and scalar units, with high bandwidth memory and specialized inter chip connections. AMD documents its GPU families through architecture references, memory hierarchy, matrix units, instruction sets, and performance counters. NVIDIA's performance guide similarly describes a GPU as parallel processing elements connected to caches and high bandwidth device memory. These are architectural descriptions, not proof that one vendor wins a particular application. Ask which operations are accelerated and which fall back to general vector, scalar, host, or custom kernel paths. Inspect support for the exact tensor shapes and numerical formats the model uses. If a candidate requires graph rewrites, operator substitutions, or reduced precision, measure whether those changes preserve the application's acceptance criteria. Treat memory capacity and movement as first class limits Model weights are only part of device memory demand. Activations, attention caches, optimizer states, gradients, temporary workspaces, communication buffers, and runtime overhead can determine whether a workload fits. Quantization may reduce some storage and bandwidth requirements, but its effect depends on runtime support and validated output quality. Memory bandwidth matters when operations perform relatively little computation per byte moved. NVIDIA's vendor specific background guide explains the general roofline idea: performance can be limited by math throughput, memory bandwidth, or insufficient parallelism and latency. It notes that operations such as activations and normalization often have lower arithmetic intensity than large matrix operations. The transferable lesson is to profile where time and bytes go instead of assuming more matrix units will accelerate every layer. Also map transfers outside device memory. Host to device links can bottleneck ingestion or offload. Multi device jobs depend on topology and collective communication. Large scale systems can be limited by interconnect bandwidth, synchronization, network contention, or storage throughput. Include these paths in the benchmark and in capacity planning. Evaluate precision, quality, and software as one system Lower precision formats can improve throughput or reduce memory use when hardware and software support them, but the available formats and accumulation behavior differ by architecture. Validate the complete model on representative data after any precision change. A speed result is not comparable if one system misses the required quality target or uses a materially different model. The software stack can outweigh a favorable specification. Inventory framework support, compiler maturity, kernel coverage, distributed libraries, model serving runtimes, profilers, debuggers, container images, orchestration integration, and the team's existing skills. Check how quickly the stack supports new model operators and how upgrades are tested. Portability layers may reduce source changes, but they do not guarantee identical performance or numerical behavior across architectures. Create a small portability test before procurement. Compile or convert the real model, run its unsupported operation report, profile representative requests, exercise failure recovery, and upgrade one runtime version. Record custom code that would tie the service to a compiler, device API, or proprietary model format. Lock in is not automatically unacceptable, but its migration cost belongs in the decision. Use benchmarks without importing the wrong conclusion MLPerf Inference provides architecture neutral benchmark definitions, reference implementations, accuracy requirements, load generation rules, and multiple scenarios for datacenter and edge systems. Its datacenter material distinguishes scenarios and metrics rather than presenting one universal score. That makes published results useful evidence about a declared system and benchmark configuration. They are not a substitute for an application benchmark. Compare only compatible benchmark versions, divisions, availability categories, accuracy targets, scenarios, system boundaries, accelerator counts, and power methods. MLCommons notes that full system power measurements apply to the accompanying benchmark; a device power rating is not the same measurement. Read submission details and change logs rather than extracting the largest number from a table. Build an internal test that fixes the model version, quality gate, dataset or request trace, warm up procedure, concurrency, batch policy, duration, and software versions. Report latency distributions as well as throughput for interactive services. Measure failures, throttling, compilation time, and recovery behavior. Repeat enough runs to expose variance, and disclose every optimization that changes the workload. Calculate total cost around the measured service Acquisition price or hourly accelerator price is only one line. Include host processors, memory, storage, networking, power delivery, cooling, rack capacity, support, software licenses, cloud data transfer, reservations or commitments, spare capacity, engineering labor, and downtime risk. For owned hardware, include installation, depreciation assumptions, utilization, maintenance, and the opportunity cost of capacity that cannot be reassigned. For rented hardware, model idle time, queue delays, regional availability, and the cost of keeping fallback capacity. Normalize cost to a useful unit that preserves service requirements, such as accepted requests at the required latency and quality, completed training runs, or validated tokens under a fixed test. Do not use cost per raw accelerator hour to compare systems that finish different amounts of acceptable work. Finally, score operational constraints: supply or quota, deployment lead time, observability, security controls, failure domains, and staff readiness. A slower system may be the better production choice if it is supportable, available, and predictable. Conversely, a fast accelerator that requires constant custom tuning may have a higher real cost than its benchmark suggests. Make the decision reversible Shortlist architectures from the workload envelope, then run the same acceptance test on each complete candidate system. Keep the winner behind a stable serving or training interface where practical. Preserve models in portable formats when that does not sacrifice required capability, automate conformance tests, and retain a CPU or alternate accelerator path for essential operations where feasible. Revisit the choice when the model, traffic shape, numerical format, software stack, or pricing changes. The correct accelerator is not a permanent property of a chip. It is a measured match between a changing workload and a complete, supportable system. Sources GPU Performance Background User's Guide https://docs.nvidia.com/deeplearning/performance/dl performance gpu background/index.html AMD GPU architecture documentation https://rocm.docs.amd.com/en/latest/reference/gpu arch/index.html TPU architecture https://docs.cloud.google.com/tpu/docs/system architecture tpu vm MLPerf Inference: Datacenter https://mlcommons.org/bench…