On-Device or Cloud AI: A Decision Framework for Product Teams

Choosing where AI inference runs is an architecture decision with consequences for users, operations, and finances. On device execution can avoid a network round trip and keep selected data local, but it must fit the device's compute, memory, storage, thermal, and energy limits. Cloud execution can expose larger or frequently updated models and centralized controls, but it introduces data transfer, service dependency, variable latency, and ongoing capacity costs. The useful unit of analysis is not the entire app. It is one inference path: a defined model or service, input, output, quality target, response time objective, device population, and usage pattern. Apply the following gates to each path. The result may be local, cloud, or a deliberately hybrid design. Gate one: define the service promise before choosing a location Write down what the feature must do and how failure appears to a user. Specify an accuracy or quality acceptance test, latency target at a stated percentile, expected requests per active user, supported device classes, offline requirement, geographic reach, accessibility needs, and the maximum acceptable stale model period. Separate interactive work from batch work: a camera effect and an overnight document index have different timing constraints even inside the same product. Include the full pipeline. Tokenization, image resizing, retrieval, safety checks, tool calls, and post processing may dominate a small model's inference time. A local model that calls a remote search service is not an offline feature. A cloud model whose inputs are first processed locally may transmit much less data than a simple cloud versus device diagram suggests. Create a representative evaluation set before comparing deployments. It should cover difficult inputs, languages, device tiers, network states, and safety relevant cases. Do not use vendor demos as the acceptance test. Gate two: map every data action and privacy obligation List what the feature collects, derives, transmits, logs, caches, uses for improvement, and deletes. The NIST Privacy Framework https://www.nist.gov/privacy framework treats privacy as enterprise risk management across data processing, which is broader than encrypting a request in transit. Location changes the data flow, but it does not automatically settle privacy. Strictly local processing can reduce disclosure to a service operator and may support sensitive or offline use. It can still create privacy risk through local logs, model outputs, shared device storage, backups, telemetry, or another app component. Cloud processing can be appropriate when contracts, region selection, retention controls, access controls, auditability, and deletion behavior satisfy the use case. Verify those properties for the exact service and configuration rather than assuming a provider wide policy applies. Use data minimization before deployment selection. Redact or transform inputs locally, send only required features, disable unnecessary retention, and separate operational telemetry from user content. When legal or contractual rules constrain processing locations, qualified privacy and legal reviewers must evaluate the actual data flow. Gate three: prove that the model fits the device population A model that runs on a developer's newest phone is not yet an on device product. Measure package and downloaded model size, peak memory, cold start time, sustained latency, battery or energy effect, thermal throttling, and behavior when the operating system reclaims resources. Test the lowest supported tier and the hardware distribution users actually have. Platform frameworks can use specialized accelerators, but support varies by model operations, operating system version, and chip. Apple's Core ML documentation https://developer.apple.com/documentation/CoreML?language= 1 states that it uses the CPU, GPU, and Neural Engine and that strictly on device execution removes the need for a network connection. Google's LiteRT NPU documentation https://ai.google.dev/edge/litert/android/npu explains that hardware delegates are supplied for particular NPUs. These are platform capabilities, not proof that a specific converted model will meet its target on every device. Quantization, pruning, distillation, and smaller context windows can improve fit, but each optimization needs quality regression testing. Also plan model delivery and rollback. Bundling increases app size; downloading later requires storage, integrity checks, version compatibility, and a network at least once. Gate four: measure latency, quality, and reliability end to end For local inference, report cold and warm performance, sustained runs, and energy under realistic competing workloads. For cloud inference, include connection setup, upload, queueing, model processing, streaming behavior, retries, and output transfer. Test from relevant regions and poor networks. An average hides the tail that users experience. Use standardized benchmarks to understand hardware, then run the product workload. MLPerf Inference: Mobile https://mlcommons.org/benchmarks/inference mobile/ defines workloads, quality targets, scenarios, and latency metrics for comparable mobile submissions. Its results can inform platform selection, but a benchmark score from a different model or software stack does not predict an app's exact latency or output quality. Reliability differs by location. Local inference can continue without a service connection but may fail on unsupported devices or under memory and thermal pressure. Cloud inference can centralize redundancy and fixes but depends on network and provider capacity. For example, Vertex AI's throughput documentation https://docs.cloud.google.com/vertex ai/generative ai/docs/resources/throughput quota distinguishes shared pay as you go capacity, which can return resource exhaustion responses during demand, from reserved provisioned throughput. Treat that as one provider's documented service behavior, not a rule for every cloud. Gate five: compare lifetime cost at the same quality target Cloud cost is not only a model's listed input and output rate. Include network transfer where applicable, retrieval and storage, safety services, observability, reserved capacity, retries, regional deployment, engineering, and support. Model demand by active users and usage distribution, not registered accounts. Run low, expected, and peak scenarios because per request spending scales with use while reserved capacity can create idle cost. On device inference avoids a per request cloud inference bill but is not free. Account for model optimization, testing across hardware, larger downloads, update infrastructure, support for incompatible devices, battery impact, and the opportunity cost of reserving device resources. Hardware cost may be borne by users, bundled into a product, or controlled by an enterprise fleet; state whose budget is being measured. Compare alternatives over the product's expected life and include migration. A cloud model can be replaced centrally. A local model may remain on devices that do not update promptly. Conversely, a cloud API price, quota, model version, or availability can change. Architecture should isolate providers and model versions where the switching risk justifies the work. Use hybrid execution only with an clear routing contract Hybrid design is often practical: a small local model handles immediate or sensitive work, while the cloud handles tasks that exceed local capability. Other patterns include local preprocessing before cloud inference, cloud generated embeddings cached on a device, or a local fallback when the network is unavailable. Define routing from observable conditions such as device capability, consent, connectivity, input size, model confidence, and task risk. Do not silently send content to the cloud after presenting a feature as on device. The interface and privacy notice should reflect actual routing. Make outputs distinguishable in telemetry so quality and incidents can be traced to the path that produced them. Fallbacks need product semantics. If the local model is less capable, decide whether to return a constrained result, queue work, or explain that the feature is unavailable. A hidden downgrade can be worse than a visible delay when correctness matters. Prevent retry loops that perform both local and remote work repeatedly or create duplicate charges. Finish with a controlled pilot and a written decision Build the smallest representative version of each viable path. On a fixed evaluation set, capture output quality, latency percentiles, failure rate, bytes transmitted, peak memory, energy or battery effect, cloud units consumed, and operational work. Repeat across device tiers and network conditions. Record the tool versions, model versions, prompts, and sampling settings so results can be reproduced. Apply hard gates first: privacy or residency constraints, offline operation, minimum quality, supported hardware, and response time objectives. Then compare total cost and maintainability among the designs that passed. Weight criteria before seeing results to reduce convenient post hoc scoring. Document the selected route, rejected alternatives, evidence, assumptions, owners, and review triggers. Revisit when the model, device mix, request volume, price, regulation, or service objective changes. The best answer can differ by feature and can change over time. On device and cloud AI are not opposing product identities. They are execution locations with different constraints. A defensible architecture makes those constraints measurable, gives users an accurate account of data movement, and preserves a fallback that behaves as designed. Sources NIST Privacy Framework https://www.nist.gov/privacy framework Core ML https://developer.apple.com/documentation/CoreML?language= 1 LiteRT delegate for NPUs https://ai.google.dev/edge/litert/android/npu MLPerf Inference: Mobile https://mlcommons.org/benchmarks/…