Local or Cloud AI? Use a Placement Scorecard

The choice between local and cloud AI is not a referendum on which environment is universally safer, cheaper, or faster. It is a placement decision for a specific model, data flow, user population, and operating team. Local can mean a phone, workstation, factory gateway, private server, or organization operated datacenter. Cloud can mean rented infrastructure, a managed model endpoint, or a complete application service. Hybrid systems divide work among several of these locations. A useful decision starts by decomposing the application. Training, fine tuning, retrieval, model inference, telemetry, and human review do not have to run in the same place. Score each component against the same requirements, test the leading placements with real traffic, and document which party controls each layer. Define the three placement options precisely For this decision, call a component local when the organization or user runs it on equipment at the site or device where the data is produced or consumed. The organization may own that equipment, but ownership alone does not establish security or operational control. A managed appliance at a customer site can still depend on a vendor control plane. NIST defines cloud computing through characteristics including on demand self service, broad network access, resource pooling, rapid elasticity, and measured service. It also distinguishes service models and public, private, community, and hybrid deployment models. Those categories matter because renting a virtual machine gives the customer different responsibilities from calling a managed AI API. Hybrid is an architecture, not a compromise label. A hybrid design might keep sensitive retrieval data and a small inference model on site, send approved requests to a cloud model for complex tasks, and centralize redacted telemetry. Every boundary needs an clear protocol, failure behavior, identity model, and data retention rule. Build a workload card before scoring locations Write a one page card for each component. Record the model and version, memory and compute requirements, request shape, expected concurrency, acceptable latency, availability target, offline requirement, update frequency, and quality evaluation. List every input, output, prompt, retrieved record, embedding, cache, log, and human annotation it touches. Classify the information according to applicable organizational policy and law. Document where data may be processed, stored, backed up, and inspected; who may administer the system; and whether a third party service may use content for abuse monitoring, support, or product improvement. Contract terms and service configuration must be verified directly because a generic cloud or local label cannot answer those questions. Define exit conditions too. How quickly must the component be moved, replaced, or disabled? Can model artifacts and logs be exported? Does the application depend on a proprietary API behavior? Placement that works during a pilot can become expensive if migration was never designed. Score data boundaries and security separately Local processing can reduce transmission of raw data and can support strict residency or disconnected operation. It does not automatically prevent exposure. Local systems still need identity controls, encryption, patching, physical protection, secrets management, audit logs, model and dependency provenance, secure deletion, and incident response. A poorly maintained workstation may provide less protection than a well governed managed service. Cloud services can provide centralized identity, policy, logging, key management, and rapid security updates, while also adding provider personnel, control planes, network paths, tenancy, contracts, and supply chain dependencies to the threat model. Determine which controls belong to the provider and which remain with the customer at the selected service layer. Verify configuration and evidence rather than relying on a provider's general security statement. NIST's zero trust guidance rejects implicit trust based only on network location or asset ownership. Apply that principle to both placements: authenticate users, devices, workloads, and services; authorize access to the specific resource; minimize privileges; protect service to service calls; and monitor behavior. Keeping a model inside the office network is not an access control design. Measure latency, resilience, and user experience Local inference can avoid a wide area network round trip and continue during an internet outage, but device contention, limited memory, thermal throttling, battery use, or slow storage may dominate response time. Cloud inference can pool larger or specialized resources and scale with demand, but network variation, regional capacity, quotas, and service incidents become part of the path. Test end to end behavior at realistic input sizes and concurrency. Measure preprocessing, retrieval, queueing, inference, streaming, postprocessing, and transfer time, not only model execution. Record median and tail latency, timeout rate, accepted output quality, resource saturation, and recovery after a failure. Test the sites and devices users actually have, including weak connectivity and cold starts. Design a declared degraded mode. A local component may fall back to a smaller model when the cloud is unavailable. A cloud first application may queue nonurgent work or disable a feature rather than sending data to an unapproved endpoint. Define which outputs are safe to cache and how stale results are identified. Compare total cost under the same demand curve Cloud pricing converts much infrastructure spending into metered service and can match bursty or uncertain demand. Local deployment requires capacity in advance but can be economical for stable, highly utilized work when the organization can operate it effectively. Neither conclusion follows from list price alone. Use a common time horizon and demand trace. For local deployment, include hardware, spare capacity, installation, power, cooling, space, networking, warranties, software, monitoring, maintenance, refresh, disposal, staff time, downtime, and remote site support. For cloud deployment, include compute or API use, storage, networking and data transfer, observability, support, idle resources, commitments, regional redundancy, integration labor, and migration risk. The FinOps Framework defines total cost of ownership broadly enough to include acquisition, management, support, communications, labor, downtime, and training effects. It also emphasizes unit economics. Choose a service unit such as a successful classified document, an accepted generated response, or a completed batch at the required quality and latency. Cost per token or device hour can mislead when placements produce different user outcomes. Run sensitivity cases for low, expected, and high demand. Include growth in model size and context, not only request count. A local fleet can strand capacity when demand falls; a cloud service can produce unexpected variable cost when usage or output length rises. Account for operations and AI risk over the full lifecycle A local fleet expands patching, inventory, remote diagnostics, hardware replacement, model distribution, and version rollback. Cloud reduces some infrastructure work but adds quota management, service changes, provider deprecations, regional architecture, spend controls, and third party monitoring. Score the team that exists, not an imagined future team. NIST's AI Risk Management Framework organizes voluntary risk work around Govern, Map, Measure, and Manage. Use those functions across placement: assign accountable owners, map affected people and dependencies, measure model and system behavior in context, and manage failures and changes. Evaluate privacy, security, reliability, transparency, and harmful impacts at design, deployment, operation, and retirement. Placement cannot compensate for an unevaluated model. Require signed or otherwise verifiable model artifacts where supported, a software bill of materials where appropriate, controlled promotion between environments, rollback tests, and monitoring that does not collect unnecessary sensitive content. Keep evaluation sets and acceptance thresholds consistent across local and cloud candidates. Run a reversible placement pilot Select one representative workflow and implement the smallest complete version in the two leading placements. Use the same workload card, quality gate, demand trace, and security review. Exercise normal traffic, bursts, network loss, component failure, model update, rollback, incident investigation, and data deletion. Record operational labor as well as machine cost. Score each option from evidence across data boundaries, security controls, latency, offline resilience, scalability, quality, total cost, staff readiness, portability, and vendor dependency. Mark any nonnegotiable requirement as a gate rather than averaging it away. A placement that violates a residency rule or cannot meet a safety threshold should not win because it scores well elsewhere. Choose hybrid only when the division has a clear benefit and can be operated. Each additional boundary creates synchronization, observability, consistency, and failure mode work. Keep interfaces stable, minimize data crossing them, and test what happens when either side is unavailable. Document the decision, assumptions, evidence, exceptions, and review date. Re score when traffic, models, regulations, contracts, prices, user locations, or team capability change. The best placement is the one that satisfies the workload's current constraints with risks and costs the organization can actually manage. Sources The NIST Definition of Cloud Computing https://csrc.nist.gov/pubs/sp/800/145/final Zero Trust Architecture https://csrc.nist.gov/pubs/sp/800/207/final AI Risk Management Framework https://www.nist.gov/itl/ai risk management framework FinOps Terminology https://framework.…