Deploying AI agents in your own environment: what stays private, and what still leaves?

TL;DR: Hosting the agent runtime in your own environment does not by itself establish data residency. Residency is a property of the whole request path: the user entry point, the runtime, the model endpoint, the tools, identity, secrets and telemetry. Map every connection with its destination, region and retention, control outbound access at the application layer as well as the network layer, and prove the result from the running deployment rather than the diagram.

Key takeaways

• Hosting the agent runtime on premises or in your cloud does not prove that prompts, retrieved data, tool inputs, telemetry and credentials remain there.

• Draw the full request path across seven components: user entry point, agent runtime, model endpoint, tools, identity, secrets and telemetry.

• Control outbound access explicitly. A cluster network policy can restrict connections at the network layer, but model and tool destinations often need application-layer policy too.

• Treat identity and secrets as separate design problems. Single sign-on identifies the user; the agent still needs a workload identity and narrowly scoped credentials for each operation.

• Test the deployment with packet, audit and telemetry evidence. Architecture claims should be verifiable from the running environment.

Two terms matter before reviewing any deployment. Residency answers where data is processed or stored. Retention answers how long it remains there. Both need to be addressed for every data type in the workflow: prompts, retrieved context, tool inputs and outputs, model responses, telemetry and credentials.

Your team deploys an AI agent in a private Kubernetes cluster. The application, orchestration code and control plane all run in your environment. The architecture is described as on premises.

The agent still calls an external model endpoint. It sends traces to a hosted observability service. One tool connects to a software-as-a-service ticketing system. A document parser downloads a remote model when the pod starts. The secret manager uses a regional endpoint outside the stated boundary.

The runtime is private. The workflow is not contained.

Data residency for an agent depends on the paths taken by different data types: prompts, retrieved context, tool calls, model responses, telemetry and credentials. Each may follow a separate route with its own destination and retention. You need to map all of them before deciding what stays private.

This article presents a deployment review method rather than a description of any particular environment.

Start with the boundary you actually need

“On premises” can refer to physical infrastructure in a company-owned data center. “Private cloud” may refer to a tenant-controlled environment in a public cloud. “In region” may allow managed services as long as processing and storage remain in an approved geography.

Those are different requirements. Write the boundary in operational terms:

• which data classifications may enter the workflow;

• which countries or regions may process or store them;

• which organizations may operate the infrastructure;

• which systems may receive the data;

• which metadata may leave the boundary;

• how long prompts, traces and tool results may be retained;

• which support personnel can access the environment.

Then apply that definition to every component. A diagram labeled “private VPC” is not enough if an endpoint inside it calls a service outside the approved boundary.

NIST’s zero-trust architecture makes the adjacent security point that network location or asset ownership does not create implicit trust. For agent deployments, location still matters for residency, but it does not replace identity, authorization or resource-level policy.

Map seven components and every connection

A useful deployment review covers at least seven components.

1. User entry point

Where does a user or upstream system submit a task? Record the application, ingress path, authentication method and the first place the request is logged.

If a chat message reaches the agent through an external collaboration platform, the input has already crossed that platform’s boundary before your runtime sees it. The same applies to email, browser extensions and webhook services.

2. Agent runtime

This is the application that holds workflow state, calls models and invokes tools. It may run in a container cluster, virtual machine or managed runtime.

Record where session state, memory, caches and temporary files live. Check whether the runtime downloads packages, model artifacts or prompt templates at start-up. A workload that has no declared external dependency may still contact package registries, license services or content-delivery networks.

3. Model endpoint: where does a residency claim usually break?

One common place residency claims break is the fallback endpoint.

Your main endpoint is easy to check. Someone set the regional endpoint in the config. A reviewer can confirm it in a minute.

The fallback is harder. Consider a plausible scenario: whoever was on call the night the regional endpoint failed pointed it at the global service because that worked. Nobody revisits it, because the main endpoint keeps working. The first time the fallback runs is the first time data leaves the region. That is a hypothetical sequence, but it reflects a real failure pattern worth testing for in any deployment review.

Check the fallback explicitly. What does the gateway do when the regional endpoint returns an error? Which model does it switch to? Was that choice deliberate, or did it come with the default config?

Then fill in the rest: endpoint and region, the fields sent in each request, encryption in transit, the provider’s retention and training settings, and what the provider logs when a request fails.

4. Tools and enterprise systems

Tool calls can surface sensitive data that was not part of the original prompt: retrieved records, system responses and error payloads all become part of the data path. Map each tool endpoint, operation and returned fields.

Separate reads from writes. A read-only database query, a ticket update and a production restart have different identity and evidence requirements. Also identify tools that make their own downstream calls: a document-processing tool running inside the cluster may submit the document to an external service without that path appearing in the agent’s own configuration.

5. Identity provider

Single sign-on can establish who initiated the request. The agent runtime still needs an identity for service-to-service calls, and the target system needs enough context to authorize the requested operation.

Record whether the tool call uses the user’s delegated scope, an agent workload identity or a shared service account. For deployment review, the question is whether identity tokens and authorization calls cross the boundary and where their audit records are stored.

6. Secrets and credentials

Record where model keys, tool credentials, certificates and signing material are stored, issued and rotated. Check how they reach the workload and whether they appear in environment variables, logs or crash reports.

Kubernetes warns that Secret objects are stored unencrypted in the underlying data store by default unless encryption at rest is configured. Its guidance also recommends least-privilege access and restricting each secret to the containers that need it. A private cluster does not make a broadly readable secret safe.

7. Telemetry and evidence

Traces can contain prompts, retrieved context, tool parameters, model responses, identity references and error payloads. That makes observability part of the data architecture.

OpenTelemetry supports several collector deployment patterns, including agents and gateways that can process signals before exporting them. You can keep a collector inside the boundary, filter or redact fields there and send only approved measures to an external backend. The design still needs verification because exporter configuration determines the destination. Which signals are worth keeping, and which ones only add retention risk, is covered in observability for AI agents.

Can we run an AI control plane on premises with data staying in region?

Yes, if every required component supports that deployment and every data path follows the regional boundary, and is configured accordingly. Hosting the control plane inside the environment is one part of the answer. What an AI control plane is and how to evaluate one sets out the capabilities to check for, and where an AI control plane fits in enterprise AI architecture places it against the rest of the stack.

When evaluating whether an AI control plane supports a residency requirement, apply the same review method used for the rest of the deployment. Verify these paths separately:

• Where is the policy store located, and which network paths can reach it?

• Where are audit records written and retained, and who can access them?

• How is the management plane accessed, and does that access path cross the boundary?

• What happens during vendor support or diagnostic sessions: which systems are accessed, by whom, and what is logged?

Those questions apply to any control plane, including Moring’s AI Control Plane. The residency claim is only as strong as the answers to all of them, not just the runtime location.

Check these paths separately:

• control-plane configuration and policy data;

• agent runtime state;

• model requests and responses;

• tool requests and returned records;

• identity and token services;

• secrets and certificate services;

• logs, traces, metrics and audit evidence;

• software updates, support bundles and diagnostic uploads.

The phrase “data stays in region” also needs a data list. Prompt text may stay while request metadata leaves. A model response may not be retained while an error payload is. Audit evidence may stay in region while support access is global.

The architecture review should therefore produce a table with data class, source, destination, region, retention and owner for every connection. If any field is unknown, the residency claim is incomplete.

How do we deploy AI agents on Kubernetes or OpenShift?

The container platform is the substrate. You still need to place the application and control responsibilities on it.

A practical deployment separates:

• agent runtimes by environment and trust boundary;

• control-plane services and their backing stores;

• model or gateway connections;

• tool connectors with narrow network and identity access;

• telemetry collectors and exporters;

• secret delivery and certificate rotation;

• admission, image and workload policies.

Kubernetes NetworkPolicy can restrict pod ingress and egress at the IP and port layers when the network plugin enforces it. The documentation also notes limits: standard NetworkPolicy does not force traffic through a common gateway, make TLS decisions or target services by name. Agent deployments often need an application-layer gateway, service mesh or egress proxy in addition to cluster network policy.

The same design applies on another Kubernetes distribution, with the distribution’s identity, network and policy controls mapped explicitly. Avoid treating installation success as the deployment review. The agent can be healthy while its external paths remain too broad.

How do we control which endpoints an AI agent can reach?

Use several controls that answer different questions:

• Network policy: which addresses and ports can the workload connect to?

• DNS and egress policy: which named destinations are allowed, and how are changes reviewed?

• Gateway policy: which model or API routes can be used, with which request rules?

• Tool registry: which capabilities are exposed to the agent runtime?

• Action policy: which operation can this identity perform on this resource?

An allowlist of domains is not an action policy. If tickets.example.com is allowed, the agent may still need permission to read one queue but not delete another. Conversely, a correct application permission is ineffective if the workload can also reach an ungoverned administrative endpoint. Taking MCP to production goes further into tool design, identity and gateway placement for those paths.

Test the deny path. Attempt a connection to an undeclared endpoint from the running workload. Attempt an unapproved tool operation through an allowed endpoint. Confirm that both fail and that the evidence shows which control stopped them.

How do we manage secrets for AI agents?

Prefer short-lived credentials issued to an authenticated workload over static keys copied into configuration. Where a target supports delegated user access, issue a token for that target and operation rather than handing the agent the user’s broad session credential.

For every secret, answer:

• Which workload can request it?

• Which target accepts it?

• What operation and resource does it authorize?

• How long is it valid?

• How is it rotated and revoked?

• Can it appear in a prompt, trace or error?

• Which person can retrieve the raw value?

Kubernetes service-account tokens illustrate the direction: the documentation recommends short-lived, automatically rotating tokens obtained through the TokenRequest API instead of long-lived token Secrets where possible. The same principle applies to model and tool access even when the implementation differs.

How do we connect AI agents to existing SSO?

Use SSO to establish the initiating user and their session. Carry only the identity context needed by the workflow. Do not assume that every downstream tool should receive the user’s full credential.

The agent also needs a workload identity so target systems can distinguish the software actor from the person. For an on-behalf-of operation, retain both: the user on whose behalf the action is requested and the agent that executes it.

The deployment review should show where authentication occurs, where tokens are issued, which audiences accept them and which audit system retains the actor chain. If the identity provider is outside the residency boundary, include the attributes and logs it receives. Why the actor chain matters after the fact is the subject of who is accountable when an AI agent makes a bad decision.

How do we support multiple business units on one AI platform?

Separate configuration and data along the boundaries the organization actually enforces. That may include business unit, legal entity, geography, environment and data classification.

For each boundary, decide whether it requires:

• separate runtime namespaces or clusters;

• separate encryption keys and secret stores;

• separate model and tool allowlists;

• separate identities and approval roles;

• separate telemetry pipelines and retention;

• a shared control plane with scoped administration, or separate control-plane instances.

A shared dashboard does not require shared credentials or evidence storage. Conversely, separate clusters do not guarantee policy separation if one administrator can change every environment without review.

Boundary-review summary

The table below consolidates the review questions from each section. Use it as a deployment-review checklist before verifying the running deployment.

Verify the running deployment

Turn the architecture diagram into tests:

1. Submit a synthetic request containing a unique marker.

2. Trace where that marker appears in application logs, model requests, tool calls and telemetry.

3. Capture outbound connections from the runtime during normal and failure paths.

4. Disable the primary model endpoint and observe the fallback behavior.

5. Attempt an undeclared destination and an unauthorized operation.

6. Rotate or revoke a credential while a workflow is active.

7. Export the audit record and check whether it reconstructs the request without exposing unnecessary sensitive values.

Use synthetic data designed for the test. The point is to establish the path, not to send production records through an uncertain setup.

What you end up with is a living map: every component, every connection, and the explicit data class, identity, region, retention rule and owner for each. That gives security something concrete to approve and gives operations a baseline to watch for drift. The map is evidence; the diagram alone is not.

Here is the question to put to your own team this week. If the regional model endpoint failed tonight, where would the next request go, and who chose that? If nobody can answer it from memory, building the map is the work to do before your next agent ships.

Sources

• NIST SP 800-207: Zero Trust Architecture

• Kubernetes: Network policies

• Kubernetes: Secrets

• Kubernetes: Good practices for Secrets

• OpenTelemetry: Deploy the Collector

Frequently asked questions