Secure image transfer into Tier 0

Secure image transfer into Tier 0
PAW deep dive
Published: 2025-05-01
Updated: 2026-10-13
Canonical URL: https://learncloud.guru/paw/secure-image-transfer-tier-0

Why pipeline agent images belong to Tier 0, how application images cross from Tier 2 without opening a path back, and the controls that keep the registry clean.

Why an agent image is Tier 0

Picture a Tier 0 deployment pipeline. A job starts, the platform hands it a service connection token, and the job runs az, terraform or kubectl against the identity control plane. Every one of those steps executes inside a container image or on an agent image. Whoever controls that image controls what the job does with the token.

Microsoft calls this the clean source principle: every security dependency must be as trustworthy as the object it secures. An image that runs Tier 0 jobs is such a dependency. That settles the most common question in one line: agent and tooling images are built inside Tier 0 from pinned, verified inputs, and never promoted from Tier 2. No promotion gate can make a Tier 2 build system trustworthy after the fact.

Application images are different. They run on Tier 0 infrastructure but do not hold pipeline tokens, so Tier 2 can build them as long as Tier 0 decides what gets in. The rest of this article follows those two routes and the controls on each.

Four eyes and a PAW do different jobs

Teams with a strict two-person rule sometimes ask whether they still need PAWs. They do, because the two controls answer different questions. Four eyes asks: does a second person agree with this change? The PAW asks: is this session really that person, on a clean device?

Four eyes only guards the pipeline path. A Tier 0 administrator’s activated roles also work directly in the portal, the CLI and Graph, where no pipeline approval applies. And if an approver’s session is stolen from a daily-driver laptop, the attacker becomes the second pair of eyes.

Situation | Four eyes in Azure DevOps | PAW | Controls
One engineer merges a malicious change alone | Stops it | Does not help | C2 C3
An approver’s browser session is stolen on a daily-driver laptop | The attacker approves as that person | Tier 0 accounts cannot sign in outside a PAW | C1
An admin uses an activated role directly in the portal to change the registry | No pipeline involved | Protects the session, not the intent; add PIM approval and alerts | C12 C13
Malware on the engineer’s device edits the PR after approval | Only if votes reset on new pushes and the last pusher cannot approve | Clean device | C1 C2
A pipeline job abuses an agent VM’s managed identity | Runs after approval | Not a human session | C3 C4 C7
An Azure DevOps admin turns the checks off | Admins edit the checks | Only if admins are Tier 0 accounts on PAWs | C12 C13

There is a corollary for the PAW itself: do not build images or install packages on it. Package installs run third-party code at install time. The Shai-Hulud npm worm ran from postinstall and preinstall scripts and harvested cloud credentials, including the Azure CLI profile. On a PAW, that code would run next to activated Tier 0 tokens. The PAW authors code and approves runs; builds move to fresh, network-isolated Tier 0 agents, and everyday development stays on the normal workstation with a non-privileged account.

Decide the route first

Before choosing tools, classify the image. The test is simple: is it referenced by a container step on a Tier 0 job, is it the image of a Tier 0 agent pool, or do Tier 0 pipeline steps call tools inside it? Then it runs Tier 0 jobs. If it only runs on the Tier 0 cluster as an application, it is a workload image, unless that application can obtain Tier 0 credentials.

Does this image run Tier 0 pipeline jobs?
- Yes, it runs Tier 0 jobs: Treat it as Tier 0 control plane. Build it inside Tier 0 from pinned, verified inputs and never promote it from Tier 2.
  - Recommended: Managed DevOps Pools. Stateless pools give every job a fresh VM in your Tier 0 virtual network. No agent image to build, sign or patch, no registration secret, no Docker-in-Docker.
  - Only with four fixes: Containerized agents. The Docker sample downloads the agent at start, keeps its token on disk and mounts the node’s Docker socket. Fix all four first.
- No, it’s an application image: Tier 2 builds it, a Tier 0 gate decides. Tier 0 pulls; Tier 2 never pushes. The promote job verifies, scans, copies by digest and re-signs. Tier 0 AKS admits only Tier 0-signed digests.
  - Route: Promotion gate. A staging registry with a second private endpoint in Tier 0, a pull-only promote identity, and no az acr import.
  - Check first: What runs on the Tier 0 cluster?. If workloads there can obtain Tier 0 credentials, even application images are Tier 0 control plane. Rebuild them from reviewed source instead.
- Not sure: How to tell. When in doubt, treat it as Tier 0: rebuilding inside Tier 0 is always safe; promoting is safe only for workloads.
  - Runs Tier 0 jobs: Pipeline or pool image. Referenced by a container step on a Tier 0 job, used as a Tier 0 agent pool image, or Tier 0 steps call az, terraform, kubectl or helm inside it.
  - Workload: Application only. Runs on the Tier 0 cluster only as an application and cannot obtain Tier 0 credentials.

When in doubt, treat it as Tier 0. Rebuilding inside Tier 0 is always safe; promoting is safe only for workloads.

The target state, step by step

Here is the whole path from a pull request on the PAW to a signed image running on Tier 0 AKS. Press Play flow to walk the nine steps, select any box to see what it does, or select a dark tag (C1 to C14) to see the control behind it and how to verify it.

Diagram: Target state: engineers author and approve on a PAW; Azure DevOps checks gate Tier 0 pools and a workload identity federation connection; stateless Managed DevOps Pools acquire, build, publish and promote images into a private Tier 0 registry; Tier 0 AKS admits only Tier 0-signed digests; Tier 2 pushes only to a staging registry.

Flow:
1. The engineer opens a PR from the PAW.
2. A different person reviews and approves the PR.
3. Merging to main runs the pipeline, which must extend the required template.
4. Pool and service connection checks run: branch control, required template, approval.
5. The second person approves the run. Self-approval is off.
6. Jobs run only on Tier 0 Managed DevOps Pools, each on a fresh VM.
7. Acquire verifies pinned inputs into the catalog. Build holds no write credentials. Publish pushes by digest and signs with the Key Vault key.
8. Tier 0 AKS pulls over the private endpoint and admits only Tier 0-signed digests.
9. Tier 2 application images reach Tier 0 only through the promote job, via the staging registry’s second private endpoint.

Three ideas carry most of the weight. First, jobs run only on stateless Managed DevOps Pools: each job gets a fresh VM that is discarded afterwards, and no managed identity is attached to any pool, so a job cannot pick up credentials from the VM. Second, internet egress and write credentials never share a job: acquire can reach pinned vendor endpoints but only writes the catalog, build has neither internet nor write access, and publish holds the push token but no internet. Third, the only identity that can push is reached through workload identity federation, so there is no secret to steal and only runs that pass the checks can use it.

Promoting application images from Tier 2

Only application images take this route, and the direction matters: Tier 0 pulls, Tier 2 never pushes.

1. Tier 2 builds the application image, signs it with the Tier 2 Notation key, attaches the SBOM and provenance, and pushes it to a staging registry with public access off.
2. The staging registry gets a second private endpoint in the Tier 0 registry endpoint subnet. Tier 2 and Tier 0 are never peered. The Tier 0 promote identity has pull-only rights on staging.
3. The promote job verifies the Tier 2 signature and the provenance (repository, branch, pipeline) against a Notation trust policy.
4. It scans the image against an internally mirrored vulnerability database.
5. It copies the image by digest into the Tier 0 registry, including referrers such as the signature and SBOM.
6. It signs the same digest with the Tier 0 key held in Key Vault.
7. Tier 0 AKS admits only digests signed by the Tier 0 key.

Why not az acr import? Import needs the Container Registry Data Importer and Data Reader role on both registries, and every network-restricted registry involved must allow trusted services to bypass its network rules. That reopens exactly the paths the design closes. With export disabled on the Tier 0 registry, import out of it is blocked, which is intended.

One DNS detail is easy to miss. If privatelink.azurecr.io is a shared, central zone, the staging registry’s second endpoint registers a record that collides with its Tier 2 record. Give Tier 0 its own zone and resolver path.

A registry no human can push to

Only two managed identities hold data-plane roles: AcrPush for the orchestration identity through workload identity federation, and AcrPull for the AKS kubelet identity. No human has a role assignment and the admin user is off. On the network side, public access is off and the registry is reachable only through its private endpoint.

Setting | Value | Why
SKU | Premium | Required for private endpoints, export policy and network rules.
Public network access | Disabled | Clients outside the virtual network cannot reach the registry endpoints.
Admin user | Disabled | No shared username and password; every push and pull is an Entra ID identity.
Export policy | Disabled | Blocks import and transfer pipelines out of the registry. Requires public access to be disabled.
Trusted-services bypass | None | Azure services use the private endpoint like everyone else.
Network rule set | Default deny, no IP rules | Defence in depth if someone re-enables public access.
Anonymous pull | Disabled | No unauthenticated reads.
DNS | Private DNS zone group | Writes both records the registry needs: the login server and the regional data endpoint.
Endpoint subnet NSG | 443 from runtime and build subnets only | Needs private endpoint network policies enabled on the subnet.
Data-plane roles | AcrPush (orchestration, via WIF), AcrPull (AKS) | ABAC-enabled registries ignore AcrPull and AcrPush; do not switch modes without replacing both.
Integrity | Deploy by digest, admit by signature | Tag locks only stop accidents. Docker Content Trust is being removed from ACR; use Notation.
Delete lock | CanNotDelete | Blocks control-plane deletion; image operations are unaffected.

Governance closes the last gap. Anyone who can grant roles at the registry’s scope, such as Owner or User Access Administrator, can grant themselves AcrPush. Those roles are therefore Tier 0 too: held through PIM, used from PAWs, and every role assignment change on the registry raises an alert. Built-in Azure Policy definitions can deny the common regressions at the Tier 0 management group, such as re-enabling export, the local admin account or anonymous pull.

Microsoft’s documentation differs on whether Defender for Cloud needs the trusted-services bypass to scan a private registry. Keep the bypass off and confirm that scan results still appear before relaxing anything.

If agents stay in containers

Managed DevOps Pools remove the agent image, the registration secret and Docker-in-Docker in one step, which is why they are the recommended route. Some teams still run Azure DevOps agents as pods on AKS. Microsoft’s own Docker sample is a reasonable starting point for a lab, but it downloads the agent at start, keeps its registration token on disk and, on AKS, mounts the node’s Docker socket. All four of the fixes below are needed before it runs Tier 0 jobs.

Diagram: containerized agent hardening. A KEDA ScaledJob creates one pod per job; an init container registers the agent, the agent container runs one job with no credential, a cleanup controller deregisters it, and image builds run on a Managed DevOps Pool.

A1 [C/L] Bake a pinned agent into the image
Microsoft’s sample start.sh downloads the latest agent package every time the container starts, so the signed image is not the code that runs. Download and SHA256-verify a pinned agent version at image build. Automatic agent updates are on by default, so turn them off for the pool and rebuild the image when a task needs a newer agent.
Verify: The image contains the agent binaries. start.sh makes no download call. Pool settings show automatic updates off.

A2 [C/L] Keep the registration credential out of the job container
The sample writes the token to a file and keeps it for cleanup at shutdown, so any job on that agent can read it and register a rogue agent. Register in an init container and deregister from a separate controller. With workload identity, list the agent container in azure.workload.identity/skip-containers so only the init container gets the token.
Verify: kubectl exec into the agent container finds no token file or token environment variable.

A3 [C] No Docker socket, no privileged pods
Microsoft’s AKS example mounts the node’s Docker socket, and Microsoft warns that this lets code in the container run as root on the host. Docker-based tasks also do not work on AKS 1.19 or later. Build images on the Managed DevOps Pool instead.
Verify: Pod Security Admission runs in restricted mode on the agent namespace. No pod has hostPath volumes.

A4 [C/L] One job per pod, short-lived credentials
The sample defaults to a PAT. A service principal’s client secret is used only during registration (agent 3.227.1 and later). Run with --once so each pod serves one job, and deregister as soon as the pod exits.
Verify: A KEDA ScaledJob creates one pod per queued job. Agent registrations disappear within minutes of pod exit.

For daemonless builds, run BuildKit on the Managed DevOps Pool VM rather than inside agent pods. Kaniko’s original repository was archived in June 2025, so do not plan new work around it.

The control set

Fourteen controls make up the target state, plus the four agent fixes above if containerized agents stay. Each is tagged C for a documented Microsoft capability or L for a design recommendation inferred from documented behaviour, and each carries a check you can run to prove it is in place.

C1 [L] Tier 0 accounts sign in only from PAWs
Conditional Access limits Tier 0 accounts to PAW sessions. The PAW authors code and approves runs; it never builds images or installs packages.
Verify: A Conditional Access policy targets the Tier 0 accounts and requires the PAW device filter or a compliant PAW. Sign-in logs show no Tier 0 sign-ins from other devices.

C2 [C] Branch policy on main enforces a second reviewer
Require a minimum number of reviewers, turn off “Allow requestors to approve their own changes”, turn on “Prohibit the most recent pusher from approving their own changes”, and reset all approval votes when new changes are pushed.
Verify: The main branch policies show those four settings. A test PR cannot be completed by its author.

C3 [C] Checks on Tier 0 pools and the service connection
Branch control allows only refs/heads/main, a required template check forces the approved pipeline shape, and an approval check stops approvers from approving their own runs. Pipeline permissions are granted per pipeline, not to all pipelines.
Verify: Approvals and checks on each Tier 0 agent pool, the WIF service connection and the Tier 0 environment list all three checks. A run from a feature branch is blocked.

C4 [C/L] Managed DevOps Pools: stateless, VNet-injected, no pool identity
“Fresh agent every time” gives each job a new VM that is discarded afterwards. Each pool has its own delegated subnet. No managed identity is attached to any pool, so a job cannot pick up credentials through IMDS.
Verify: Pool agent state is Stateless. Pool identity is None. Each pool’s subnet is delegated to Microsoft.DevOpsInfrastructure/pools and used by no other pool.

C5 [L] Internet egress and write credentials never share a job
Acquire has internet egress and may write only the catalog. Build has no internet and no write credentials. Publish has no internet and holds the WIF token for the Tier 0 registry. Each job runs on its own fresh VM.
Verify: The required template defines the three jobs and their pools. Only the publish and promote jobs reference the WIF service connection.

C6 [L] Every input pinned in reviewed source and verified
Base images by digest, downloads by SHA256 (plus GPG or vendor signature where published), the agent version and its SHA256. Inputs are verified before they enter the catalog, and builds use catalog inputs only.
Verify: The repo holds a lock file of digests and hashes. A changed hash fails the acquire job. Build logs show only catalog registry pulls.

C7 [C/L] Push only through workload identity federation
The orchestration identity holds AcrPush on the Tier 0 registry only and is reachable only through a WIF service connection, with no secret. Only the publish and promote jobs use that connection.
Verify: The identity has exactly one federated credential (the service connection) and one role assignment (AcrPush on the registry). It is not attached to any VM.

C8 [C/L] Tier 0 registry locked to its private endpoint
Premium SKU, public network access off, admin user off, export policy disabled, private endpoint only, DNS through a zone group, an NSG allowing 443 only from the runtime and build subnets. Trusted-services bypass off.
Verify: az acr show reports publicNetworkAccess Disabled, adminUserEnabled false, exportPolicy disabled and networkRuleBypassOptions None. Both A records exist in privatelink.azurecr.io.

C9 [C/L] Sign with a Tier 0 key; deploy by digest
The publish job signs the pushed digest with a Tier 0 key held in Key Vault (Notation Key Vault plugin) and attaches the SBOM. Deployments reference image@sha256 digests, never tags.
Verify: notation ls lists a Tier 0 signature for each deployed digest. Kubernetes manifests contain digests.

C10 [C/L] Admission allows only Tier 0-signed digests
Self-managed Ratify with Gatekeeper on Tier 0 AKS verifies the Tier 0 signature before admission. The AKS Image Integrity add-on is still in preview.
Verify: Deploying an unsigned or Tier 2-only-signed image to the Tier 0 cluster is denied.

C11 [C/L] Promotion gate for Tier 2 application images
Verify the Tier 2 signature and provenance, scan, copy by digest including referrers, re-sign with the Tier 0 key. Do not use az acr import: it needs the Importer role on both registries and the trusted-services bypass on every network-restricted registry involved.
Verify: Promote job logs show verify, scan, copy-by-digest and sign steps. No Tier 0 identity holds Container Registry Data Importer and Data Reader.

C12 [L] Azure DevOps admins are Tier 0 accounts
Organization, project, agent pool and service connection administrators can edit every check above, so they are dedicated Tier 0 accounts restricted to PAWs.
Verify: Project Collection Administrators and the relevant admin groups contain only dedicated Tier 0 accounts. Membership changes raise alerts.

C13 [C] Registry and pipeline audit logs reach Sentinel
ContainerRegistryRepositoryEvents and ContainerRegistryLoginEvents go to Log Analytics, alongside the Azure DevOps audit stream, firewall logs and pool logs.
Verify: Both tables return rows in the workspace. Analytics rules for unexpected pushes, deletes and role assignment changes are enabled.

C14 [C/L] Per-subnet egress allowlists
Managed DevOps Pools documents its required egress, including *.queue.core.windows.net and Ubuntu package endpoints. Allow only the documented Managed DevOps Pools and Azure DevOps endpoints, plus pinned vendor FQDNs for acquire. Never allow github.com or *.blob.core.windows.net broadly.
Verify: Firewall rule collections exist per subnet. Denied-traffic logs from the build subnet stay at zero during normal runs.

Sources and further reading (checked 2026-10-13)
Privileged access: clean source principle: https://learn.microsoft.com/en-us/security/privileged-access-workstations/privileged-access-success-criteria#clean-source-principle
Secure Azure Pipelines: https://learn.microsoft.com/en-us/azure/devops/pipelines/security/overview
Azure Pipelines: agent and other security considerations: https://learn.microsoft.com/en-us/azure/devops/pipelines/security/misc
Protected resources in Azure Pipelines: https://learn.microsoft.com/en-us/azure/devops/pipelines/security/resources
Define approvals and checks: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals
Use templates for security: https://learn.microsoft.com/en-us/azure/devops/pipelines/security/templates
Set and manage branch policies: https://learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies
Connect to Azure with workload identity federation: https://learn.microsoft.com/en-us/azure/devops/pipelines/library/connect-to-azure
Managed DevOps Pools overview: https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/overview
Managed DevOps Pools: configure scaling (stateless agents): https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/configure-scaling
Managed DevOps Pools: configure networking: https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/configure-networking
Managed DevOps Pools: configure images: https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/configure-images
Run a self-hosted agent in Docker: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker
Azure Pipelines agents: agent version and upgrades: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents#agent-version-and-upgrades
Register an agent using a service principal: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/service-principal-agent-registration
Microsoft Entra Workload ID with AKS: https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
Containers Secure Supply Chain framework: https://learn.microsoft.com/en-us/azure/security/container-secure-supply-chain
CSSC framework: Build stage: https://learn.microsoft.com/en-us/azure/security/container-secure-supply-chain/articles/container-secure-supply-chain-implementation/build-overview
Sign container images with Notation and Azure Key Vault: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tutorial-sign-build-push
Verify container image signatures with Ratify and Azure Policy: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-tutorial-verify-with-ratify-aks
AKS Image Integrity (preview): https://learn.microsoft.com/azure/aks/image-integrity
Docker Content Trust deprecation in ACR: https://learn.microsoft.com/azure/container-registry/container-registry-content-trust-deprecation
Connect privately to a container registry with Private Link: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-private-link
Disable export of artifacts from a container registry: https://learn.microsoft.com/azure/container-registry/data-loss-prevention
Import container images to a container registry: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-import-images
ACR repository permissions with Azure ABAC: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-rbac-abac-repository-permissions
Azure Policy built-in definitions for Container Registry: https://learn.microsoft.com/azure/container-registry/policy-reference
Defender for Containers: network access and permissions: https://learn.microsoft.com/en-us/azure/defender-for-cloud/defender-for-containers-network-access
Monitor Azure Container Registry: https://learn.microsoft.com/en-us/azure/container-registry/monitor-container-registry
MITRE ATT&CK S9008: Shai-Hulud: https://attack.mitre.org/software/S9008/
GoogleContainerTools/kaniko (archived June 2025): https://github.com/GoogleContainerTools/kaniko