Kubernetes Blog

Every container image you pull from registry.k8s.io got there through kpromo, the Kubernetes image promoter. It copies images from staging registries to production, signs them with cosign, replicates signatures across more than 20 regional mirrors, and generates SLSA provenance attestations. If this tool breaks, no Kubernetes release ships. Over the past few weeks, we rewrote its core from scratch, deleted 20% of the codebase, made it dramatically faster, and nobody noticed. That was the whole point.

A bit of history

The image promoter started in late 2018 as an internal Google project by Linus Arver. The goal was simple: replace the manual, Googler-gated process of copying container images into k8s.gcr.io with a community-owned, GitOps-based workflow. Push to a staging registry, open a PR with a YAML manifest, get it reviewed and merged, and automation handles the rest. KEP-1734 formalized this proposal.

In early 2019, the code moved to kubernetes-sigs/k8s-container-image-promoter and grew quickly. Over the next few years, Stephen Augustus consolidated multiple tools (cip, gh2gcs, krel promote-images, promobot-files) into a single CLI called kpromo. The repository was renamed to promo-tools. Adolfo Garcia Veytia (Puerco) added cosign signing and SBOM support. Tyler Ferrara built vulnerability scanning. Carlos Panato kept the project in a healthy and releasable state. 42 contributors made about 3,500 commits across more than 60 releases.

It worked. But by 2025 the codebase carried the weight of seven years of incremental additions from multiple SIGs and subprojects. The README said it plainly: you will see duplicated code, multiple techniques for accomplishing the same thing, and several TODOs.

The problems we needed to solve

Production promotion jobs for Kubernetes core images regularly took over 30 minutes and frequently failed with rate limit errors. The core promotion logic had grown into a monolith that was hard to extend and difficult to test, making new features like provenance or vulnerability scanning painful to add.

On the SIG Release roadmap, two work items had been sitting for a while: "Rewrite artifact promoter" and "Make artifact validation more robust". We had discussed these at SIG Release meetings and KubeCons, and the open research spikes on project board #171 captured eight questions that needed answers before we could move forward.

One issue to answer them all

In February 2026, we opened issue #1701 ("Rewrite artifact promoter pipeline") and answered all eight spikes in a single tracking issue. The rewrite was deliberately phased so that each step could be reviewed, merged, and validated independently. Here is what we did:

Phase 1: Rate Limiting (#1702). Rewrote rate limiting to properly throttle all registry operations with adaptive backoff.

Phase 2: Interfaces (#1704). Put registry and auth operations behind clean interfaces so they can be swapped out and tested independently.

Phase 3: Pipeline Engine (#1705). Built a pipeline engine that runs promotion as a sequence of distinct phases instead of one large function.

Phase 4: Provenance (#1706). Added SLSA provenance verification for staging images.

Phase 5: Scanner and SBOMs (#1709). Added vulnerability scanning and SBOM support. Flipped the default to the new pipeline engine. At this point we cut v4.2.0 and let it soak in production before continuing.

Phase 6: Split Signing from Replication (#1713). Separated image signing from signature replication into their own pipeline phases, eliminating the rate limit contention that caused most production failures.

Phase 7: Remove Legacy Pipeline (#1712). Deleted the old code path entirely.

Phase 8: Remove Legacy Dependencies (#1716). Deleted the audit subsystem, deprecated tools, and e2e test infrastructure.

Phase 9: Delete the Monolith (#1718). Removed the old monolithic core and its supporting packages. Thousands of lines deleted across phases 7 through 9.

Each phase shipped independently. v4.3.0 followed the next day with the legacy code fully removed.

With the new architecture in place, a series of follow-up improvements landed: parallelized registry reads (#1736), retry logic for all network operations (#1742), per-request timeouts to prevent pipeline hangs (#1763), HTTP connection reuse (#1759), local registry integration tests (#1746), the removal of deprecated credential file support (#1758), a rework of attestation handling to use cosign's OCI APIs and the removal of deprecated SBOM support (#1764), and a dedicated promotion record predicate type registered with the in-toto attestation framework (#1767). These would have been much harder to land without the clean separation the rewrite provided. v4.4.0 shipped all of these improvements and enabled provenance generation and verification by default.

The new pipeline

The promotion pipeline now has seven clearly separated phases:

graph LR Setup --> Plan --> Provenance --> Validate --> Promote --> Sign --> Attest Phase What it does Setup Validate options, prewarm TUF cache. Plan Parse manifests, read registries, compute which images need promotion. Provenance Verify SLSA attestations on staging images. Validate Check cosign signatures, exit here for dry runs. Promote Copy images server-side, preserving digests. Sign Sign promoted images with keyless cosign. Attest Generate promotion provenance attestations using a dedicated in-toto predicate type.

Phases run sequentially, so each one gets exclusive access to the full rate limit budget. No more contention. Signature replication to mirror registries is no longer part of this pipeline and runs as a dedicated periodic Prow job instead.

Making it fast

With the architecture in place, we turned to performance.

Parallel registry reads (#1736): The plan phase reads 1,350 registries. We parallelized this and the plan phase dropped from about 20 minutes to about 2 minutes.

Two-phase tag listing (#1761): Instead of checking all 46,000 image groups across more than 20 mirrors, we first check only the source repositories. About 57% of images have no signatures at all because they were promoted before signing was enabled. We skip those entirely, cutting API calls roughly in half.

Source check before replication (#1727): Before iterating all mirrors for a given image, we check if the signature exists on the primary registry first. In steady state where most signatures are already replicated, this reduced the work from about 17 hours to about 15 minutes.

Per-request timeouts (#1763): We observed intermittent hangs where a stalled connection blocked the pipeline for over 9 hours. Every network operation now has its own timeout and transient failures are retried automatically.

Connection reuse (#1759): We started reusing HTTP connections and auth state across operations, eliminating redundant token negotiations. This closed a long-standing request from 2023.

By the numbers

Here is what the rewrite looks like in aggregate.

• Over 40 PRs merged, 3 releases shipped (v4.2.0, v4.3.0, v4.4.0)
• Over 10,000 lines added and over 16,000 lines deleted, a net reduction of about 5,000 lines (20% smaller codebase)
• Performance drastically improved across the board
• Robustness improved with retry logic, per-request timeouts, and adaptive rate limiting
• 19 long-standing issues closed

The codebase shrank by a fifth while gaining provenance attestations, a pipeline engine, vulnerability scanning integration, parallelized operations, retry logic, integration tests against local registries, and a standalone signature replication mode.

No user-facing changes

This was a hard requirement. The kpromo cip command accepts the same flags and reads the same YAML manifests. The post-k8sio-image-promo Prow job continued working throughout. The promotion manifests in kubernetes/k8s.io did not change. Nobody had to update their workflows or configuration.

We caught two regressions early in production. One (#1731) caused a registry key mismatch that made every image appear as "lost" so that nothing was promoted. Another (#1733) set the default thread count to zero, blocking all goroutines. Both were fixed within hours. The phased release strategy (v4.2.0 with the new engine, v4.3.0 with legacy code removed) gave us a clear rollback path that we fortunately never needed.

What comes next

Signature replication across all mirror registries remains the most expensive part of the promotion cycle. Issue #1762 proposes eliminating it entirely by having archeio (the registry.k8s.io redirect service) route signature tag requests to a single canonical upstream instead of per-region backends. Another option would be to move signing closer to the registry infrastructure itself. Both approaches need further discussion with the SIG Release and infrastructure teams, but either one would remove thousands of API calls per promotion cycle and simplify the codebase even further.

Thank you

This project has been a community effort spanning seven years. Thank you to Linus, Stephen, Adolfo, Carlos, Ben, Marko, Lauri, Tyler, Arnaud, and many others who contributed code, reviews, and planning over the years. The SIG Release and Release Engineering communities provided the context, the discussions, and the patience for a rewrite of infrastructure that every Kubernetes release depends on.

If you want to get involved, join us in #release-management on the Kubernetes Slack or check out the repository.

The community around Kubernetes includes a number of Special Interest Groups (SIGs) and Working Groups (WGs) facilitating discussions on important topics between interested contributors. Today, we're excited to announce the formation of the AI Gateway Working Group, a new initiative focused on developing standards and best practices for networking infrastructure that supports AI workloads in Kubernetes environments.

What is an AI Gateway?

In a Kubernetes context, an AI Gateway refers to network gateway infrastructure (including proxy servers, load-balancers, etc.) that generally implements the Gateway API specification with enhanced capabilities for AI workloads. Rather than defining a distinct product category, AI Gateways describe infrastructure designed to enforce policy on AI traffic, including:

• Token-based rate limiting for AI APIs.
• Fine-grained access controls for inference APIs.
• Payload inspection enabling intelligent routing, caching, and guardrails.
• Support for AI-specific protocols and routing patterns.

Working group charter and mission

The AI Gateway Working Group operates under a clear charter with the mission to develop proposals for Kubernetes Special Interest Groups (SIGs) and their sub-projects. Its primary goals include:

• Standards Development: Create declarative APIs, standards, and guidance for AI workload networking in Kubernetes.
• Community Collaboration: Foster discussions and build consensus around best practices for AI infrastructure.
• Extensible Architecture: Ensure composability, pluggability, and ordered processing for AI-specific gateway extensions.
• Standards-Based Approach: Build on established networking foundations, layering AI-specific capabilities on top of proven standards.

Active proposals

WG AI Gateway currently has several active proposals that address key challenges in AI workload networking:

Payload Processing

The payload processing proposal addresses the critical need for AI workloads to inspect and transform full HTTP request and response payloads. This enables:

AI Inference Security

• Guard against malicious prompts and prompt injection attacks.
• Content filtering for AI responses.
• Signature-based detection and anomaly detection for AI traffic.

AI Inference Optimization

• Semantic routing based on request content.
• Intelligent caching to reduce inference costs and improve response times.
• RAG (Retrieval-Augmented Generation) system integration for context enhancement.

The proposal defines standards for declarative payload processor configuration, ordered processing pipelines, and configurable failure modes - all essential for production AI workload deployments.

Egress gateways

Modern AI applications increasingly depend on external inference services, whether for specialized models, failover scenarios, or cost optimization. The egress gateways proposal aims to define standards for securely routing traffic outside the cluster. Key features include:

External AI Service Integration

• Secure access to cloud-based AI services (OpenAI, Vertex AI, Bedrock, etc.).
• Managed authentication and token injection for third-party AI APIs.
• Regional compliance and failover capabilities.

Advanced Traffic Management

• Backend resource definitions for external FQDNs and services.
• TLS policy management and certificate authority control.
• Cross-cluster routing for centralized AI infrastructure.

User Stories We're Addressing

• Platform operators providing managed access to external AI services.
• Developers requiring inference failover across multiple cloud providers.
• Compliance engineers enforcing regional restrictions on AI traffic.
• Organizations centralizing AI workloads on dedicated clusters.

Upcoming events

KubeCon + CloudNativeCon Europe 2026, Amsterdam

AI Gateway working group members will be presenting at KubeCon + CloudNativeCon Europe in Amsterdam, discussing the problems at the intersection of AI and networking, including the working group's active proposals, as well as the intersection of AI gateways with Model Context Protocol (MCP) and agent networking patterns.
This session will showcase how AI Gateway working group proposals enable the infrastructure needed for next-generation AI deployments and communication patterns.
The session will also include the initial designs, early prototypes, and emerging directions shaping the WG’s roadmap.
For more details see our session here:

• AI'm at the Gate! Introducing the AI Gateway Working Group in Kubernetes

Get involved

The AI Gateway Working Group represents the Kubernetes community's commitment to standardizing AI workload networking. As AI becomes increasingly integral to modern applications, we need robust, standardized infrastructure that can support the unique requirements of inference workloads while maintaining the security, observability, and reliability standards that Kubernetes users expect.
Our proposals are currently in active development, with implementations beginning across various gateway projects. We're working closely with SIG Network on Gateway API enhancements and collaborating with the broader cloud-native community to ensure our standards meet real-world production needs.

Whether you're a gateway implementer, platform operator, AI application developer, or simply interested in the intersection of Kubernetes and AI, we'd love your input. The working group follows an open contribution model - you can review our proposals, join our weekly meetings, or start discussions on our GitHub repository. To learn more:

• Visit the working group's umbrella GitHub repository.
• Read the working group's charter.
• Join the weekly meeting on Thursdays at 2PM EST.
• Connect with the working group on Slack (#wg-ai-gateway) (visit https://slack.k8s.io/ for an invitation).
• Join the AI Gateway mailing list.

The future of AI infrastructure in Kubernetes is being built today, join up and learn how you can contribute and help shape the future of AI-aware gateway capabilities in Kubernetes.

As announced November 2025, Kubernetes will retire Ingress-NGINX in March 2026. Despite its widespread usage, Ingress-NGINX is full of surprising defaults and side effects that are probably present in your cluster today. This blog highlights these behaviors so that you can migrate away safely and make a conscious decision about which behaviors to keep. This post also compares Ingress-NGINX with Gateway API and shows you how to preserve Ingress-NGINX behavior in Gateway API. The recurring risk pattern in every section is the same: a seemingly correct translation can still cause outages if it does not consider Ingress-NGINX's quirks.

I'm going to assume that you, the reader, have some familiarity with Ingress-NGINX and the Ingress API. Most examples use httpbin as the backend.

Also, note that Ingress-NGINX and NGINX Ingress are two separate Ingress controllers. Ingress-NGINX is an Ingress controller maintained and governed by the Kubernetes community that is retiring March 2026. NGINX Ingress is an Ingress controller by F5. Both use NGINX as the dataplane, but are otherwise unrelated. From now on, this blog post only discusses Ingress-NGINX.

1. Regex matches are prefix-based and case insensitive

Suppose that you wanted to route all requests with a path consisting of only three uppercase letters to the httpbin service. You might create the following Ingress with the nginx.ingress.kubernetes.io/use-regex: "true" annotation and the regex pattern of /[A-Z]{3}.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: regex-match-ingress
 annotations:
 nginx.ingress.kubernetes.io/use-regex: "true"
spec:
 ingressClassName: nginx
 rules:
 - host: regex-match.example.com
 http:
 paths:
 - path: "/[A-Z]{3}"
 pathType: ImplementationSpecific
 backend:
 service:
 name: httpbin
 port:
 number: 8000

However, because regex matches are prefix and case insensitive, Ingress-NGINX routes any request with a path that starts with any three letters to httpbin:

curl -sS -H "Host: regex-match.example.com" http://<your-ingress-ip>/uuid

The output is similar to:

{
 "uuid": "e55ef929-25a0-49e9-9175-1b6e87f40af7"
}

Note: The /uuid endpoint of httpbin returns a random UUID. A UUID in the response body means that the request was successfully routed to httpbin.

With Gateway API, you can use an HTTP path match with a type of RegularExpression for regular expression path matching. RegularExpression matches are implementation specific, so check with your Gateway API implementation to verify the semantics of RegularExpression matching. Popular Envoy-based Gateway API implementations such as Istio1, Envoy Gateway, and Kgateway do a full case-sensitive match.

Thus, if you are unaware that Ingress-NGINX patterns are prefix and case-insensitive, and, unbeknownst to you, clients or applications send traffic to /uuid (or /uuid/some/other/path), you might create the following HTTP route.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: regex-match-route
spec:
 hostnames:
 - regex-match.example.com
 parentRefs:
 - name: <your gateway>  # Change this depending on your use case
 rules:
 - matches:
 - path:
 type: RegularExpression
 value: "/[A-Z]{3}"
 backendRefs:
 - name: httpbin
 port: 8000

However, if your Gateway API implementation does full case-sensitive matches, the above HTTP route would not match a request with a path of /uuid. The above HTTP route would thus cause an outage because requests that Ingress-NGINX routed to httpbin would fail with a 404 Not Found at the gateway.

To preserve the case-insensitive regex matching, you can use the following HTTP route.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: regex-match-route
spec:
 hostnames:
 - regex-match.example.com
 parentRefs:
 - name: <your gateway>  # Change this depending on your use case
 rules:
 - matches:
 - path:
 type: RegularExpression
 value: "/[a-zA-Z]{3}.*"
 backendRefs:
 - name: httpbin
 port: 8000

Alternatively, the aforementioned proxies support the (?i) flag to indicate case insensitive matches. Using the flag, the pattern could be (?i)/[a-z]{3}.*.

2. The nginx.ingress.kubernetes.io/use-regex applies to all paths of a host across all (Ingress-NGINX) Ingresses

Now, suppose that you have an Ingress with the nginx.ingress.kubernetes.io/use-regex: "true" annotation, but you want to route requests with a path of exactly /headers to httpbin. Unfortunately, you made a typo and set the path to /Header instead of /headers.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: regex-match-ingress
 annotations:
 nginx.ingress.kubernetes.io/use-regex: "true"
spec:
 ingressClassName: nginx
 rules:
 - host: regex-match.example.com
 http:
 paths:
 - path: "<some regex pattern>"
 pathType: ImplementationSpecific
 backend:
 service:
 name: <your backend>
 port:
 number: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: regex-match-ingress-other
spec:
 ingressClassName: nginx
 rules:
 - host: regex-match.example.com
 http:
 paths:
 - path: "/Header" # typo here, should be /headers
 pathType: Exact
 backend:
 service:
 name: httpbin
 port:
 number: 8000

Most would expect a request to /headers to respond with a 404 Not Found, since /headers does not match the Exact path of /Header. However, because the regex-match-ingress Ingress has the nginx.ingress.kubernetes.io/use-regex: "true" annotation and the regex-match.example.com host, all paths with the regex-match.example.com host are treated as regular expressions across all (Ingress-NGINX) Ingresses. Since regex patterns are case-insensitive prefix matches, /headers matches the /Header pattern and Ingress-NGINX routes such requests to httpbin. Running the command

curl -sS -H "Host: regex-match.example.com" http://<your-ingress-ip>/headers

the output looks like:

{
 "headers": {
 ...
 }
}

Note: The /headers endpoint of httpbin returns the request headers. The fact that the response contains the request headers in the body means that the request was successfully routed to httpbin.

Gateway API does not silently convert or interpret Exact and Prefix matches as regex patterns. So if you converted the above Ingresses into the following HTTP route and preserved the typo and match types, requests to /headers will respond with a 404 Not Found instead of a 200 OK.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: regex-match-route
spec:
 hostnames:
 - regex-match.example.com
 rules:
 ...
 - matches:
 - path:
 type: Exact
 value: "/Header"
 backendRefs:
 - name: httpbin
 port: 8000

To keep the case-insensitive prefix matching, you can change

 - matches:
 - path:
 type: Exact
 value: "/Header"

to

 - matches:
 - path:
 type: RegularExpression
 value: "(?i)/Header"

Or even better, you could fix the typo and change the match to

 - matches:
 - path:
 type: Exact
 value: "/headers"

3. Rewrite target implies regex

In this case, suppose you want to rewrite the path of requests with a path of /ip to /uuid before routing them to httpbin, and as in Section 2, you want to route requests with the path of exactly /headers to httpbin. However, you accidentally make a typo and set the path to /IP instead of /ip and /Header instead of /headers.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: rewrite-target-ingress
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: "/uuid"
spec:
 ingressClassName: nginx
 rules:
 - host: rewrite-target.example.com
 http:
 paths:
 - path: "/IP"
 pathType: Exact
 backend:
 service:
 name: httpbin
 port:
 number: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: rewrite-target-ingress-other
spec:
 ingressClassName: nginx
 rules:
 - host: rewrite-target.example.com
 http:
 paths:
 - path: "/Header"
 pathType: Exact
 backend:
 service:
 name: httpbin
 port:
 number: 8000

The nginx.ingress.kubernetes.io/rewrite-target: "/uuid" annotation causes requests that match paths in the rewrite-target-ingress Ingress to have their paths rewritten to /uuid before being routed to the backend.

Even though no Ingress has the nginx.ingress.kubernetes.io/use-regex: "true" annotation, the presence of the nginx.ingress.kubernetes.io/rewrite-target annotation in the rewrite-target-ingress Ingress causes all paths with the rewrite-target.example.com host to be treated as regex patterns. In other words, the nginx.ingress.kubernetes.io/rewrite-target silently adds the nginx.ingress.kubernetes.io/use-regex: "true" annotation, along with all the side effects discussed above.

For example, a request to /ip has its path rewritten to /uuid because /ip matches the case-insensitive prefix pattern of /IP in the rewrite-target-ingress Ingress. After running the command

curl -sS -H "Host: rewrite-target.example.com" http://<your-ingress-ip>/ip

the output is similar to:

{
 "uuid": "12a0def9-1adg-2943-adcd-1234aadfgc67"
}

Like in the nginx.ingress.kubernetes.io/use-regex example, Ingress-NGINX treats paths of other ingresses with the rewrite-target.example.com host as case-insensitive prefix patterns. Running the command

curl -sS -H "Host: rewrite-target.example.com" http://<your-ingress-ip>/headers

gives an output that looks like

{
 "headers": {
 ...
 }
}

You can configure path rewrites in Gateway API with the HTTP URL rewrite filter which does not silently convert your Exact and Prefix matches into regex patterns. However, if you are unaware of the side effects of the nginx.ingress.kubernetes.io/rewrite-target annotation and do not realize that /Header and /IP are both typos, you might create the following HTTP route.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: rewrite-target-route
spec:
 hostnames:
 - rewrite-target.example.com
 parentRefs:
 - name: <your-gateway>
 rules:
 - matches:
 - path:
 type: Exact
 value: "/IP"
 filters:
 - type: URLRewrite
 urlRewrite:
 path:
 type: ReplaceFullPath
 replaceFullPath: /uuid
 backendRefs:
 - name: httpbin
 port: 8000
 - matches:
 - path:
 # This is an exact match, irrespective of other rules
 type: Exact
 value: "/Header"
 backendRefs:
 - name: httpbin
 port: 8000

As with Section 2, because /IP is now an Exact match type in your HTTP route, requests to /ip will respond with a 404 Not Found instead of a 200 OK. Similarly, requests to /headers will also respond with a 404 Not Found instead of a 200 OK. Thus, this HTTP route will break applications and clients that rely on the /ip and /headers routes.

To fix this, you can change the matches in the HTTP route to be regex matches, and change the path patterns to be case-insensitive prefix matches, as follows.

 - matches:
 - path:
 type: RegularExpression
 value: "(?i)/IP.*"
...
 - matches:
 - path:
 type: RegularExpression
 value: "(?i)/Header.*"

Or, you can keep the Exact match type and fix the typos.

4. Requests missing a trailing slash are redirected to the same path with a trailing slash

Consider the following Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: trailing-slash-ingress
spec:
 ingressClassName: nginx
 rules:
 - host: trailing-slash.example.com
 http:
 paths:
 - path: "/my-path/"
 pathType: Exact
 backend:
 service:
 name: <your-backend>
 port:
 number: 8000

You might expect Ingress-NGINX to respond to /my-path with a 404 Not Found since the /my-path does not exactly match the Exact path of /my-path/. However, Ingress-NGINX redirects the request to /my-path/ with a 301 Moved Permanently because the only difference between /my-path and /my-path/ is a trailing slash.

curl -isS -H "Host: trailing-slash.example.com" http://<your-ingress-ip>/my-path

The output looks like:

HTTP/1.1 301 Moved Permanently
...
Location: http://trailing-slash.example.com/my-path/
...

The same applies if you change the pathType to Prefix. However, the redirect does not happen if the path is a regex pattern.

Conformant Gateway API implementations do not silently configure any kind of redirects. If clients or downstream services depend on this redirect, a migration to Gateway API that does not explicitly configure request redirects will cause an outage because requests to /my-path will now respond with a 404 Not Found instead of a 301 Moved Permanently. You can explicitly configure redirects using the HTTP request redirect filter as follows:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: trailing-slash-route
spec:
 hostnames:
 - trailing-slash.example.com
 parentRefs:
 - name: <your-gateway>
 rules:
 - matches:
 - path:
 type: Exact
 value: "/my-path"
 filters:
 requestRedirect:
 statusCode: 301
 path:
 type: ReplaceFullPath
 replaceFullPath: /my-path/
 - matches:
 - path:
 type: Exact # or Prefix
 value: "/my-path/"
 backendRefs:
 - name: <your-backend>
 port: 8000

5. Ingress-NGINX normalizes URLs

URL normalization is the process of converting a URL into a canonical form before matching it against Ingress rules and routing it. The specifics of URL normalization are defined in RFC 3986 Section 6.2, but some examples are

• removing path segments that are just a .: my/./path -> my/path
• having a .. path segment remove the previous segment: my/../path -> /path
• deduplicating consecutive slashes in a path: my//path -> my/path

Ingress-NGINX normalizes URLs before matching them against Ingress rules. For example, consider the following Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: path-normalization-ingress
spec:
 ingressClassName: nginx
 rules:
 - host: path-normalization.example.com
 http:
 paths:
 - path: "/uuid"
 pathType: Exact
 backend:
 service:
 name: httpbin
 port:
 number: 8000

Ingress-NGINX normalizes the path of the following requests to /uuid. Now that the request matches the Exact path of /uuid, Ingress-NGINX responds with either a 200 OK response or a 301 Moved Permanently to /uuid.

For the following commands

curl -sS -H "Host: path-normalization.example.com" http://<your-ingress-ip>/uuid
curl -sS -H "Host: path-normalization.example.com" http://<your-ingress-ip>/ip/abc/../../uuid
curl -sSi -H "Host: path-normalization.example.com" http://<your-ingress-ip>////uuid

the outputs are similar to

{
 "uuid": "29c77dfe-73ec-4449-b70a-ef328ea9dbce"
}

{
 "uuid": "d20d92e8-af57-4014-80ba-cf21c0c4ffae"
}

HTTP/1.1 301 Moved Permanently
...
Location: /uuid
...

Your backends might rely on the Ingress/Gateway API implementation to normalize URLs. That said, most Gateway API implementations will have some path normalization enabled by default. For example, Istio, Envoy Gateway, and Kgateway all normalize . and .. segments out of the box. For more details, check the documentation for each Gateway API implementation that you use.

Conclusion

As we all race to respond to the Ingress-NGINX retirement, I hope this blog post instills some confidence that you can migrate safely and effectively despite all the intricacies of Ingress-NGINX.

SIG Network has also been working on supporting the most common Ingress-NGINX annotations (and some of these unexpected behaviors) in Ingress2Gateway to help you translate Ingress-NGINX configuration into Gateway API, and offer alternatives to unsupported behavior.

SIG Network released Gateway API 1.5 earlier today (27th February 2026), which graduates features such as ListenerSet (that allow app developers to better manage TLS certificates), and the HTTPRoute CORS filter that allows CORS configuration.

1. You can use Istio purely as Gateway API controller with no other service mesh features. ↩︎

Kubernetes v1.36 introduces a new alpha counter metric route_controller_route_sync_total to the Cloud Controller Manager (CCM) route controller implementation at k8s.io/cloud-provider. This metric increments each time routes are synced with the cloud provider.

A/B testing watch-based route reconciliation

This metric was added to help operators validate the CloudControllerManagerWatchBasedRoutesReconciliation feature gate introduced in Kubernetes v1.35. That feature gate switches the route controller from a fixed-interval loop to a watch-based approach that only reconciles when nodes actually change. This reduces unnecessary API calls to the infrastructure provider, lowering pressure on rate-limited APIs and allowing operators to make more efficient use of their available quota.

To A/B test this, compare route_controller_route_sync_total with the feature gate disabled (default) versus enabled. In clusters where node changes are infrequent, you should see a significant drop in the sync rate with the feature gate turned on.

Example: expected behavior

With the feature gate disabled (the default fixed-interval loop), the counter increments steadily regardless of whether any node changes occurred:

# After 10 minutes with no node changes
route_controller_route_sync_total 60
# After 20 minutes, still no node changes
route_controller_route_sync_total 120

With the feature gate enabled (watch-based reconciliation), the counter only increments when nodes are actually added, removed, or updated:

# After 10 minutes with no node changes
route_controller_route_sync_total 1
# After 20 minutes, still no node changes — counter unchanged
route_controller_route_sync_total 1
# A new node joins the cluster — counter increments
route_controller_route_sync_total 2

The difference is especially visible in stable clusters where nodes rarely change.

Where can I give feedback?

If you have feedback, feel free to reach out through any of the following channels:

• The #sig-cloud-provider channel on Kubernetes Slack
• The KEP-5237 issue on GitHub
• The SIG Cloud Provider community page for other communication channels

How can I learn more?

For more details, refer to KEP-5237.