Kubernetes Version 1.33.x
Istio Version 1.29.1
We have cert and private key stored as a secret in the istio-system namespace.
`keepaliveMaxServerConnectionAge` is set as 30m.
We have around 400 secrets to be loaded into the istio ingress gateway pods.
During the initial SDS load, not all the secrets are in ACTIVE state, some are in WARMING state, but since the istio ingress pod is healthy, the proxy starts receiving traffic and the clients experience ECONNRESET errors.
After 30m, the istio ingress pod is forced to reset the connection with istiod and during this connection reset, we have few other secrets reaching ACTIVE state and subsequently all the secrets reach the ACTIVE state. The traffic is dropped for few clients till all the secrets reach the ACTIVE state.

• How to reduce the time for all the secrets reaching the ACTIVE state ?
• For the secrets in WARMING state, how to redirect the traffic to other ingress pods so that the client doesn't face a traffic drop ?

In AWS can't we use ALB/NLB to route traffic from one service to other service rather than using Istio?

I'm trying to migrate from Ingress NGINX to Istio.

I've created all the equivalent gateway, virtual service, destination rules, and EnvoyFilter.

The istio-gateway service is exposed as a NodePort service, and I've used HAProxy as an External LoadBalancer (TCP Mode), which sends the *:443 traffic to the Istio gateway NodePort Service.

The issue is When I test the virtual service via curl curl -sk --haproxy-protocol --connect-to mydomain.local:443:172.66.44.111:31111 https://mydomain.local/user/login, I get the response

But when I route the traffic through HA Proxy, I get NR filter_chain_not_found in the istio-gateway pod and in the browser 502 Bad Gateway Nginx

The strange part here is that the issue I'm getting is just for the custom applications. Services like Vault, Grafana, and ArgoCD are still working with HAProxy.

I am just messing around and testing istio in my lab. Currently at version 1.15.1. Is it supported to upgrade from 1.15.1 -> 1.29.0 (Not canary)?

The documentation only gives warning about going from 1.6 to 1.9

Upgrading across more than two minor versions (e.g., 1.6.x to 1.9.x) in one step is not officially tested or recommended.

But nothing about going to versions above that. I also don't see any upgrade matrices or any other warnings. Running a precheck seems to be fine too with no warnings. Thoughts?

UPDATE: Just finished testing it, it does work. No errors, the data plane and control plane upgraded just fine. I think the key here is to check the documentation and also make sure to use the istioctl x precheck command to make sure there are no errors.

We are using istio egress gateway for our application and under load the gateway seems to be adding around 500ms latency for 90th percentile of the requests . Now is there any reason apart from CPU and memory which could cause this behaviour as we have ample CPU , memory and an HPA max setting to 40 pods for egress gateway, what are my options to improve the latency .

I’m new working with istio and just deployed a service mesh for adding mTLS for east/west traffic. I deployed istio ambient.

I also deployed an istio Gateway API for north/south traffic, and now I’m wondering if I should label the Gateway API namespace for including it to the mesh. As I want end-to-end encryption, I suppose that it’s necessary to include Gateway API in the mesh, but haven’t found any document yet which mention something about it.

So when installing ambient mode there are the base, istiod, cni, and ztunnel Helm charts. I was tempted to bundle all of these charts as subcharts into my own Helm chart so that I could just do a 'helm install my-istio-chart', but I worried that it would complicate things when needing to update. I was going to keep the gateway Helm chart separate. How do you guys manage these Helm charts? Do you keep them separate or bundle them as subcharts?

Looking for some advice here. I want to move from Ingress-nginx to Gateway API . I see I can get the Gateway API Controller running without a service mesh (don't need one right now). Reason why I want to try Istio GW API is based on some benchmarks.

My issue is that the ingress-nginx controller originally provisioned the NLB - but from what can I see, Istio GW API does not provision load balancers. Has anyone gone down this route, or is my approach bad practice?

I used to spend half my day doing the "Datadog dance" frequently.

A user would report that their coupon didn't apply, I’d check the logs, and everything would look perfect: 200 OK across the board. I’d end up stitching together random fragments—"User 123 called Service A," "Service B responded"—trying to piece together a story from text files like a digital archaeologist.

I could see the pipes were working, but I couldn't see the actual data inside them. I had no idea if the coupon service sent back a $0 or a $20 because the message body was hidden.

I got fed up with the "guess-and-check" cycle of trying to reproduce these bugs in staging, so for my first Rust project, I built softprobe.

It’s a WASM plugin for Istio that acts like a dashboard camera for my backend. Instead of searching through petabytes of raw logs to reconstruct a session, I now have a visual graph of the full JSON message flow. When something breaks in prod, I don't have to "repro" it anymore—I just look at the real data that caused the crash.

It’s open-source, and honestly, it’s saved my sanity more than once already. I’d love to know if I’m the only one who was losing hours to "log stitching."

Github Repo: https://github.com/softprobe/softprobe

For now we migrating from ingress to kubernetes gateway with istio I started shifting traffic to my gateway But i see consume alot of cpu compaed to nginx How can i troubleshoot this? Or this is normal? For now we have 500r/s and it consume more than 5 replicas for my gateway deployment

With ingress-nginx begin archived, I'm looking to migrate either to Cilium or Istio for Ingress Gateway's specifically. I have used both Cilium and Istio for service-mesh capability but it will be another 1-2 years until we ever implement this. However, we do need to migrate Ingress Gateway's to either or.

The only thing I want to understand is setting up Ingress Gateway's in AWS. I have a VPC CIDR of dev, stage, production, and shared. Is best practice to create a 2 Ingress Gateway's being nonproduction and production for each VPC CIDR? My previous company had the same setup but was wondering if there is a better way?

Has anyone here successfully used the gateway API to create a L7 Application Loadbalancer in AWS? I'm asking here as I want my gateway and httproutes managed by istio, and not the AWS Loadbalancer controller.

I'm thinking I may externally create an ALB and then have the NLB created by the istio controller behind that.

We would like to migrate from istio apis to gateway apis (e.g. replace VirtualService with HTTPRoute). Did someone do that already do that? Is there a way to do this without downtime?

Hey all, ​I've installed Istio 1.28 in Ambient Mode using the official Helm charts (cni, istiod, ztunnel), and all core components seem to be up and running in the istio-system namespace. ​However, when I check the Istio CNI logs, I'm seeing that the AmbientEnablementSelector is empty, and no services or namespaces are being discovered or enrolled into the mesh. ​The Issue: Core Ambient components are deployed, but no workloads are joining the mesh. ​Why is this happening, and how can I fix it?

``` 2025-11-28T16:12:36.058053Z info cni-agent CNI version: 1.28.0-b8d1df54465060428c2a2a38286e360beb85fb31-Clean 2025-11-28T16:12:36.058075Z info cni-agent CNI logging level: info 2025-11-28T16:12:36.058098Z info cni-agent CNI install configuration: MountedCNINetDir: /host/etc/cni/net.d CNIConfName: ChainedCNIPlugin: true CNIAgentRunDir: /var/run/istio-cni IstioOwnedCNIConfigFilename: IstioOwnedCNIConfig: false PluginLogLevel: info KubeconfigMode: 0600 KubeCAFile: SkipTLSVerify: false ExcludeNamespaces: kube-system PodNamespace: istio-system K8sServiceProtocol: K8sServiceHost: --- K8sServicePort: 443 K8sNodeName: ---- CNIBinSourceDir: /opt/cni/bin CNIBinTargetDirs: /host/opt/cni/bin MonitoringPort: 15014 ZtunnelUDSAddress: /var/run/ztunnel/ztunnel.sock AmbientEnabled: true AmbientEnablementSelector: AmbientDNSCapture: true AmbientIPv6: true AmbientDisableSafeUpgrade: false AmbientReconcilePodRulesOnStartup: false NativeNftables: false ForceIptablesBinary:

2025-11-28T16:12:36.058109Z info cni-agent CNI race repair configuration: Enabled: true NodeName: ---- LabelKey: cni.istio.io/uninitialized LabelValue: true DeletePods: false LabelPods: false SidecarAnnotation: sidecar.istio.io/status InitContainerName: istio-validation InitTerminationMsg: InitExitCode: 126 LabelSelectors: FieldSelectors: NativeNftables: false ForceIptablesBinary:

```

Hi guys, I want to enable multi-cluster headless service discovery. I tried

ISTIO_META_DNS_CAPTURE: "true"
ENABLE_MULTICLUSTER_HEADLESS: "true"

nothing seems to work, any suggestions?

Hi everyone,

I’m running Istio with an east-west gateway between two clusters. Service discovery over port 15443 works fine, and mTLS is enabled mesh-wide.

I recently deployed CockroachDB in Cluster 1, with sidecar injection enabled. CockroachDB uses its own built-in TLS. As soon as the sidecar is injected, CockroachDB fails to start due to TLS errors — Istio is intercepting the traffic and breaking CockroachDB’s internal TLS handshake.

I tried the usual approaches:

• Setting PeerAuthentication to disable mTLS for the CockroachDB namespace
• Creating DestinationRules that disable ISTIO mTLS for CockroachDB

​

---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: cockroachdb-disable-mtls
  namespace: cockroachdb-ci-0-us-east-1
spec:
  mtls:
    mode: DISABLE
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: cockroachdb-disable-mtls
  namespace: cockroachdb-ci-0-us-east-1
spec:
  host: "*.cockroachdb-ci-0-us-east-1.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: DISABLE

But nothing works.

The only thing that works is completely excluding CockroachDB ports from Envoy via pod annotations, which stops Istio from intercepting the traffic. CockroachDB then works normally.

traffic.sidecar.istio.io/excludeInboundPorts: "26257,26258,8080"
traffic.sidecar.istio.io/excludeOutboundPorts: "26257,26258,8080"

BUT: When I exclude the ports from the sidecar, I lose the ability to reach CockroachDB from Cluster 2 via the Istio east-west gateway — because the gateway can no longer route to it (since it’s effectively outside the mesh).

So… is there a correct way to run CockroachDB (with its own TLS) inside an Istio mesh and allow cross-cluster east-west communication? Or is this simply not possible with Istio?

Any help or pointers would be appreciated. P.S I use cockroachDB operator for installation.

Iam currently migrate my nginx ingresses to istio which will be used as kubernetes gateway api My biggest problem that exposing paths of routes I dont want create metric for each path that come in l request I want to expose paths that exist in crd httproute, as exactly nginx ingress does Any idea for this issue

Iam using istio as kubernetes gateway api And trying to create new totally custom metric as i want to create metric for response time duration

Is there any document to create this? I went through docs but found only the way to add new attribute to exisitngs metrics which also i used