text:
istio-cni - needed kubelet args are docker only?

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities
To ask questions about how to use Istio, please visit https://discuss.istio.io)
Bug description
So my question might seem strange at first but let me provide some context:
I am getting the following message from istio-validation init container:
kubectl logs <pod> -n <namespace> -c istio-validation

in new validator: 10.66.26.76
Listening on 127.0.0.1:15001
Listening on 127.0.0.1:15006
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
2021-07-12T19:57:33.687386Z     error   validation timeout

here is what kubelet is saying something like this:
Jul 12 18:43:27 ip-10-42-20-161.ec2.internal kubelet[67025]: E0712 18:43:27.349387   67025 pod_workers.go:191] Error syncing pod dcc1cbed-3680-4a8a-9121-cdbea42f63f8 ("istio-cni-node-2ws8s_istio-system(dcc1cbed-3680-4a8a-9121-cdbea42f63f8)"), skipping: failed to "StartContainer" for "install-cni" with CrashLoopBackOff: "back-off 10s restarting failed container=install-cni pod=istio-cni-node-2ws8s_istio-system(dcc1cbed-3680-4a8a-9121-cdbea42f63f8)"
Jul 12 18:43:29 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:29.021940   67025 scope.go:111] [topologymanager] RemoveContainer - Container ID: 9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038
Jul 12 18:43:29 ip-10-42-20-161.ec2.internal kubelet[67025]: E0712 18:43:29.022595   67025 pod_workers.go:191] Error syncing pod dcc1cbed-3680-4a8a-9121-cdbea42f63f8 ("istio-cni-node-2ws8s_istio-system(dcc1cbed-3680-4a8a-9121-cdbea42f63f8)"), skipping: failed to "StartContainer" for "install-cni" with CrashLoopBackOff: "back-off 10s restarting failed container=install-cni pod=istio-cni-node-2ws8s_istio-system(dcc1cbed-3680-4a8a-9121-cdbea42f63f8)"
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.393549   67025 scope.go:111] [topologymanager] RemoveContainer - Container ID: 9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.415153   67025 scope.go:111] [topologymanager] RemoveContainer - Container ID: 98894d5766323d9d87cfcb06fd25521cace336969451b0db95b0ffd379828b63
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.446003   67025 scope.go:111] [topologymanager] RemoveContainer - Container ID: 9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: E0712 18:43:39.446416   67025 remote_runtime.go:332] ContainerStatus "9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038" from runtime service failed: rpc error: code = NotFound desc = an error occurred when try to find container "9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038": not found
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: W0712 18:43:39.446447   67025 pod_container_deletor.go:52] [pod_container_deletor] DeleteContainer returned error for (id={containerd 9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038}): failed to get container status "9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038": rpc error: code = NotFound desc = an error occurred when try to find container "9af08de38c1a7165151f5a075a4d0be62a6cfa2a4c5ae686fc5200f58ee4b038": not found
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.446463   67025 scope.go:111] [topologymanager] RemoveContainer - Container ID: 98894d5766323d9d87cfcb06fd25521cace336969451b0db95b0ffd379828b63
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: E0712 18:43:39.446777   67025 remote_runtime.go:332] ContainerStatus "98894d5766323d9d87cfcb06fd25521cace336969451b0db95b0ffd379828b63" from runtime service failed: rpc error: code = NotFound desc = an error occurred when try to find container "98894d5766323d9d87cfcb06fd25521cace336969451b0db95b0ffd379828b63": not found
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: W0712 18:43:39.446800   67025 pod_container_deletor.go:52] [pod_container_deletor] DeleteContainer returned error for (id={containerd 98894d5766323d9d87cfcb06fd25521cace336969451b0db95b0ffd379828b63}): failed to get container status "98894d5766323d9d87cfcb06fd25521cace336969451b0db95b0ffd379828b63": rpc error: code = NotFound desc = an error occurred when try to find container "98894d5766323d9d87cfcb06fd25521cace336969451b0db95b0ffd379828b63": not found
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.532150   67025 reconciler.go:196] operationExecutor.UnmountVolume started for volume "istio-cni-token-kqdmt" (UniqueName: "kubernetes.io/secret/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-istio-cni-token-kqdmt") pod "dcc1cbed-3680-4a8a-9121-cdbea42f63f8" (UID: "dcc1cbed-3680-4a8a-9121-cdbea42f63f8")
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.532193   67025 reconciler.go:196] operationExecutor.UnmountVolume started for volume "cni-bin-dir" (UniqueName: "kubernetes.io/host-path/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-cni-bin-dir") pod "dcc1cbed-3680-4a8a-9121-cdbea42f63f8" (UID: "dcc1cbed-3680-4a8a-9121-cdbea42f63f8")
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.532218   67025 reconciler.go:196] operationExecutor.UnmountVolume started for volume "cni-net-dir" (UniqueName: "kubernetes.io/host-path/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-cni-net-dir") pod "dcc1cbed-3680-4a8a-9121-cdbea42f63f8" (UID: "dcc1cbed-3680-4a8a-9121-cdbea42f63f8")
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.532297   67025 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-cni-net-dir" (OuterVolumeSpecName: "cni-net-dir") pod "dcc1cbed-3680-4a8a-9121-cdbea42f63f8" (UID: "dcc1cbed-3680-4a8a-9121-cdbea42f63f8"). InnerVolumeSpecName "cni-net-dir". PluginName "kubernetes.io/host-path", VolumeGidValue ""
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.532520   67025 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-cni-bin-dir" (OuterVolumeSpecName: "cni-bin-dir") pod "dcc1cbed-3680-4a8a-9121-cdbea42f63f8" (UID: "dcc1cbed-3680-4a8a-9121-cdbea42f63f8"). InnerVolumeSpecName "cni-bin-dir". PluginName "kubernetes.io/host-path", VolumeGidValue ""
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.543059   67025 operation_generator.go:797] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-istio-cni-token-kqdmt" (OuterVolumeSpecName: "istio-cni-token-kqdmt") pod "dcc1cbed-3680-4a8a-9121-cdbea42f63f8" (UID: "dcc1cbed-3680-4a8a-9121-cdbea42f63f8"). InnerVolumeSpecName "istio-cni-token-kqdmt". PluginName "kubernetes.io/secret", VolumeGidValue ""
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.632451   67025 reconciler.go:319] Volume detached for volume "istio-cni-token-kqdmt" (UniqueName: "kubernetes.io/secret/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-istio-cni-token-kqdmt") on node "ip-10-42-20-161.ec2.internal" DevicePath ""
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.632483   67025 reconciler.go:319] Volume detached for volume "cni-bin-dir" (UniqueName: "kubernetes.io/host-path/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-cni-bin-dir") on node "ip-10-42-20-161.ec2.internal" DevicePath ""
Jul 12 18:43:39 ip-10-42-20-161.ec2.internal kubelet[67025]: I0712 18:43:39.632493   67025 reconciler.go:319] Volume detached for volume "cni-net-dir" (UniqueName: "kubernetes.io/host-path/dcc1cbed-3680-4a8a-9121-cdbea42f63f8-cni-net-dir") on node "ip-10-42-20-161.ec2.internal" DevicePath ""

Were probably wondering why I questioned containerd support here, but in the istio cni docs it says:

with the container runtime supporting CNI and kubelet configured with the main CNI plugin enabled via --network-plugin=cni

and reading others who are troubleshooting the cni, mentioned these paths also in the cni istio docs:

Determine the Kubernetes environment’s CNI plugin --cni-bin-dir and --cni-conf-dir settings. Refer to Hosted Kubernetes settings for any non-default settings required

but from what I"m seeing all three of these in the kubelet docs say:

only works when container-runtime is set to docker

I was going to start looking at maybe selinux, but i dont see any denials coming from there.
So just to rule this out, does the cni install only work with docker for the time being? Otherwise, what could be causing this? Those args are apparently ignored if not set to docker, containerd is set to "remote", not docker. Or I'm not sure what else I can do to track down the cause of this issue. also if it helps, I'm using calico, and that kubelet line is interesting:
failed to "StartContainer" for "install-cni" with CrashLoopBackOff: "back-off 10s restarting failed container=install-cni pod=istio-cni-node-2ws8s_istio-system(dcc1cbed-3680-4a8a-9121-cdbea42f63f8)
because theres a "install-cni" container, but the pod isn't istio-cni its "io.kubernetes.pod.name": "calico-node-fcg84" (according to the labels from an inspect), so that must be from calico...could it be they are clashing over the same name?
[ ] Docs
[ X] Installation
[X ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
Expected behavior
istio cni is installed successfully and gets past the init container: istio-validate
Steps to reproduce the bug
Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)
client version: 1.10.0
control plane version: 1.9.4
data plane version: 1.9.4 (1 proxies)
kubectl version --short
Client Version: v1.21.2
How was Istio installed?
istio operator which was done by istioctl init
Environment where the bug was observed (cloud vendor, OS, etc)
aws
Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.
will do this if needed, this might be just a quick "cannot use containerd" response.

The crash says pod=istio-cni-node-2ws8s_istio-system, so I think it is from our pod not calico? If that is crashing then this error is expected.
You  definitely should be able to run with containerd

Sorry let me clarify, istio-cni-node-2ws8s_istio-system never shows up, and there is no trace of it in this node, i dont see it in the container runtime (crictl) as exited nor the pods (kubectl), not even in a crashloop state or anything.
there is an install-cni but it appears to be calico's install cni. so I was just entertaining the idea maybe they cant have the same name or something since it doesnt even seem to get as far as trying to run them and crashing from what I can see. But I dont know where else to look if I can't even find the crashed containers logging etc...
good to know containerd should be fine. the k8s docs make it sound like the needed kubelet args are docker only.

I found out why I can't find the daemonset or pods or containers, the operator claims to be removing it, but I dont see a good explanation as to why:
kubectl logs istio-operator-5569dc8964-9lml8 -n istio-operator | grep -i cni
2021-07-13T13:16:39.000289Z     info    installer       Cni is waiting on dependency...
2021-07-13T13:16:59.671371Z     info    installer       Unblocking dependency Cni.
2021-07-13T13:16:59.671404Z     info    installer       Dependency for Cni has completed, proceeding.
2021-07-13T13:16:59.671475Z     info    installer       Processing resources from manifest: Cni for CR istio-controlplane-istio-system-Cni-https://10.96.0.1:443
 - ClusterRole::istio-cni
 - ClusterRole::istio-cni-repair-role
 - ClusterRoleBinding::istio-cni
 - ClusterRoleBinding::istio-cni-repair-rolebinding
 - ConfigMap:istio-system:istio-cni-config
 - DaemonSet:istio-system:istio-cni-node
 - ServiceAccount:istio-system:istio-cni
2021-07-13T13:16:59.674139Z     info    installer       using server side apply to update obj: ServiceAccount/istio-system/istio-cni
2021-07-13T13:16:59.688130Z     info    installer       using server side apply to update obj: ClusterRole//istio-cni
2021-07-13T13:16:59.708621Z     info    installer       using server side apply to update obj: ClusterRole//istio-cni-repair-role
2021-07-13T13:16:59.748380Z     info    installer       using server side apply to update obj: ClusterRoleBinding//istio-cni
2021-07-13T13:16:59.816055Z     info    installer       using server side apply to update obj: ClusterRoleBinding//istio-cni-repair-rolebinding
2021-07-13T13:16:59.927003Z     info    installer       using server side apply to update obj: ConfigMap/istio-system/istio-cni-config
2021-07-13T13:17:00.003645Z     info    installer       using server side apply to update obj: DaemonSet/istio-system/istio-cni-node
- Processing resources for CNI, Ingress gateways.
✔ CNI installed
2021-07-13T13:17:24.003808Z     info    installer       Cni is waiting on dependency...
2021-07-13T13:17:38.996511Z     info    installer       Unblocking dependency Cni.
2021-07-13T13:17:38.996612Z     info    installer       Dependency for Cni has completed, proceeding.
2021-07-13T13:17:38.996630Z     info    installer       Processing resources from manifest: Cni for CR istio-controlplane-istio-system-Cni-https://10.96.0.1:443
2021-07-13T13:17:38.996639Z     info    installer       Generated manifest objects are the same as cached for component Cni.
2021-07-13T13:17:38.996645Z     info    installer       Pruning object ServiceAccount:istio-system:istio-cni from cache.
2021-07-13T13:17:38.996650Z     info    installer       Pruning object ClusterRole::istio-cni from cache.
2021-07-13T13:17:38.996654Z     info    installer       Pruning object ClusterRole::istio-cni-repair-role from cache.
2021-07-13T13:17:38.996658Z     info    installer       Pruning object ClusterRoleBinding::istio-cni from cache.
2021-07-13T13:17:38.996663Z     info    installer       Pruning object ClusterRoleBinding::istio-cni-repair-rolebinding from cache.
2021-07-13T13:17:38.996667Z     info    installer       Pruning object ConfigMap:istio-system:istio-cni-config from cache.
2021-07-13T13:17:38.996671Z     info    installer       Pruning object DaemonSet:istio-system:istio-cni-node from cache.
2021-07-13T13:17:39.932036Z     info    installer       Deleting DaemonSet:istio-system:istio-cni-node (istio-controlplane/)
2021-07-13T13:17:39.932067Z     info    installer       Removed object DaemonSet:istio-system:istio-cni-node from Cache.
  Removed DaemonSet:istio-system:istio-cni-node.
2021-07-13T13:17:39.932100Z     info      Removed DaemonSet:istio-system:istio-cni-node.
2021-07-13T13:17:39.933075Z     info    installer       Watching a change for istio resource: istio-system/istio-cni-node
2021-07-13T13:17:40.899996Z     info    installer       Deleting ConfigMap:istio-system:istio-cni-config (istio-controlplane/)
2021-07-13T13:17:40.900030Z     info    installer       Removed object ConfigMap:istio-system:istio-cni-config from Cache.
  Removed ConfigMap:istio-system:istio-cni-config.
2021-07-13T13:17:40.900054Z     info      Removed ConfigMap:istio-system:istio-cni-config.
2021-07-13T13:17:40.900295Z     info    installer       Watching a change for istio resource: istio-system/istio-cni-config
2021-07-13T13:17:41.998172Z     info    installer       Watching a change for istio resource: istio-system/istio-cni
2021-07-13T13:17:42.085268Z     info    installer       Deleting ServiceAccount:istio-system:istio-cni (istio-controlplane/)
2021-07-13T13:17:42.085306Z     info    installer       Removed object ServiceAccount:istio-system:istio-cni from Cache.
  Removed ServiceAccount:istio-system:istio-cni.
2021-07-13T13:17:42.085331Z     info      Removed ServiceAccount:istio-system:istio-cni.
2021-07-13T13:17:44.898875Z     info    installer       Deleting ClusterRole::istio-cni (istio-controlplane/)
2021-07-13T13:17:44.898907Z     info    installer       Removed object ClusterRole::istio-cni from Cache.
  Removed ClusterRole::istio-cni.
2021-07-13T13:17:44.898932Z     info      Removed ClusterRole::istio-cni.
2021-07-13T13:17:44.899176Z     info    installer       Watching a change for istio resource: /istio-cni
2021-07-13T13:17:44.905235Z     info    installer       Watching a change for istio resource: /istio-cni-repair-role
2021-07-13T13:17:44.905324Z     info    installer       Deleting ClusterRole::istio-cni-repair-role (istio-controlplane/)
2021-07-13T13:17:44.905339Z     info    installer       Removed object ClusterRole::istio-cni-repair-role from Cache.
  Removed ClusterRole::istio-cni-repair-role.
2021-07-13T13:17:44.905368Z     info      Removed ClusterRole::istio-cni-repair-role.
2021-07-13T13:17:44.923287Z     info    installer       Watching a change for istio resource: /istio-cni
2021-07-13T13:17:44.923370Z     info    installer       Deleting ClusterRoleBinding::istio-cni (istio-controlplane/)
2021-07-13T13:17:44.923394Z     info    installer       Removed object ClusterRoleBinding::istio-cni from Cache.
  Removed ClusterRoleBinding::istio-cni.
2021-07-13T13:17:44.923432Z     info      Removed ClusterRoleBinding::istio-cni.
2021-07-13T13:17:44.931061Z     info    installer       Deleting ClusterRoleBinding::istio-cni-repair-rolebinding (istio-controlplane/)
2021-07-13T13:17:44.931098Z     info    installer       Removed object ClusterRoleBinding::istio-cni-repair-rolebinding from Cache.
  Removed ClusterRoleBinding::istio-cni-repair-rolebinding.
2021-07-13T13:17:44.931131Z     info      Removed ClusterRoleBinding::istio-cni-repair-rolebinding.
2021-07-13T13:17:44.931312Z     info    installer       Watching a change for istio resource: /istio-cni-repair-rolebinding
2021-07-13T13:17:47.700130Z     info    installer       Cni is waiting on dependency...
2021-07-13T13:17:48.197026Z     info    installer       Unblocking dependency Cni.
2021-07-13T13:17:48.197082Z     info    installer       Dependency for Cni has completed, proceeding.
2021-07-13T13:17:48.197206Z     info    installer       Processing resources from manifest: Cni for CR istio-controlplane-istio-system-Cni-https://10.96.0.1:443
2021-07-13T13:17:48.197229Z     info    installer       Generated manifest objects are the same as cached for component Cni.

any ideas from the messages why its removing it?

I think it is probably because of priority class. Istio-cni has system-node-critical priority class, which requires the scheduled namespace to have ResourceQuota defined. If you describe istio-cni daemonset you should be able to see why the pod cannot be scheduled.
apiVersion: v1
kind: ResourceQuota
metadata:
  name: critical-pods
  namespace: istio-system
spec:
  hard:
    pods: 1G
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values:
      - system-node-critical
      - system-cluster-critical

@bianpengyuan based on the messages i pasted it looks like the daemonset is being deleted so I can't view that or any pod logs from them:
2021-07-13T13:17:39.932036Z     info    installer       Deleting DaemonSet:istio-system:istio-cni-node (istio-controlplane/)
part of what makes this so puzzling is I cannot view anything, to see status/messages, and kubelet just has generic failed messages.
So we have to have a ResourceQuota defined? Shouldn't that be in the prerequisites section then?

I might have figured out the problem, if I am correct I will update here.

ok I think im on to something. First, I used istioctl to install instead of just generating manifest and using kubectl, this actually kept the pods up even though they were crashing so I could investigate better.
Then...this helped me see I was getting the latest istioctl but the repository and tag were using is not latest were using 1.9.4, but the manifest didnt seem to match (namely the entrypoint), so the pods were crashing complaining about not being able to find the repair cni binary. Once I matched the istioctl version with the version were actually using repair container came up but install-cni is still crashing but now I can see its because of selinux:
time->Wed Jul 14 18:40:44 2021
type=PROCTITLE msg=audit(1626288044.336:3433): proctitle="install-cni"
type=SYSCALL msg=audit(1626288044.336:3433): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=c00002c5c0 a2=800c2 a3=180 items=0 ppid=122424 pid=143011 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="install-cni" exe="/usr/local/bin/install-cni" subj=system_u:system_r:container_t:s0:c120,c942 key=(null)
type=AVC msg=audit(1626288044.336:3433): avc:  denied  { write } for  pid=143011 comm="install-cni" name="net.d" dev="nvme0n1p2" ino=13158669 scontext=system_u:system_r:container_t:s0:c120,c942 tcontext=system_u:object_r:etc_t:s0 tclass=dir permissive=0

before, I wasn't even getting this far, because there were no selinux denials, most likely because the binary couldnt be found in the container (wrong path). but at least with this I can try to deal with selinux.

ok so that selinux message was the last hurdle to getting this working.
so in conclusion:
It seems you want to match istioctl version with w/e version of istio you are trying to use, so in my case targetting 1.9.x do not use the newest istioctl, it affects the manifest (i think) and can result in issues such as a different path for the binary(s) than what is in the container.
i think this was the core issue. the selinux denial was expected. I'm closing this.

authors: bianpengyuan howardjohn perezjasonr


text:
cluster VIPs are not correctly updated when a multi-cluster Service stub is removed

Bug Description
Cluster VIPs can get into an incorrect state when multi-cluster Service stub resources are removed.
To reproduce:

Set Up two clusters: one local and one remote with DNS proxy enabled
Set up a Service example.example-namespace.svc.cluster.local in remote
Set up a Service example.example-namespace.svc.cluster.local in local
Check the debug endpoint for a control plane in local cluster at /debug/endpointz. Look for clusterVIPs for hostname example.example-namespace.svc.cluster.local.
Perform a DNS lookup on a pod in remote cluster and note the clusterIP returned
Remove the Service from remote cluster
Check the debug endpoint for a control plane in local cluster at /debug/endpointz. Look for clusterVIPs for hostname example.example-namespace.svc.cluster.local. Note that the removed clusterIP still exists just as in the result of 4.
Perform a DNS lookup on a pod in remote cluster and note the clusterIP returned is the same as 5.
Restart the local cluster control plane pods.
Check the debug endpoint for a control plane in local cluster at /debug/endpointz. Look for clusterVIPs for hostname example.example-namespace.svc.cluster.local. Note that the clusterIP returned in 5 is no longer present.
Perform a DNS lookup on a pod in remote cluster and note the clusterIP returned is no longer the same as 5. It is now the clusterIP of the local cluster.

Expected Behavior:
It is expected that the cluster registry is updated when the remote services are removed. Otherwise incorrect routing may occur when a new Service is created with the old clusterIP. A restart should not be required to get the control plane in a correct state.
Version
tested on istio 1.12.4/1.12.5

Additional Information
No response

This is a bug if as described

/assign

I have reproduced,  thanks for the info

source: gharchive/issue

authors: Minh-Ng hzxuzhonghu

repo: istio/istio

license_type: permissive

license_source: github-api


text:
can we add a feature like kubectl get all to get all resource in istio

(This is used to request new product features, please visit https://discuss.istio.io for questions on using Istio)
Describe the feature request
can we add a feature like kubectl get all to get all resource in istio
Describe alternatives you've considered
istioctl get all
Affected product area (please put an X in all that apply)
[ x ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Additional context

kubectl get istio-io already does this

Its a builtin feature of k8s by the category field on CRD

authors: howardjohn xiayanzheng


text:
Getting error during helm install task with the gateway and virtual service version

Bug Description
##[error]Error: unable to build kubernetes objects from release manifest: [unable to recognize "": no matches for kind "Gateway" in version "networking.istio.io/v1", unable to recognize "": no matches for kind "VirtualService" in version "networking.istio.io/v1"]
I tried v1aplha3 and v1beta1 and also v1 but always getting this issue in the pipeline
Version
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: {{ .Values.app.name }}-gateway
  namespace: {{ .Values.app.namespace }}
spec:
  selector:
    istio: {{ .Values.ingressgateway.name}}-gateway # use istio default controller
  servers:
  - port:
      number: {{ .Values.port }}
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name:  {{ .Values.app.name }}-virtualservice
  namespace: {{ .Values.app.namespace }}
spec:
  hosts:
  - "*"
  gateways:
  - {{ .Values.app.name }}-gateway
  http:
  - match:
    - uri:
        exact: /
    route:
    - destination:
        host: {{ .Values.app.name }}-service
        port:
          number: {{ .Values.port }}

Additional Information
No response

did you install CRD and CR in same chart?

did you install CRD and CR in same chart?

This task used to work for me with the same code and suddenly it stopped working.
Can you please let me know how to do it?

CRDs needs to be installed prior to the CRs. Can you check if they have been correctly installed?

Istio has been correctly installed on the aks cluster. I have verified that. I have upgraded the repo just now as well

incorrect api version. want networking.istio.io/v1 but got networking.istio.io/v1alpha3.

check the crd installed use kubectl describe crd gateways.networking.istio.io | grep 'API Version'

check the crd installed use kubectl describe crd gateways.networking.istio.io | grep 'API Version'

got it like this :
API Version:  apiextensions.k8s.io/v1
API Version:  apiextensions.k8s.io/v1
API Version:  apiextensions.k8s.io/v1
I am getting error even when I am using v1

kubectl describe crd gateways.networking.istio.io | grep 'API Version'

It is also not editable. Do you know how we can reinstall this particular CRD

kubectl describe crd gateways.networking.istio.io | grep 'API Version'

It is also not editable. Do you know how we can reinstall this particular CRD
I am getting the stored version in the CRD as below:
storedVersions:

v1alpha3

sorry, misread, what I said above is wrong.

sorry, misread, what I said above is wrong. kubectl describe crd gateways.networking.istio.io | grep Name may be useful (eg. check gateway crd version).
the definition of crd is under the istio/manifests/charts/base/crds/ directory of the source code

sorry, misread, what I said above is wrong. kubectl describe crd gateways.networking.istio.io | grep Name may be useful (eg. check gateway crd version).
the definition of crd is under the istio/manifests/charts/base/crds/ directory of the source code

I deleted and reinstalled , but i am still facing the same issue

The above issue has been fixed. The fix was that in the main branch the gateway file had the version v1 and I was trying to change the version to one of the v1alpha3 and v1beta1 to fix the issue in one the child branches and it was somehow pointing to the main branch. I changed the version to v1alpha3 on main branch it got fixed. Closing this issue

authors: hanxiaop sahilvarmacsi wulianglongrd zirain


text:
Certificate Challenges failing with Istio + Istio Gateway when installed via Helm (Part 2)

Please note, all of this is installed per https://istio.io/latest/docs/setup/install/helm/ as of 2024-02-01, and I've installed istio/base, istio/istiod, and istio/gateway with no change in the default configurations.
          Confirming the findings of @afreemanio, these steps no longer work.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: redacted
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - selector: {}
      http01:
        ingress:
          class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: istio-test-cert
  namespace: istio-system
spec:
  secretName: istio-test-cert
  issuerRef:
   name: letsencrypt-staging
   kind: ClusterIssuer
  commonName: f3.redacted
  dnsNames:
  - f3.redacted

And I have
ingressService: istio-gateway
ingressSelector: gateway

In the mesh config for istiod.
cert-manager creates the ingress, and seemingly istiod notices that per these logs:
024-02-01T19:23:20.158224Z	info	ads	Push debounce stable[18] 2 for config VirtualService/istio-system/cm-acme-http-solver-vqd4f-virtualservice and 1 more configs: 100.252732ms since last change, 100.255732ms since last push, full=true
2024-02-01T19:23:20.158561Z	info	ads	XDS: Pushing Services:17 ConnectedEndpoints:1 Version:2024-02-01T19:23:20Z/12
2024-02-01T19:23:20.158943Z	info	ads	CDS: PUSH for node:istio-ingress-6f9c5dd58d-f67px.istio-ingress resources:25 size:25.9kB cached:24/24
2024-02-01T19:23:20.158982Z	info	ads	LDS: PUSH for node:istio-ingress-6f9c5dd58d-f67px.istio-ingress resources:0 size:0B
2024-02-01T19:23:20.158996Z	warn	buildGatewayRoutes: no gateways for router istio-ingress-6f9c5dd58d-f67px.istio-ingress
2024-02-01T19:23:20.159015Z	info	ads	RDS: PUSH for node:istio-ingress-6f9c5dd58d-f67px.istio-ingress resources:1 size:11B cached:0/0

Since the errors specified there was no gateway, I thought maybe I need to pre-setup the gateway for it:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cert-gateway
  namespace: istio-ingress
spec:
  selector:
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "f3.redacted"

But this still doesn't work, and istiod has this in its logs this time:
2024-02-01T19:22:00.043306Z	info	ads	Push debounce stable[15] 2 for config VirtualService/istio-system/cm-acme-http-solver-2ftck-virtualservice and 1 more configs: 100.366129ms since last change, 100.409929ms since last push, full=true
2024-02-01T19:22:00.043560Z	info	ads	XDS: Pushing Services:17 ConnectedEndpoints:1 Version:2024-02-01T19:22:00Z/9
2024-02-01T19:22:00.044009Z	info	ads	CDS: PUSH for node:istio-ingress-6f9c5dd58d-f67px.istio-ingress resources:25 size:25.9kB cached:24/24
2024-02-01T19:22:00.044191Z	info	ads	LDS: PUSH for node:istio-ingress-6f9c5dd58d-f67px.istio-ingress resources:1 size:3.4kB
2024-02-01T19:22:00.044213Z	warn	constructed http route config for route http.80 on port 80 with no vhosts; Setting up a default 404 vhost
2024-02-01T19:22:00.044235Z	info	ads	RDS: PUSH for node:istio-ingress-6f9c5dd58d-f67px.istio-ingress resources:1 size:38B cached:0/0

Originally posted by @aaomidi in https://github.com/istio/istio/issues/37329#issuecomment-1922044826

ingressSelector: gateway

Your ingressSelector should match the value of the selector in the Gateway. So it should be ingressSelector: ingress

Thanks! That solved the problem.
I'm honestly not sure how I would've come across this. A few questions here:

What is the default value for ingressSelector? I'm not sure I understand what https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/ is. I've looked at the istiod logs when it comes up and that field was not populated in the big JSON dump of the configuration it gives.

Where can we document this behavior better? I feel like istio should work OOTB with cert-manager as much as possible. I'm trying to understand the implications of changing this config value.

The crux of the issue is the different selector used helm installs of the ingress gateway vs. istioctl installs. There are historical reasons for this, but the short version is that we changed the recommended namespace of ingress gateways, and as such, changed the selector. To answer your enumerated questions:

The default value is istio-ingressgateway (note this is different from the Helm default)
I'm open to suggestions; what would've helped you in your journey?

For 1. I think when istiod dumps the configuration loaded, I would like to see the values for ingressService, and ingressSelector dumped.
For example, I've now tried this setup with https://learn.microsoft.com/en-us/azure/aks/istio-deploy-ingress, but I'm not sure what value for ingressSelector they're using, and I've not found a way to get that value dumped out.
For 2. I think honestly, probably either having a page explaining how to integrate with cert-manager on the istio site or on the cert-manager site (i believe their docs are FOSS, so could just be maintaining them).

https://github.com/istio/istio/issues/49154#issuecomment-1922215258 Was the solution. What's important to note is that unfortunately due to limitations on how this is handled, the certificate object needs to be in the same namespace as the istio gateway :/

authors: aaomidi keithmattix


text:
failoverPriority and missing IstioEndpoint labels in Multi-Primary on different networks

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
I have a working Multi-Primary on different networks setup that consists of two k8s clusters, almost identical to the setup described here except deployed with helm. Locality load balancing using basic "just enabled", distribute and failover settings works perfectly. But the failoverPriority setting doesn't seem to work at all. After debugging this issue for a bit, it seems to me that there are just too few labels added to IstioEndpoint objects belonging to endpoints from remote cluster.
Here's a dump of IstioEndpoint.Labels of a local endpoint from inside this function. Basically every label from the workload is here and also the topology labels from underlying node are merged in correctly.
app.kubernetes.io/instance=httpbin
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=httpbin
app.kubernetes.io/version=0.0.0
helm.sh/chart=httpbin-0.0.0
istio.io/rev=stable
kubernetes.io/hostname=$hostname1
pod-template-hash=***
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=httpbin
service.istio.io/canonical-revision=0.0.0
topology.istio.io/cluster=$cluster1
topology.istio.io/network=$network1
topology.istio.io/subzone=$subzone1
topology.kubernetes.io/region=$region1
topology.kubernetes.io/zone=$zone1

And here's a dump of IstioEndpoint.Labels of an endpoint discovered from remote kube-apiserver from inside this function. There are just two istio.io labels. And if i try using failoverPriority with those, it works just fine.
topology.istio.io/cluster=$cluster2
topology.istio.io/network=$network2

And as an extra, here's a dump of IstioEndpoint.Labels of an endpoint of ServiceEntry with defined locality.
topology.istio.io/cluster=$cluster1
topology.istio.io/subzone=$subzone3
topology.kubernetes.io/region=$region2
topology.kubernetes.io/zone=$zone3

At the same time, the locality data is correct inside all the LocalityLbEndpoints objects. I guess that's why basic locality LB, failover and distribute methods work despite missing labels.
Looking at /debug/endpointShardz everything seems to be fine. All the labels are present here.
        "Kubernetes/$cluster1": [
          {
            "Labels": {
              "app.kubernetes.io/instance": "httpbin",
              "app.kubernetes.io/managed-by": "Helm",
              "app.kubernetes.io/name": "httpbin",
              "app.kubernetes.io/version": "0.0.0",
              "helm.sh/chart": "httpbin-0.0.0",
              "istio.io/rev": "stable",
              "kubernetes.io/hostname": "$hostname1",
              "pod-template-hash": "***",
              "security.istio.io/tlsMode": "istio",
              "service.istio.io/canonical-name": "httpbin",
              "service.istio.io/canonical-revision": "0.0.0",
              "topology.istio.io/cluster": "$network1",
              "topology.istio.io/network": "$network1",
              "topology.istio.io/subzone": "$subzone1",
              "topology.kubernetes.io/region": "$region1",
              "topology.kubernetes.io/zone": "$zone1"
            },
            "Address": "***",
            "ServicePortName": "http",
            "ServiceAccount": "spiffe://cluster.local/ns/***/sa/default",
            "Network": "$network1",
            "Locality": {
              "Label": "$region1/$cluster1/$subzone1",
              "ClusterID": "$cluster1"
            },
            "EndpointPort": 80,
            "LbWeight": 0,
            "TLSMode": "istio",
            "Namespace": "***",
            "WorkloadName": "httpbin",
            "HostName": "",
            "SubDomain": "",
            "HealthStatus": 1,
            "NodeName": "$hostname1"
          }
        ],
        "Kubernetes/$cluster2": [
          {
            "Labels": {
              "app.kubernetes.io/instance": "httpbin",
              "app.kubernetes.io/managed-by": "Helm",
              "app.kubernetes.io/name": "httpbin",
              "app.kubernetes.io/version": "0.0.0",
              "helm.sh/chart": "httpbin-0.0.0",
              "istio.io/rev": "stable",
              "kubernetes.io/hostname": "$hostname2",
              "pod-template-hash": "***",
              "security.istio.io/tlsMode": "istio",
              "service.istio.io/canonical-name": "httpbin",
              "service.istio.io/canonical-revision": "0.0.0",
              "topology.istio.io/cluster": "$cluster2",
              "topology.istio.io/network": "$network2",
              "topology.istio.io/subzone": "$subzone2",
              "topology.kubernetes.io/region": "$region1",
              "topology.kubernetes.io/zone": "$zone2"
            },
            "Address": "***",
            "ServicePortName": "http",
            "ServiceAccount": "spiffe://cluster.local/ns/***/sa/default",
            "Network": "$network2",
            "Locality": {
              "Label": "$region1/$zone2/$subzone2",
              "ClusterID": "$cluster2"
            },
            "EndpointPort": 80,
            "LbWeight": 0,
            "TLSMode": "istio",
            "Namespace": "***",
            "WorkloadName": "httpbin",
            "HostName": "",
            "SubDomain": "",
            "HealthStatus": 1,
            "NodeName": "$hostname2"

Is it possible that all the labels are somehow lost due to some east-west gateway related endpoint transformations? Or maybe there's some misconfiguration? Should there even be any more labels in this setup with east-west gateways?
I'd appreciate any help debugging this further.
Thanks!
Version
$ istioctl version
client version: 1.20.2
control plane version: 1.20.2
data plane version: 1.20.2 (11 proxies)
$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.6
Kustomize Version: v5.0.1
Server Version: v1.27.7
$ helm version --short
v3.12.2+g1e210a2

Additional Information
No response

I assume this is due to https://github.com/istio/istio/blob/954dcf909cb451095fdc06102e29da959a3ac5b4/pilot/pkg/xds/endpoints/ep_filters.go#L129-L136.
I don't really know what we can do here. The only info we know about the gateway is what IP and cluster its in, we don't know anything about the region, zone, etc. Region perhaps could be implied by the cluster (pretty weird to have a cluster span regions...) but that is all a bit heuristic
WDYT @stevenctl @hzxuzhonghu ?
Although I don't understand why failover works

@howardjohn, thanks! I think you're exactly right about reason.
Here's a quick hack i tested:
diff --git a/pilot/pkg/xds/endpoints/ep_filters.go b/pilot/pkg/xds/endpoints/ep_filters.go
index ce89e724d0..040fcf9fea 100644
--- a/pilot/pkg/xds/endpoints/ep_filters.go
+++ b/pilot/pkg/xds/endpoints/ep_filters.go
@@ -113,6 +113,33 @@ func (b *EndpointBuilder) EndpointsByNetworkFilter(endpoints []*LocalityEndpoint
                        splitWeightAmongGateways(weight, gateways, gatewayWeights)
                }

+               // Find common labels for all IstioEndpoints
+               commonIEpLabels := labels.Instance{}
+               if len(ep.istioEndpoints) > 0 {
+                       for k, v := range ep.istioEndpoints[0].Labels {
+                               labelIsCommon := true
+                               for i := 1; i < len(ep.istioEndpoints); i++ {
+                                       label, ok := ep.istioEndpoints[i].Labels[k]
+
+                                       if !ok {
+                                               labelIsCommon = false
+
+                                               break
+                                       }
+
+                                       if label != v {
+                                               labelIsCommon = false
+
+                                               break
+                                       }
+                               }
+
+                               if labelIsCommon {
+                                       commonIEpLabels[k] = v
+                               }
+                       }
+               }
+
                // Sort the gateways into an ordered list so that the generated endpoints are deterministic.
                gateways := maps.Keys(gatewayWeights)
                gateways = model.SortGateways(gateways)
@@ -132,7 +159,7 @@ func (b *EndpointBuilder) EndpointsByNetworkFilter(endpoints []*LocalityEndpoint
                                Locality: model.Locality{
                                        ClusterID: gw.Cluster,
                                },
-                               Labels: labelutil.AugmentLabels(nil, gw.Cluster, "", "", gw.Network),
+                               Labels: labelutil.AugmentLabels(commonIEpLabels, gw.Cluster, "", "", gw.Network),
                        }

                        // Generate the EDS endpoint for this gateway.

I also added another replica in remote cluster. Here are the dumps of IstioEndpoint.Labels again from inside this function.
Remote endpoint A:
app.kubernetes.io/instance=httpbin
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=httpbin
app.kubernetes.io/version=0.0.0
helm.sh/chart=httpbin-0.0.0
istio.io/rev=stable
kubernetes.io/hostname=$hostname1
pod-template-hash=***
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=httpbin
service.istio.io/canonical-revision=0.0.0
topology.istio.io/cluster=$cluster2
topology.istio.io/network=$network2
topology.istio.io/subzone=$zone2-$rack1
topology.kubernetes.io/region=$region1
topology.kubernetes.io/zone=$zone2

Remote endpoint B:
app.kubernetes.io/instance=httpbin
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=httpbin
app.kubernetes.io/version=0.0.0
helm.sh/chart=httpbin-0.0.0
istio.io/rev=stable
kubernetes.io/hostname=$hostname2
pod-template-hash=***
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=httpbin
service.istio.io/canonical-revision=0.0.0
topology.istio.io/cluster=$cluster2
topology.istio.io/network=$network2
topology.istio.io/subzone=$zone2-$rack2
topology.kubernetes.io/region=$region1
topology.kubernetes.io/zone=$zone2

Now, with this hack, failoverPriority works just fine on any of those labels.

Is it actually reasonable to passthrough some labels from discovered workloads to gateway endpoint? This is the most convenient option in my case. But this is probably wrong for different cases.
Alternatively, maybe there should be a way to pass those labels manually, for example through discovered k8s Service as was suggested, or something like that.

Does failover really work but failoverPriority doesn't? that doesn't seem to make sense.
Automatically using the common labels feels like it will be problematic outside if simple cases. Especially if we talk about arbitrary labels not well know ones

Does failover really work but failoverPriority doesn't? that doesn't seem to make sense.

Yes, it does work. Actually, locality looks totally "correct" outside of IstioEndpoint objects.
Here is how priority looks for a few endpoints from examples above.
DestinationRule looks like this, with just localityLbSetting.enabled: true and nothing else.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  annotations:
    meta.helm.sh/release-name: httpbin
    meta.helm.sh/release-namespace: ***
  labels:
    app.kubernetes.io/instance: httpbin
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: httpbin
    app.kubernetes.io/version: 0.0.0
    helm.sh/chart: httpbin-0.0.0
  name: httpbin
  namespace: ***
spec:
  host: httpbin.***.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
    outlierDetection:
      baseEjectionTime: 1m
      consecutive5xxErrors: 1
      interval: 15s

And here is proxy-config endpoints
 $ istioctl pc endpoints -n *** httpbin-5df8f77f4f-z55l5 --cluster 'outbound|80||httpbin.***.svc.cluster.local' --output json | grep locality -A 3 -B 1
                "weight": 1,
                "locality": {
                    "region": "$region1",
                    "zone": "$zone1",
                    "subZone": "$zone1-$rack1"
--
                "priority": 1,
                "locality": {
                    "region": "$region1",
                    "zone": "$zone2",
                    "subZone": "$zone2-$rack1"
--
                "priority": 2,
                "locality": {
                    "region": "$region2",
                    "zone": "$zone3",
                    "subZone": "$zone3-$rack1"

First endpoint with priority 0 is local, the one in the same cluster. Second endpoint with priority 1 is in the same region, but in a different zone. Third endpoint with priority 2 is a ServiceEntry with a made up different region.
Basically the only thing that doesn't work is failoverPriority. I guess because it depends on the labels, not the locality data.

Ah I get it now. Despite the gateway endpoints have the fake Locality set (https://github.com/istio/istio/blob/954dcf909cb451095fdc06102e29da959a3ac5b4/pilot/pkg/xds/endpoints/ep_filters.go#L129-L136), the failover code is reading the envoy Endpoint info which has locality set.
So I think what this is basically doing, and what we should do for failoverPriority is what @stevenctl mentioned: "When not homogeneous, expand into multiple addresses and our existing re-balancing calculation can maybe deal with this to some degree?"
That might be as simple as copying the labels over in https://github.com/istio/istio/blob/954dcf909cb451095fdc06102e29da959a3ac5b4/pilot/pkg/xds/endpoints/ep_filters.go#L130 but I am not sure, would need some testing.

Basically, logically if I have 10 remote network endpoints, we would have 10 endpoints in envoy, all which can have the original metadata. We happen to optimize this down to 1 endpoint with weight=10, but we could split it up (and already do with locality).
Note this does actually have a small flaw: I can have client in zone1, gateway in zone2, and server in zone1; the current/proposed approach would not consider that the gateway is in another zone. Solving that seems extremely challenging though

Basically, logically if I have 10 remote network endpoints, we would have 10 endpoints in envoy, all which can have the original metadata.

Here's a very rough PoC. The idea is, instead of aggregating weight on gateway endpoints, we create separate endpoints for everything. In the real world, this should probably be aggregated by matching label map or something.
diff --git a/pilot/pkg/xds/endpoints/ep_filters.go b/pilot/pkg/xds/endpoints/ep_filters.go
index ce89e724d0..297a78934f 100644
--- a/pilot/pkg/xds/endpoints/ep_filters.go
+++ b/pilot/pkg/xds/endpoints/ep_filters.go
@@ -27,7 +27,6 @@ import (
 	labelutil "istio.io/istio/pilot/pkg/serviceregistry/util/label"
 	"istio.io/istio/pkg/cluster"
 	"istio.io/istio/pkg/config/labels"
-	"istio.io/istio/pkg/maps"
 	"istio.io/istio/pkg/network"
 )

@@ -63,9 +62,6 @@ func (b *EndpointBuilder) EndpointsByNetworkFilter(endpoints []*LocalityEndpoint
 			},
 		}

-		// Create a map to keep track of the gateways used and their aggregate weights.
-		gatewayWeights := make(map[model.NetworkGateway]uint32)
-
 		// Process all the endpoints.
 		for i, lbEp := range ep.llbEndpoints.LbEndpoints {
 			istioEndpoint := ep.istioEndpoints[i]
@@ -109,53 +105,46 @@ func (b *EndpointBuilder) EndpointsByNetworkFilter(endpoints []*LocalityEndpoint
 				continue
 			}

-			// Apply the weight for this endpoint to the network gateways.
-			splitWeightAmongGateways(weight, gateways, gatewayWeights)
-		}
-
-		// Sort the gateways into an ordered list so that the generated endpoints are deterministic.
-		gateways := maps.Keys(gatewayWeights)
-		gateways = model.SortGateways(gateways)
-
-		// Create endpoints for the gateways.
-		for _, gw := range gateways {
-			epWeight := gatewayWeights[gw]
-			if epWeight == 0 {
+			// Create separate endpoint for each gateway and attach all the labels to it.
+			for _, gw := range gateways {
+				epWeight := weight / uint32(len(gateways))
+				if epWeight == 0 {
 				log.Warnf("gateway weight must be greater than 0, scaleFactor is %d", scaleFactor)
-				epWeight = 1
-			}
-			epAddr := util.BuildAddress(gw.Addr, gw.Port)
-
-			// Generate a fake IstioEndpoint to carry network and cluster information.
-			gwIstioEp := &model.IstioEndpoint{
-				Network: gw.Network,
-				Locality: model.Locality{
-					ClusterID: gw.Cluster,
-				},
-				Labels: labelutil.AugmentLabels(nil, gw.Cluster, "", "", gw.Network),
-			}
+					epWeight = 1
+				}
+				epAddr := util.BuildAddress(gw.Addr, gw.Port)
+
+				// Generate a fake IstioEndpoint to carry network and cluster information.
+				gwIstioEp := &model.IstioEndpoint{
+					Network: gw.Network,
+					Locality: model.Locality{
+						ClusterID: gw.Cluster,
+					},
+					Labels: labelutil.AugmentLabels(ep.istioEndpoints[i].Labels, gw.Cluster, "", "", gw.Network),
+				}

-			// Generate the EDS endpoint for this gateway.
-			gwEp := &endpoint.LbEndpoint{
-				HostIdentifier: &endpoint.LbEndpoint_Endpoint{
-					Endpoint: &endpoint.Endpoint{
-						Address: epAddr,
+				// Generate the EDS endpoint for this gateway.
+				gwEp := &endpoint.LbEndpoint{
+					HostIdentifier: &endpoint.LbEndpoint_Endpoint{
+						Endpoint: &endpoint.Endpoint{
+							Address: epAddr,
+						},
+					},
+					LoadBalancingWeight: &wrappers.UInt32Value{
+						Value: epWeight,
 					},
-				},
-				LoadBalancingWeight: &wrappers.UInt32Value{
-					Value: epWeight,
-				},
-				Metadata: &core.Metadata{},
+					Metadata: &core.Metadata{},
+				}
+				// TODO: figure out a way to extract locality data from the gateway public endpoints in meshNetworks
+				util.AppendLbEndpointMetadata(&model.EndpointMetadata{
+					Network:   gw.Network,
+					TLSMode:   model.IstioMutualTLSModeLabel,
+					ClusterID: b.clusterID,
+					Labels:    labels.Instance{},
+				}, gwEp.Metadata)
+				// Currently gateway endpoint does not support tunnel.
+				lbEndpoints.append(gwIstioEp, gwEp)
 			}
-			// TODO: figure out a way to extract locality data from the gateway public endpoints in meshNetworks
-			util.AppendLbEndpointMetadata(&model.EndpointMetadata{
-				Network:   gw.Network,
-				TLSMode:   model.IstioMutualTLSModeLabel,
-				ClusterID: b.clusterID,
-				Labels:    labels.Instance{},
-			}, gwEp.Metadata)
-			// Currently gateway endpoint does not support tunnel.
-			lbEndpoints.append(gwIstioEp, gwEp)
 		}

 		// Endpoint members could be stripped or aggregated by network. Adjust weight value here.

This kinda works on the Istio side. However, i hit a roadblock from the envoy side :disappointed: Envoy deduplicates endpoints with the same address received from EDS: https://github.com/envoyproxy/envoy/issues/22417. Because of this, envoy throws away all endpoints with duplicate address of the east-west gateway except the first one in every cluster.
Here's /debug/edsz of my service using the patch. As you can see, there are two separate endpoints with different metadata for a single east-west gateway, because there are two pods in the remote cluster. This looks perfect in theory.
{
  "clusterName": "outbound|80||httpbin.***.svc.cluster.local",
  "endpoints": [
    {
      "locality": {
        "region": "$region1",
        "zone": "$zone1",
        "subZone": "$zone1-$rack1"
      },
      "lbEndpoints": [
        {
          "endpoint": {
            "address": {
              "socketAddress": {
                "address": "$pod_addr",
                "portValue": 80
              }
            }
          },
          "healthStatus": "HEALTHY",
          "metadata": {
            "filterMetadata": {
              "envoy.transport_socket_match": {
                "tlsMode": "istio"
              },
              "istio": {
                "workload": "httpbin;***;httpbin;0.0.0;$cluster1"
              }
            }
          },
          "loadBalancingWeight": 1
        }
      ],
      "loadBalancingWeight": 1
    },
    {
      "locality": {
        "region": "$region1",
        "zone": "$zone2",
        "subZone": "$zone2-$rack1"
      },
      "lbEndpoints": [
        {
          "endpoint": {
            "address": {
              "socketAddress": {
                "address": "$gw_address",
                "portValue": 15443
              }
            }
          },
          "metadata": {
            "filterMetadata": {
              "envoy.transport_socket_match": {
                "tlsMode": "istio"
              },
              "istio": {
                "workload": ";;;;$cluster1"
              }
            }
          },
          "loadBalancingWeight": 1
        }
      ],
      "loadBalancingWeight": 1
    },
    {
      "locality": {
        "region": "$region1",
        "zone": "$zone2",
        "subZone": "$zone2-$rack2"
      },
      "lbEndpoints": [
        {
          "endpoint": {
            "address": {
              "socketAddress": {
                "address": "$gw_address",
                "portValue": 15443
              }
            }
          },
          "metadata": {
            "filterMetadata": {
              "envoy.transport_socket_match": {
                "tlsMode": "istio"
              },
              "istio": {
                "workload": ";;;;$cluster1"
              }
            }
          },
          "loadBalancingWeight": 1
        }
      ],
      "loadBalancingWeight": 1
    }
  ]
},

However, here's endpoints dump from the envoy. As you can see, envoy just quietly dropped second endpoint.
$ istioctl pc endpoints -n *** httpbin-5df8f77f4f-z55l5 --cluster 'outbound|80||httpbin.***.svc.cluster.local' --output json
[
    {
        "name": "outbound|80||httpbin.***.svc.cluster.local",
        "addedViaApi": true,
        "hostStatuses": [
            {
                "address": {
                    "socketAddress": {
                        "address": "$pod_addr",
                        "portValue": 80
                    }
                },
                "stats": [
                    ...
                ],
                "healthStatus": {
                    "edsHealthStatus": "HEALTHY"
                },
                "weight": 1,
                "locality": {
                    "region": "$region1",
                    "zone": "$zone1",
                    "subZone": "$zone1-$rack1"
                }
            },
            {
                "address": {
                    "socketAddress": {
                        "address": "$gw_addr",
                        "portValue": 15443
                    }
                },
                "stats": [
                    ...
                ],
                "healthStatus": {
                    "edsHealthStatus": "HEALTHY"
                },
                "weight": 1,
                "locality": {
                    "region": "$region1",
                    "zone": "$zone2",
                    "subZone": "$zone2-$rack1"
                }
            }
        ],
        "circuitBreakers": {
            "thresholds": [
                ...
            ]
        },
        "observabilityName": "outbound|80||httpbin.***.svc.cluster.local",
        "edsServiceName": "outbound|80||httpbin.***.svc.cluster.local"
    }
]

This approach seems impossible to implement without that envoy issue being resolved. And judging by this comment https://github.com/envoyproxy/envoy/issues/22417#issuecomment-1353471480, it may not be a simple issue from envoy's point of view :disappointed:

I think we could implement this in Istio by apply the locality LB config then deduping. Instead of deduping then applying locality LB config

I think we could implement this in Istio by apply the locality LB config then deduping. Instead of deduping then applying locality LB config

I think this will only cover the simplest cases.
Let's say you have two k8s clusters inside one region.
Cluster A is in zone A. Zone A includes subzone A.
Cluster B is in zone B. Zone B includes subzones B and C.
Let's say you want to failover from cluster A to cluster B. Because of envoy limitation, you can only have a single endpoint for east-west gateway. That means you can't distinguish between subzones B and C, since single endpoint can't belong to both, only to neither. You can only use labels that are common for every workload, which is ok i guess.
But, the idea with separate endpoint for every label set combination is way cooler :slightly_smiling_face: Then you could use any set of labels for cross-network failover, not even just topology related ones. Like in the example in documentation for failoverPriority, you could use version label or anything else.

Does https://github.com/envoyproxy/envoy/issues/22417 apply even if the endpoint is in a different priority level?
What I was trying to express would allow 'the idea with separate endpoint for every label set combination is way cooler'.

Does https://github.com/envoyproxy/envoy/issues/22417 apply even if the endpoint is in a different priority level?

Just checked and yes it still applies. Even if the priority level is different only one endpoint remains.
Looking at the code it seems that only address field matters, in our case there are just address.socketAddress.address and address.socketAddress.portValue.

Just checked and yes it still applies. Even if the priority level is different only one endpoint remains.

To clarify, envoy keeps just the first endpoint from the list of "duplicates". Even if the following "duplicate" endpoints have numerically lower priority.

Is there any progress?
It seems that until this issue is resolved, the remote cluster cannot use multi-zone in Multi-Primary on different networks.

authors: akakakakakaa howardjohn yshch


text:
Istio is Truncating (cut off ) Json Response Body when curl is executed

we have multi container pod.
php-fpm
nginx
Without istio, things are working and curl gives full json response body. But when we enable istio and execute curl request, response body becomes truncated and gives partial response.
First,  I used nginx:1.25.2 and enable debug logs on this container, I saw a log when response is truncated.
upstream sent more data than specified in "content-length" header while reading upstream

this error is gone when response is full and not truncated.  Some of the curl request gives may be 4/100 gives full response.
Then I changed the nginx image to nginx:1.15.2.
Still gives me the partial response but with log.
2024/04/24 19:44:58 [info] 7#7: *594 client 127.0.0.6 closed keepalive connection

In case of full response this log is not present.
nginx configuration:
    upstream api-backend {
        least_conn;
        server 127.0.0.1:9002 max_fails=15 fail_timeout=45s weight=2;
        keepalive 80;
    }
    server {
            error_log /var/log/nginx/error.log debug;
            listen 80 default_server;
            listen [::]:80 default_server;
            server_name _;

            keepalive_timeout 600s;
            client_body_timeout 600s;
            client_header_timeout 600s;

            # Set nginx to serve files from the shared volume!
            set $static_root /var/www/hub-api/public;
            root $static_root;
            index index.php index.html index.htm index.nginx-debian.html;

            include /etc/lb-config/site.d/headers_api.conf;
            include /etc/lb-config/site.d/error_page.conf;

            include /etc/lb-config/site.d/extra_protect.conf;
            # static content
            include /etc/lb-config/site.d/static_location.conf;

            # The minimum length of the response to be compress
            gzip_min_length  2056;

            # Disable PHP execution in var and media
            location /var { location ~ \.php$ {return 403;} }

            ## These locations are protected
            location ~ /(app|includes|pkginfo|var|composer.json|errors/local.xml)/ {
                deny all;
            }

            ## Bots trap from nginx.conf

            ## Main @location
            location / {
                index index.php index.html index.htm;
                try_files $uri $uri/ @rewrite;
            }

            location @rewrite {
                rewrite / /index.php?$args;
            }

            set $fastcgi_https "";

            if ($http_x_forwarded_proto = "https") {
                set $fastcgi_https "on";
            }

            # include /etc/lb-config/site.d/lb_opt.conf;

            proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
            proxy_next_upstream_timeout 600s;
            proxy_read_timeout 600s;
            proxy_connect_timeout 600s;
            ###
            fastcgi_keep_conn on;
            fastcgi_next_upstream error timeout http_500  http_503;
            fastcgi_next_upstream_timeout 5s;
            fastcgi_connect_timeout 10;
            fastcgi_ignore_client_abort on;
            #fastcgi_send_timeout 180;
            #fastcgi_read_timeout 180;

            # Adjust buffering settings
            fastcgi_buffering off;  # Disable buffering

            fastcgi_buffer_size 64k;  # Adjust buffer size as needed
            fastcgi_buffers 4 64k;    # Adjust number of buffers

            ## Execute PHP scripts
            location ~ \.php$ {
                include /etc/lb-config/site.d/fastcgi_params.conf;

                fastcgi_pass   api-backend;
                fastcgi_param  HTTPS       $fastcgi_https;
                fastcgi_param  SCRIPT_FILENAME $document_root/$fastcgi_script_name;
            }

     }

Partial Response:
{
    "airline": [
        {
            "code": "ET",
            "label": {
                "en": "Ethiopian Airlines",
                "ar":"الخطوط الجو\

Full response:
{
    "airline": [
        {
            "code": "ET",
            "label": {
                "en": "Ethiopian Airlines",
                "ar": "الخطوط الجوية الأثيوبية"
            }
        }
    ],
    "airport": []
}

With istio enabled sometimes I get the full response but usually response body is truncated.
we do not have istio gateway, no virtual service and destination, we just enabled the sidecar injection.
istio version:
client version: 1.18.2
control plane version: 1.18.2
data plane version: 1.18.2 (559 proxies)

EKS version:
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.7-eks-b9c9ed7

1.18 is EOL for a while, please try with latest version, it maybe fixed in envoy.


text:
[perf swat] produce profiling data

one of the items for
https://docs.google.com/document/d/1TiMmzbM5r9QlBfs79owBqiAZJgflMyXa3Zuev7xZKSc/edit#
is to rerun the scenarios to produce profiling information for the fortio max ingress -> s1 (mixer call not cached case)
e.g https://fortio.istio.io/?xMin=&xMax=&xLog=false&yLog=false&sel=qps_max-ingress_to_s1-0.7.1-2018-04-05-22-04
as a first step

Pinging to keep alive as this is still valid.
On Thu, Jul 19, 2018 at 4:36 PM stale[bot] notifications@github.com wrote:

This issue has been automatically marked as stale because it has not had
activity in the last 30 days. It will be closed in the next 2 weeks unless
it is tagged "help wanted" or other activity occurs. Thank you for your
contributions.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/6409#issuecomment-406445421, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIoKPOrH6YPgiByQytelIBBjo06gFEVPks5uIRf5gaJpZM4UuSp2
.

authors: ldemailly louiscryan


text:
[galley] Flaky test: TestFsSource DeletePartResorceInFile

https://k8s-gubernator.appspot.com/build/istio-prow/pull/istio_istio/8953/istio-unit-tests/14005

@irisdingbj could you take a look?


text:
Proposal: Support short names for Istio resources

Describe the feature request
There are many resources defined in Istio such as VirtualService, ServiceEntry, DestinationRule and so on.
They don't have short names, so it is not convenient to operate them through kubectl.
Describe alternatives you've considered
I think it is necessary to support short names for frequently used resources.
Additional context
xref: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/istio-dev/_OX4WW_KzM4/c7mR8ixpCwAJ

/cc @andraxylia
/kind enhancement

This should be simple, adding shortNames entries to install/kubernetes/helm/istio-init/files/crd-10.yaml

anything left to be done here or should this be closed?

authors: duderino esnible supereagle


text:
Initial istioctl integration test in new framework

A simple test of istioctl version with --remote=true option.
There were no istioctl-specific integration tests.
Please review my usage of the new test framework because this is my first time using it.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: esnible
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: nmittler
If they are not already assigned, you can assign the PR to them by writing /assign @nmittler in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

istioctl/OWNERS [esnible]
tests/integration/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@esnible: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test istio-integ-local-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test istio-integ-local-tests

prow/istio-integ-k8s-tests.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test istio-integ-k8s-tests

prow/release-test.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test release-test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test istio-integ-k8s-tests

prow/release-test.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test release-test

prow/istio-integ-local-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-local-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test istio-integ-k8s-tests

prow/release-test.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test release-test

prow/istio-integ-local-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-local-tests

prow/e2e-bookInfoTests-v1alpha3.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/release-test.sh
fcca61ea57de4c185999b4b0354bf376377378ff
link
/test release-test

prow/istio-integ-local-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-local-tests

prow/e2e-bookInfoTests-v1alpha3.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-integ-k8s-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-local-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-local-tests

prow/e2e-bookInfoTests-v1alpha3.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-integ-k8s-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-k8s-tests

prow/release-test.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test release-test

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-bookInfoTests-v1alpha3.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test e2e-bookInfoTests-envoyv2-v1alpha3

prow/istio-integ-k8s-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-k8s-tests

prow/release-test.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test release-test

prow/integ-pilot-k8s-presubmit-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test integ-pilot-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
9e2b263d58b2f3a5ae5c5b19d29f2bbf2d8fa836
link
/test istio-integ-k8s-tests

prow/integ-pilot-k8s-presubmit-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test integ-pilot-k8s-presubmit-tests-master

prow/integ-security-k8s-presubmit-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test integ-security-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-pilot-k8s-presubmit-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test integ-pilot-k8s-presubmit-tests-master

prow/integ-security-k8s-presubmit-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test integ-security-k8s-presubmit-tests-master

prow/istio-integ-k8s-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test istio-integ-k8s-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ozevren I have made the changes.  Hoping for a review.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: esnible
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: ozevren
If they are not already assigned, you can assign the PR to them by writing /assign @ozevren in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

istioctl/OWNERS [esnible]
pkg/OWNERS
tests/integration/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-security-k8s-presubmit-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test integ-security-k8s-presubmit-tests-master

prow/istio-integ-k8s-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test istio-integ-k8s-tests-master

prow/integ-mixer-k8s-presubmit-tests.sh
861dbaa0aa21cb7e5c8179450e160f6d5d00731c
link
/test integ-mixer-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test istio-integ-k8s-tests-master

prow/integ-mixer-k8s-presubmit-tests.sh
861dbaa0aa21cb7e5c8179450e160f6d5d00731c
link
/test integ-mixer-k8s-presubmit-tests-master

prow/e2e-mixer-no_auth.sh
160fa833825d2ab3cbad31bdaada78fb1064fe2c
link
/test e2e-mixer-no_auth-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@esnible: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
ff79bc2fda4c60830348186b861d7097cd9b9fb6
link
/test istio-integ-k8s-tests-master

prow/e2e-mixer-no_auth.sh
160fa833825d2ab3cbad31bdaada78fb1064fe2c
link
/test e2e-mixer-no_auth-master

prow/integ-mixer-k8s-presubmit-tests.sh
160fa833825d2ab3cbad31bdaada78fb1064fe2c
link
/test integ-mixer-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/lgtm
/approve

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: esnible, ozevren
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

istioctl/OWNERS [esnible,ozevren]
pkg/OWNERS [ozevren]
tests/integration/OWNERS [ozevren]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

source: gharchive/pull-request

authors: esnible istio-testing ozevren


text:
Move validation webhook to shared informers

This is a perf optimization to minimize the number of watches we open to the api server.
Part of #24640

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

analyze-tests_istio
99af8db6152a6419e8f5c63ab4cedf0ff77b7bef
link
/test analyze-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

authors: howardjohn istio-testing