text:
Headless services - pod fails mTLS handshake with itself

Describe the bug
With mTLS auth enabled, when using <dashed-pod-ip>.my-headless-service.my-namespace.svc.cluster.local endpoints (provided a matching ServiceEntry) - pods successfully make HTTP calls to all pods except themselves.
This appears to be the root cause behind some of our headless services problems.
It is a repeating issue with many gossip-based clusters - e.g. akka, kafka, cassandra.
The provided reproduction does not use StatefulSet to keep it simple, but the same principal applies to all workloads using headless services.
NOTE: there are some open issues around the headless-service topic, but they are either stale or broader than this exact issue.
Expected behavior
Pods should be able to call themselves on their headless-generated endpoint.
Steps to reproduce the bug

Install Istio with auth enabled (istio-demo-auth.yaml is fine for this) and run:
kubectl label ns default istio-injection=enabled

Create {ServiceAccount, Service[headless], ServiceEntry, Deployment}:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: curly
  labels:
    app: curly
automountServiceAccountToken: false
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: curly
  name: curly-headless
spec:
  ports:
  - name: http
    port: 8855
    protocol: TCP
  selector:
    app: curly
  clusterIP: None
  publishNotReadyAddresses: true
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: curly
spec:
  hosts:
  - "*.curly-headless.default.svc.cluster.local"
  location: MESH_INTERNAL
  ports:
  - number: 8855
    name: http
    protocol: HTTP
  resolution: NONE
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: curly
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: curly
    spec:
      serviceAccountName: curly
      containers:
      - name: curly
        image: tutum/curl
        command: ["python3", "-m", "http.server", "8855"]
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8855

Run kubectl get pods -o wide and wait for the pods to become ready:

NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE
curly-75649cb6f8-n96tv   2/2     Running   0          1m    172.17.0.17   minikube   <none>
curly-75649cb6f8-qdh77   2/2     Running   0          1m    172.17.0.6    minikube   <none>

Run curl from POD1 to POD2:
In this example, POD1's generated name would be 172-17-0-17.curly-headless.default.svc.cluster.local (more: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#srv-records)

POD1=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].metadata.name}') && \
POD2_DASHED_IP=$(kubectl get pods -l app=curly -o jsonpath='{.items[1].status.podIP}' | sed 's/\./-/g') && \
kubectl exec $POD1 -c curly -- curl -s -I ${POD2_DASHED_IP}.curly-headless:8855

Output should be something like:
HTTP/1.1 200 OK
server: envoy
date: Sun, 17 Mar 2019 07:58:20 GMT
content-type: text/html; charset=ascii
content-length: 987
x-envoy-upstream-service-time: 1

Run curl from POD1 to POD1:

POD1=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].metadata.name}') && \
POD1_DASHED_IP=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].status.podIP}' | sed 's/\./-/g') && \
kubectl exec $POD1 -c curly -- curl -v -s -I ${POD1_DASHED_IP}.curly-headless:8855

Output:
* Rebuilt URL to: 172-17-0-17.curly-headless:8855/
* Hostname was NOT found in DNS cache
*   Trying 172.17.0.17...
* Connected to 172-17-0-17.curly-headless (172.17.0.17) port 8855 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 172-17-0-17.curly-headless:8855
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
command terminated with exit code 56

Some attempt can be made to work around the problem with a Policy that make this specific port mTLS permissive, but then RBAC stops working (as there is no auth info in the call to itself).

When setting connection logging to debug, I see the following in my setup:

# this is pod1->pod2

[2019-03-17 07:32:01.755][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:639] [C41] connecting to 172.17.0.6:8855
[2019-03-17 07:32:01.755][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:648] [C41] connection in progress
[2019-03-17 07:32:01.756][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:517] [C41] connected
[2019-03-17 07:32:01.756][24][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:138] [C41] handshake error: 2
[2019-03-17 07:32:01.760][24][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:127] [C41] handshake complete
[2019-03-17 07:32:01.765][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:502] [C40] remote close
[2019-03-17 07:32:01.766][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:183] [C40] closing socket: 0
[2019-03-17T07:32:01.754Z] "HEAD / HTTP/1.1" 200 - "-" 0 0 10 9 "-" "curl/7.35.0" "0a079597-58fe-9be3-adbc-a0abe10d67cb" "172-17-0-6.curly-headless.alice-dev.svc.cluster.local:8855" "172.17.0.6:8855" outbound|8855||*.curly-headless.alice-dev.svc.cluster.local - 172.17.0.6:8855 172.17.0.17:42332 -

# this is pod1->pod1

[2019-03-17 07:32:10.567][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:101] [C41] closing data_to_write=0 type=1
[2019-03-17 07:32:10.567][24][debug][connection] [external/envoy/source/common/network/connection_impl.cc:183] [C41] closing socket: 1
[2019-03-17 07:32:10.568][24][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:236] [C41] SSL shutdown: rc=0
[2019-03-17 07:32:10.673][25][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:138] [C42] handshake error: 1
[2019-03-17 07:32:10.673][25][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:166] [C42] SSL error: 268435612:SSL routines:OPENSSL_internal:HTTP_REQUEST
[2019-03-17 07:32:10.673][25][debug][connection] [external/envoy/source/common/network/connection_impl.cc:183] [C42] closing socket: 0

Version
[~]$ istioctl version --remote
client version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-rc.5-5-g82797c0"}
citadel version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e-dirty", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-rc.5-5-g82797c0"}
egressgateway version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-rc.5-5-g82797c0"}
galley version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e-dirty", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-rc.5-5-g82797c0"}
ingressgateway version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-rc.5-5-g82797c0"}
pilot version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e-dirty", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-rc.5-5-g82797c0"}
policy version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e-dirty", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-rc.5-5-g82797c0"}
sidecar-injector version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e-dirty", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-rc.5-5-g82797c0"}
telemetry version: version.BuildInfo{Version:"1.1.0-rc.6", GitRevision:"82797c0c0649a3f73029b33957ae105260458c6e-dirty", User:"root", Host:"22373299-4805-11e9-8dad-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-rc.5-5-g82797c0"}
[~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.8", GitCommit:"4e209c9383fa00631d124c8adcc011d617339b3c", GitTreeState:"clean", BuildDate:"2019-02-28T18:40:05Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Installation
"Quick Start Evaluation Install" with istio-demo-auth.yaml and Minikube.
The same issue appears in a real deployment on AWS with kops (same versions).
Environment
Minikube / AWS+kops
Cluster State
Some config dumps for POD1:
# relevant clusters

inbound|8855|http|curly-headless.default.svc.cluster.local::default_priority::max_connections::1024
inbound|8855|http|curly-headless.default.svc.cluster.local::default_priority::max_pending_requests::1024
inbound|8855|http|curly-headless.default.svc.cluster.local::default_priority::max_requests::1024
inbound|8855|http|curly-headless.default.svc.cluster.local::default_priority::max_retries::3
inbound|8855|http|curly-headless.default.svc.cluster.local::high_priority::max_connections::1024
inbound|8855|http|curly-headless.default.svc.cluster.local::high_priority::max_pending_requests::1024
inbound|8855|http|curly-headless.default.svc.cluster.local::high_priority::max_requests::1024
inbound|8855|http|curly-headless.default.svc.cluster.local::high_priority::max_retries::3
inbound|8855|http|curly-headless.default.svc.cluster.local::added_via_api::true
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::cx_active::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::cx_connect_fail::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::cx_total::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::rq_active::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::rq_error::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::rq_success::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::rq_timeout::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::rq_total::0
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::health_flags::healthy
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::weight::1
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::region::
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::zone::
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::sub_zone::
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::canary::false
inbound|8855|http|curly-headless.default.svc.cluster.local::127.0.0.1:8855::success_rate::-1

outbound|8855||*.curly-headless.default.svc.cluster.local::default_priority::max_connections::1024
outbound|8855||*.curly-headless.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|8855||*.curly-headless.default.svc.cluster.local::default_priority::max_requests::1024
outbound|8855||*.curly-headless.default.svc.cluster.local::default_priority::max_retries::1024
outbound|8855||*.curly-headless.default.svc.cluster.local::high_priority::max_connections::1024
outbound|8855||*.curly-headless.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|8855||*.curly-headless.default.svc.cluster.local::high_priority::max_requests::1024
outbound|8855||*.curly-headless.default.svc.cluster.local::high_priority::max_retries::3
outbound|8855||*.curly-headless.default.svc.cluster.local::added_via_api::true

outbound|8855||curly-headless.default.svc.cluster.local::default_priority::max_connections::1024
outbound|8855||curly-headless.default.svc.cluster.local::default_priority::max_pending_requests::1024
outbound|8855||curly-headless.default.svc.cluster.local::default_priority::max_requests::1024
outbound|8855||curly-headless.default.svc.cluster.local::default_priority::max_retries::1024
outbound|8855||curly-headless.default.svc.cluster.local::high_priority::max_connections::1024
outbound|8855||curly-headless.default.svc.cluster.local::high_priority::max_pending_requests::1024
outbound|8855||curly-headless.default.svc.cluster.local::high_priority::max_requests::1024
outbound|8855||curly-headless.default.svc.cluster.local::high_priority::max_retries::3
outbound|8855||curly-headless.default.svc.cluster.local::added_via_api::true

# all listeners

[
  "0.0.0.0:15090",
  "172.17.0.17:8855",
  "10.103.154.232:15029",
  "10.103.154.232:15030",
  "10.96.0.10:53",
  "10.97.245.106:14267",
  "10.98.146.122:16686",
  "10.106.130.238:443",
  "10.103.154.232:15031",
  "10.109.154.127:15011",
  "10.97.155.171:443",
  "10.103.154.232:15443",
  "10.107.110.66:42422",
  "10.96.0.1:443",
  "10.106.130.238:15443",
  "10.96.99.165:443",
  "10.103.154.232:15032",
  "10.103.154.232:443",
  "10.103.154.232:15020",
  "10.103.154.232:31400",
  "10.97.245.106:14268",
  "0.0.0.0:20001",
  "0.0.0.0:9411",
  "0.0.0.0:8855",
  "0.0.0.0:80",
  "0.0.0.0:9901",
  "0.0.0.0:8080",
  "0.0.0.0:15014",
  "0.0.0.0:15010",
  "0.0.0.0:15004",
  "0.0.0.0:9090",
  "0.0.0.0:3000",
  "0.0.0.0:8060",
  "0.0.0.0:9091",
  "172.17.0.17:15020",
  "0.0.0.0:15001"
]

Full config dump json from POD1:
https://pastebin.com/raw/TrsjmKJD

I am also seeing this issue.  I noticed that POD1->POD2 traffic does get the Envoy access log showing the "outbound|8855||*.curly-headless.alice-dev.svc.cluster.local" cluster was matched, but POD1->POD1 traffic never does.  I believe it works like this:

The iptables rules redirect the POD1->POD1 traffic to Envoy's listening port.
Envoy accepts the connection and uses SO_ORIGINAL_DST to figure out the connection was destined for POD1 (172.17.0.17:8855 in your example)
**Envoy matches the 172.17.0.17:8855 listener, not the 0.0.0.0:8855 listener. **
172.17.0.17:8855 is the "inbound" listener, not the outbound listener.
Envoy applies the inbound policy requiring mTLS, but nothing applied the outbound policy (of being the mTLS client).
The connection is dropped.

For POD1->POD2, the story is different:

The iptables rules redirect the POD1->POD2 traffic to Envoy's listening port.
Envoy accepts the connection and uses SO_ORIGINAL_DST to figure out the connection was destined for POD1 (172.17.0.6:8855 in your example)
Envoy matches the 0.0.0.0:8855 listener.
172.17.0.17:8855 is the "outbound" listener.
Envoy applies the outbound policy of being the mTLS client and sends it over the network.
The Envoy in POD2 catches the connection, maps to its inbound policy and does the server-side of the mTLS connection.
Everyone is happy.

If you get the logs from the "proxy-init" sidecar you can see the IPTables rules installed and step through them.
Workaround: My hacky temporary workaround is to "sed" /etc/hosts so that the pod's own name maps to 127.0.0.1 instead of 172.17.0.6; this causes loopback connections to skip Envoy/Istio entirely.  This makes kafka work-for-me but is bad because now I'm not going through Istio, so all the Istio policy doesn't apply.  In this case, I was using it just for mTLS, so I'm ok with not using mTLS for literally loopback connections.  But it's still not a generally good solution.
I think the root problem here is that "Pods that connect to their own IPs sometimes only go through inbound policy instead of outbound+inbound policy"

btw I would be happy to help out with a fix but I couldn't think of an easy one (not having thought very hard).  These were my rough thoughts:

Maybe a listenerFilter in Envoy could be used to say "If the source is us, then force this through the Outbound and Inbound policy  (https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/listener/listener.proto#listener-filterchainmatch - source_type?).  Have to not make a loop here.

Make Envoy have two lists of listeners: one are "outbound" listeners, one are "inbound" listeners.  They don't overlap, ever.  IPTables and Envoy conspire so that Envoy can classify outbound traffic and only match it to outbound listeners, inbound traffic to inbound listeners.
2a) Run two Envoy proxies in the sidecar - one on 15000 (inbound), one on 15001 (outbound).  Bad resource utilization but would work.
2b) Can Envoy have two separate listener lists, one that actually listens on 15000 and one that actually listens on 15001?  I didn't see that option.
2c) 2b, but maybe using iptables packet marking instead of two ports.

I'm hoping there's something easier here.

Change iptables rules so that POD1->POD1 traffic skips Istio/Envoy if it is using POD1's Pod IP 172.17.0.17 (just like how it skips if it is 127.0.0.1)

$ kubectl logs kafka-0 -c istio-init
...
# Completed on Wed Mar  6 17:49:47 2019
# Generated by iptables-save v1.6.0 on Wed Mar  6 17:49:47 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 9094 -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN     # <---- add a rule: -d 172.17.0.17 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT

Disadvantage here is that if POD1 calls POD1 on the pod IP, maybe it should go through Envoy/Istio.  Comes down to behavior and expectations but at first pass, I think I would want metrics, traces, rate limiting etc. on this connection.

@andrewjjenkins thanks for looking into this.
I think that direct self-connections are going to almost always be part of a gossip protocol using the "bound" hostname as an endpoint.
For example, the initial use case that brought us to this issue was akka-cluster-bootstrap - the coordinator resolves SRV records using the headless service and calls them (there's no logic branching dealing with 'is this hostname actually me?').
For these types of self-connections, I would guess metrics and traces are pretty much spam.
Out-of-process rate limiting (and fault tolerance) on my own instance are also things that would personally surprise me as a client application relying or implementing gossip protocols.
Solution 3 seems good to me, perhaps enabled by a flag to istio-init.
IMO it's even worth considering as the default behavior.

Just tried it (in a hacky way) - I think there's an ordering issue, or I'm missing something.
Changed the injector template to this:
@@ -14161,6 +14161,11 @@ data:
       - name: istio-init
         image: "docker.io/istio/proxy_init:1.1.0-rc.6"
+        env:
+        - name: INSTANCE_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
         args:
         - "-p"
         - [[ .MeshConfig.ProxyListenPort ]]
@@ -14171,7 +14176,7 @@ data:
         - "-i"
         - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges`  "*"  ]]"
         - "-x"
-        - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges`  ""  ]]"
+        - $(INSTANCE_IP)/32

That produces this istio-init log:
Environment:
------------
ENVOY_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
INBOUND_CAPTURE_PORT=15001
PROXY_UID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
+ iptables -t nat -N ISTIO_REDIRECT
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=8855
INBOUND_PORTS_EXCLUDE=15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=172.17.0.21/32
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=

+ iptables -t nat -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001
+ iptables -t nat -N ISTIO_IN_REDIRECT
+ iptables -t nat -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15001
+ '[' -n 8855 ']'
+ '[' REDIRECT = TPROXY ']'
+ table=nat
+ iptables -t nat -N ISTIO_INBOUND
+ iptables -t nat -A PREROUTING -p tcp -j ISTIO_INBOUND
+ '[' 8855 == '*' ']'
+ for port in '${INBOUND_PORTS_INCLUDE}'
+ '[' REDIRECT = TPROXY ']'
+ iptables -t nat -A ISTIO_INBOUND -p tcp --dport 8855 -j ISTIO_IN_REDIRECT
+ iptables -t nat -N ISTIO_OUTPUT
+ iptables -t nat -A OUTPUT -p tcp -j ISTIO_OUTPUT
+ '[' -z '' ']'
+ iptables -t nat -A ISTIO_OUTPUT -o lo '!' -d 127.0.0.1/32 -j ISTIO_REDIRECT
+ for uid in '${PROXY_UID}'
+ iptables -t nat -A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
+ for gid in '${PROXY_GID}'
+ iptables -t nat -A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
+ iptables -t nat -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
+ '[' -n 172.17.0.21/32 ']'
+ for cidr in '${OUTBOUND_IP_RANGES_EXCLUDE}'
+ iptables -t nat -A ISTIO_OUTPUT -d 172.17.0.21/32 -j RETURN
+ '[' '*' == '*' ']'
+ iptables -t nat -A ISTIO_OUTPUT -j ISTIO_REDIRECT
+ set +o nounset
+ '[' -n '' ']'
+ ip6tables -F INPUT
+ ip6tables -A INPUT -m state --state ESTABLISHED -j ACCEPT
+ ip6tables -A INPUT -i lo -d ::1 -j ACCEPT
+ ip6tables -A INPUT -j REJECT
+ dump
+ iptables-save
# Generated by iptables-save v1.6.0 on Tue Mar 19 16:54:24 2019
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Tue Mar 19 16:54:24 2019
# Generated by iptables-save v1.6.0 on Tue Mar 19 16:54:24 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 8855 -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -d 172.17.0.21/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Tue Mar 19 16:54:24 2019
+ ip6tables-save
# Generated by ip6tables-save v1.6.0 on Tue Mar 19 16:54:24 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED -j ACCEPT
-A INPUT -d ::1/128 -i lo -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-port-unreachable
COMMIT
# Completed on Tue Mar 19 16:54:24 2019

But still:
[~]$ POD1=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].metadata.name}') && \
> POD1_DASHED_IP=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].status.podIP}' | sed 's/\./-/g') && \
> kubectl exec $POD1 -c curly -- curl -v -s -I ${POD1_DASHED_IP}.curly-headless:8855
* Rebuilt URL to: 172-17-0-21.curly-headless:8855/
* Hostname was NOT found in DNS cache
*   Trying 172.17.0.21...
* Connected to 172-17-0-21.curly-headless (172.17.0.21) port 8855 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 172-17-0-21.curly-headless:8855
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
command terminated with exit code 56

OK, thanks for feedback @adamglt .  I think whatever we come up with should probably be the default - the current behavior seems incorrect.  We might need some broader buy-in as the approaches so far seem a little bit bigger than a simple bugfix.
I didn't try any of the approaches (aside from the /etc/hosts hack); I guessed approach 3 would work by adding that iptables rule but I'm probably missing something.  I'll try it out over here when I get a chance and then probably start debugging with iptables -v to get some counts and see where we're hitting/not hitting.

Looks like the POD1->POD1 packets are hitting the top rule
Chain ISTIO_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
 3949  237K ISTIO_REDIRECT  all  --  any    lo      anywhere            !localhost
 5763  346K RETURN     all  --  any    any     anywhere             anywhere             owner UID match 1337
    0     0 RETURN     all  --  any    any     anywhere             anywhere             owner GID match 1337
  540 32400 RETURN     all  --  any    any     anywhere             localhost
  953 57180 ISTIO_REDIRECT  all  --  any    any     anywhere             anywhere

If I add our new rule right after the first rule nothing happens (no hits, and kafka is still upset):
Chain ISTIO_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
 4545  273K ISTIO_REDIRECT  all  --  *      lo      0.0.0.0/0           !127.0.0.1
    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.96.1.81
 6436  386K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            owner UID match 1337
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            owner GID match 1337
  563 33780 RETURN     all  --  *      *       0.0.0.0/0            127.0.0.1
  976 58560 ISTIO_REDIRECT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

If I add it before the first rule, it hits and kafka gets happy:
Chain ISTIO_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
    2   120 RETURN     all  --  *      *       0.0.0.0/0            100.96.1.81
 4722  283K ISTIO_REDIRECT  all  --  *      lo      0.0.0.0/0           !127.0.0.1
    0     0 RETURN     all  --  *      *       0.0.0.0/0            100.96.1.81
 6639  398K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            owner UID match 1337
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            owner GID match 1337
  571 34260 RETURN     all  --  *      *       0.0.0.0/0            127.0.0.1
  985 59100 ISTIO_REDIRECT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

The original first rule (to ISTIO_REDIRECT) comes from here:
# Redirect app calls to back itself via Envoy when using the service VIP or endpoint
# address, e.g. appN => Envoy (client) => Envoy (server) => appN.
iptables -t nat -A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -j ISTIO_REDIRECT

@costinm , that was added in #2331 though it sounds like that might have been a code-move from someplace else.  Thoughts on whether what we're seeing is intentional?

Also sounds like based on the original comment:
appN => Envoy (client) => Envoy (server) => appN

the plan was to force traffic through Envoy client and server (or "outbound" and "inbound" in the listener/cluster lingo).  That's great with me but wasn't what you said you expected @adamglt - you thought it would make more sense to skip Envoy entirely in the appN->appN case.

Yeah, I expected explicit self-connections (localhost:1234) to behave the same as implicit self connections (my-pod.headless-service:1234).
"Opaque" self-connections (normal-service-that-includes-my-pod:1234) should obviously go through the usual flow.
I haven't come across headless service use cases outside app cluster gossip/coordination, so there might be something I'm not thinking about.

I added this bug to the Networking WG meeting agenda here, to discuss with the broader networking team on Thursday.

Would this be solved if we split the inbound and outbound listeners?

@andraxylia Yes definitely, that's my "Option 2" up above.  I think there are other reasons why that would be nice.  How to do it will take some discussing...

Were there any ideas/decisions?
Can't find the meeting video on youtube, not uploaded yet I guess.

Important note about using the steps above to reproduce:
It seems like kube-dns and coredns behave a bit differently with SRV records created for headless services: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#srv-records
In kube-dns, the auto-generated-name is really auto-generated, and would be something like 23894289347923.my-svc.my-ns.svc.cluster.local.
In coredns, the auto-generated-name is the dashed pod IP, so 172.20.1.2 -> 172-20-1-2.my-svc.my-ns.svc.cluster.local (see endpoint_pod_names here: https://coredns.io/plugins/kubernetes/).
This means my POD_DASHED_IP script above doesn't work on clusters with kube-dns.
Anyway, both of those behave the same way with StatefulSets, so this would be a more consistent base for testing:
apiVersion: v1
kind: ServiceAccount
metadata:
  name: curly
  labels:
    app: curly
automountServiceAccountToken: false
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: curly
  name: curly-discovery
spec:
  ports:
  - name: http
    port: 8855
    protocol: TCP
  selector:
    app: curly
  clusterIP: None
  publishNotReadyAddresses: true
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: curly-discovery
spec:
  hosts:
  - "*.curly-discovery.default.svc.cluster.local"
  location: MESH_INTERNAL
  ports:
  - number: 8855
    name: http
    protocol: HTTP
  resolution: NONE
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: curly
spec:
  selector:
    matchLabels:
      app: curly
  serviceName: curly-discovery
  replicas: 2
  template:
    metadata:
      labels:
        app: curly
    spec:
      serviceAccountName: curly
      containers:
      - name: curly
        image: tutum/curl
        command: ["python3", "-m", "http.server", "8855"]
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8855

@adamglt Sorry, while we discussed some options at the networking meeting the challenge is that there's a growing emphasis on quality and this bug happens to be in the middle of some stuff that everyone is hesitant to change.
@costinm - you suggested maybe there's a way the Sidecar resource could fix this problem.  I didn't see any but please let us know if there is a way.  (That would require upgrading to 1.1 as that resource isn't available in 1.0.4 but could be a good workaround)
@andraxylia - what should we do next to make progress on this issue?  I view the current behavior pretty strongly as purely a bug in 1.0 and 1.1.
I think we need a decision on what behavior we should have in this case ("pod talks to its own IP") (even if in the interim we only make incremental progress towards it).  I can write up my thoughts as well as the alternatives at least as a discussion point if that helps.

I had the same problem with kafka in K8S when I enabled mTLS.
The leader couldn't call himself through his DNS Name.
I fixed that locally by adding a container that adds the IP Table rule described by @andrewjjenkins
I added the snippet below in case anyone else needs it.
It would be nice to have an option in Istio that handles this case without changing Istios iptable-rule directly.
    spec:
      containers:
      - name: netshoot
         env:
         - name: POD_IP
           valueFrom:
             fieldRef:
               fieldPath: status.podIP
         image: nicolaka/netshoot
        command:
         - sh
         - -exc
         - |
           iptables -t nat -I ISTIO_OUTPUT  -d ${POD_IP}  -j RETURN && \
           sleep "100d"

@christianwoehrle very interesting ! I wonder why this rule could not be added by default on each pod where envoy runs ? @andrewjjenkins any thoughts ?

@christianwoehrle one issue I have with your trick is that it slows down (even seems to prevent) the deletion of my statefulset. This is because signals are not caught by the running sleep process. I was able to fix it by building my own image with dumb-init embedded. I wonder if there is a way to do it without. I tried to use trap and exec unsuccessfully. I'd like to reuse istio's image to avoid having to download a third image in each of my pod to add that route but I need to fix this signal issue first.

I think it's possible that #13666 addresses this by splitting inbound and outbound listeners, it looks like that is an implementation of my idea way back up here, that we talked about in Networking SIG but didn't get clear consensus on: https://github.com/istio/istio/issues/12551#issuecomment-474087161 .  That is slated to be an alpha feature in 1.2.  I will try to get some time and test it and report back.  Would getting a fix in 1.2 work for the various folks interested in this issue?

@andrewjjenkins, happy to test it whenever it’s out. Any idea when 1.2 is supposed to be released as GA ? I can test it using a RC version until it is

@andrewjjenkins, I tried to use that feature by setting (to an empty string, might be the reason) the annotation sidecar.istio.io/inboundInterceptionSplit in my StatefulSet. However it doesn't seem to do anything. I looked up for inboundInterceptionSplit in 1.2.0-rc.2 sources but did not find it. I did the same for the annotation traffic.sidecar.istio.io/includeInboundPorts which I found in the code. I see that ISTIO_META_INCLUDE_INBOUND_PORTS is empty and also -b argument in proxy command. And it definitely does not work. Will you have time anytime soon to look at it ?

Yes, it looks like the reviewers requested a rename on some pieces of #13666 before merge.  I'm looking into it, and following this from the original PR:
** How to enable this feature **

use iptables REDIRECT mode as interception method
provide env var for iptables script INBOUND_CAPTURE_PORT=15006
short cut for k8s deployment
Any pod running sidecar proxy could opt-in by set the annotationtraffic.sidecar.istio.io/includeInboundPorts to "*" (STAR)

I see that the iptables script in 1.2.0-rc.2 has a new option INBOUND_CAPTURE_PORT that appears to do the redirection.  Check out https://github.com/istio/istio/blob/7abc166577853239b05e50c68b02b4830e2417a8/tools/packaging/common/istio-iptables.sh#L325 .
I'm still trying to understand the annotation.  Did you do anything to cause INBOUND_CAPTURE_PORT to be set?  Maybe INBOUND_CAPTURE_PORT plus the traffic.sidecar.istio.io/includeInboundPorts is the trick?  I will try and report back unless you beat me to it.
p.s. I agree, I don't see inboundInterceptionSplit anywhere in the source.

@andrewjjenkins I'm trying to understand how the inbound capture would add the rule we're missing
i.e iptables -t nat -I ISTIO_OUTPUT  -d ${POD_IP}  -j RETURN or avoid us to need it. Do you see another rule that could act like that one.
It seems we could try to use https://github.com/istio/istio/blob/7abc166577853239b05e50c68b02b4830e2417a8/tools/packaging/common/istio-iptables.sh#L450-L455 or https://github.com/istio/istio/blob/7abc166577853239b05e50c68b02b4830e2417a8/tools/packaging/common/istio-iptables.sh#L578-L582 which comes from https://github.com/istio/istio/blob/7abc166577853239b05e50c68b02b4830e2417a8/tools/packaging/common/istio-iptables.sh#L235. I don't know yet we can override it or not. But I still feel like it should be done automatically and not by us by applying a specific rule, don't you think ?

I think this is on the "avoid us to need it" at least how I was thinking about it.  I think the feature in #13666 is to cause the inbound and outbound listeners to be split, so that the self-ip traffic can be properly mapped to the outbound and then the inbound listeners.
Another option would be to skip both the inbound and outbound listeners in the special case that a pod is talking to itself.  I think this is equivalent to iptables -t nat -I ISTIO_OUTPUT -d ${POD_IP} -j RETURN.
So, choices are "both" (inbound and outbound listeners) or "neither"; the only one that doesn't work is "go through only the inbound listener" but this is what we are observing today.  I'm thinking this new feature may allow us to implement "both", I think you might have been looking for "neither".
So I can get the traffic.sidecar.istio.io/includeInboundPorts: "*" annotation to cause the iptables to look like how I think they should.
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: kafka
spec:
  serviceName: kafka-svc
  replicas: 3
  template:
    metadata:
      annotations:
        # IMPORTANT: this annotation goes on spec.template.metadata, not top-level metadata
        traffic.sidecar.istio.io/includeInboundPorts: "*"
      labels:
        app: kafka
    spec:
...

$ kubectl logs kafka-0 -c istio-init
...
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006  # <--- Annotation is working
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
...

Envoy is properly catching connections on both 15001 and 15006.  But it looks like they are mapping to the same set of listeners still (or at least I haven't figured that part out), and we are seeing the same end-behavior; pods cannot call themselves.
I will try some more digging.  @silentdai , are we seeing what you'd expect?  Once the inbound and outbound listen sockets are split, is there anything in the Envoy config layer to cause them to map to different "inbound listeners" and "outbound listeners"?  I see these two "virtual" and "virtualInbound" listeners but they seem to map to the same cluster, I don't know if that is the goal of #13666 or not.
   {
     "version_info": "2019-06-18T22:22:26Z/19",
     "listener": {
      "name": "virtual",
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 15001
       }
      },
      "filter_chains": [
       {
        "filters": [
         {
          "name": "envoy.tcp_proxy",
          "config": {
           "cluster": "PassthroughCluster",
           "access_log": [
            {
             "name": "envoy.file_access_log",
             "config": {
              "path": "/dev/stdout",
              "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME%\n"
             }
            }
           ],
           "stat_prefix": "PassthroughCluster"
          }
         }
        ]
       }
      ],
      "use_original_dst": true
     },
     "last_updated": "2019-06-18T22:56:06.125Z"
    },
    {
     "version_info": "2019-06-18T22:22:26Z/19",
     "listener": {
      "name": "virtualInbound",
      "address": {
       "socket_address": {
        "address": "0.0.0.0",
        "port_value": 15006
       }
      },
      "filter_chains": [
       {
        "filters": [
         {
          "name": "envoy.tcp_proxy",
          "config": {
           "access_log": [
            {
             "name": "envoy.file_access_log",
             "config": {
              "path": "/dev/stdout",
              "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME%\n"
             }
            }
           ],
           "stat_prefix": "PassthroughCluster",
           "cluster": "PassthroughCluster"
          }
         }
        ]
       }
      ],
      "use_original_dst": true
     },
     "last_updated": "2019-06-18T22:56:06.125Z"
    }

So, choices are "both" (inbound and outbound listeners) or "neither"; the only one that doesn't work is "go through only the inbound listener" but this is what we are observing today. I'm thinking this new feature may allow us to implement "both", I think you might have been looking for "neither"

Gotcha. Yeah, I'm trying to do "neither". Unfortunately the parameter I was looking into, excludeOutboundIPRanges, can't handle POD_IP https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/files/injection-template.yaml#L19-L22 but only fixed IP ranges ....

@silentdai could you have a look by any chance 👆 ?

Ack. Inbound split should not improve the connectivity ability. Split unlock the possibility but I need to read through what occurs here before I propose the next step

Inbound split should not improve the connectivity ability

what do you mean by "improve" here ?

@andrewjjenkins @silentdai, don't know if it helps understanding it or fixing it, but I made some tests using istio-cni and it works without having to do any hack regarding the local communication. It means I don't need to force it to listen on 127.0.0.1 for local communications and do no longer need that special rule regarding POD_IP.
I'm just surprised by the output of istioctl :
istioctl authn tls-check cassandra-1
HOST:PORT                                                     STATUS     SERVER        CLIENT     AUTHN POLICY                                 DESTINATION RULE
cassandra.cassandra-e2e.svc.cluster.local:7000                OK         HTTP/mTLS     HTTP       default/                                     -
cassandra.cassandra-e2e.svc.cluster.local:7001                OK         HTTP/mTLS     HTTP       default/                                     -
cassandra.cassandra-e2e.svc.cluster.local:7199                OK         HTTP/mTLS     HTTP       default/                                     -
cassandra.cassandra-e2e.svc.cluster.local:9042                OK         HTTP/mTLS     HTTP       default/                                     -

Does it mean anything I should worry about ? I see that envoy is started with
--controlPlaneAuthPolicy
      MUTUAL_TLS

and that there are certificates in the configuration dump.

I created a new issue after having tested istio-cni

thanks @cscetbon for checking out istio-cni, agree with your bug report.  It's not desirable for us that "communication works but silently turns off encryption" so we should keep working on #15171 .

@andrewjjenkins Thanks for the work. I assigned this issue to you. Please free free to let me know if not appropriate.

This should be fixed in 1.3. No need to use cni
@adamglt @cscetbon Could you verify?

The reproduction is a bit different than above - but this is still broken.
Install Istio with auth enabled, then:
$ kubectl create ns podtest
namespace/podtest created

$ kubectl label ns podtest istio-injection=enabled
namespace/podtest labeled

$ kubectl config set-context --current --namespace podtest
Context "minikube" modified.

$ kubectl apply -f - <<EOF
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: curly
>   labels:
>     app: curly
> automountServiceAccountToken: false
> ---
> apiVersion: v1
> kind: Service
> metadata:
>   labels:
>     app: curly
>   name: curly-headless
> spec:
>   ports:
>   - name: http
>     port: 8855
>     protocol: TCP
>   selector:
>     app: curly
>   clusterIP: None
>   publishNotReadyAddresses: true
> ---
> apiVersion: apps/v1
> kind: Deployment
> metadata:
>   name: curly
> spec:
>   replicas: 2
>   selector:
>     matchLabels:
>       app: curly
>   template:
>     metadata:
>       labels:
>         app: curly
>     spec:
>       serviceAccountName: curly
>       containers:
>       - name: curly
>         image: tutum/curl
>         command: ["python3", "-m", "http.server", "8855"]
>         imagePullPolicy: IfNotPresent
>         ports:
>         - containerPort: 8855
> EOF
serviceaccount/curly created
service/curly-headless created
deployment.apps/curly created

$ kubectl wait --for=condition=Ready $(kubectl get pods -o=go-template='{{range .items}}pod/{{println .metadata.name}}{{end}}')
pod/curly-7fdcfd5cb7-hlbbs condition met
pod/curly-7fdcfd5cb7-z9ffg condition met

$ POD1=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].metadata.name}') && \
> POD2_DASHED_IP=$(kubectl get pods -l app=curly -o jsonpath='{.items[1].status.podIP}' | sed 's/\./-/g') && \
> kubectl exec $POD1 -c curly -- curl -s -I ${POD2_DASHED_IP}.curly-headless:8855
HTTP/1.1 200 OK
server: envoy
date: Fri, 25 Oct 2019 08:37:05 GMT
content-type: text/html; charset=ascii
content-length: 987
x-envoy-upstream-service-time: 6

$ POD1=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].metadata.name}') && \
> POD1_DASHED_IP=$(kubectl get pods -l app=curly -o jsonpath='{.items[0].status.podIP}' | sed 's/\./-/g') && \
> kubectl exec $POD1 -c curly -- curl -v -s -I ${POD1_DASHED_IP}.curly-headless:8855
* Rebuilt URL to: 172-17-0-22.curly-headless:8855/
* Hostname was NOT found in DNS cache
*   Trying 172.17.0.22...
* Connected to 172-17-0-22.curly-headless (172.17.0.22) port 8855 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 172-17-0-22.curly-headless:8855
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
command terminated with exit code 56

@lambdai when you say no need to CNI, you forget it's needed when containers can't run as root in the cluster.

@adamglt I like your method of testing it ;) Definitely still broken and that's what I confirmed in my tests too.

15 days ago someone posted a case showing that issue is still not resolved. This issue should not be closed.

With this #19308, i think it should not have such mtls issue when calling itself.

Can confirm it works with istio v1.5.0-beta.1 after deleting pods using headless service

That's a good point, as the iptables is set in init-container during pod bootstrap, so users should restart the pod.

The test above seems to be working with 1.5.0-beta.2:
# pod1 -> pod2
HTTP/1.1 200 OK
server: envoy
date: Tue, 18 Feb 2020 17:35:28 GMT
content-type: text/html; charset=ascii
content-length: 942
x-envoy-upstream-service-time: 22

# pod1 -> pod1
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.4.0
Date: Tue, 18 Feb 2020 17:35:29 GMT
Content-type: text/html; charset=ascii
Content-Length: 942

We'll keep testing with our existing Akka clusters and update.
Thanks to everyone that helped with this issue.

Hi @andrewjjenkins
I am having issues istio mtls for kafka. When I make namespace Strict, kafka pod to pod communication fails. I have got zookeeper working using quorumListenAllIps set to true and excluded 3888,2888 from inbound/outbound.
https://github.com/istio/istio/issues/26791
I tried additional below solution, but did not work. Can you please help what I am missing
--- Kafka clusterip service
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka
heritage: Tiller
release: infra-kafka
name: infra-kafka
namespace: infra
spec:
clusterIP: 172.19.131.1
ports:

name: tcp-kafka
port: 9092
protocol: TCP
targetPort: tcp-kafka
selector:
app: kafka
release: infra-kafka
type: ClusterIP

--- kafka headless service
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka
heritage: Tiller
release: infra-kafka
name: infra-kafka-headless
namespace: infra
spec:
clusterIP: None
ports:

name: tcp-kafka
port: 9092
protocol: TCP
targetPort: tcp-kafka
selector:
app: kafka
release: infra-kafka
sessionAffinity: None
type: ClusterIP

--- service entries below
kubectl apply -n infra -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: infra-kafka
spec:
hosts:

infra-kafka-0.infra-kafka-headless.infra.svc.cluster.local
infra-kafka-1.infra-kafka-headless.infra.svc.cluster.local
infra-kafka-2.infra-kafka-headless.infra.svc.cluster.local
location: MESH_INTERNAL
ports:
number: 9092
name: tcp-kafka
protocol: TCP
resolution: NONE
EOF

--- virtual service entries below for routing
kubectl apply -n infra -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: infra-0
namespace: infra
spec:
gateways:

mesh
hosts:
infra-kafka-0.infra-kafka-headless.infra.svc.cluster.local
http:
route:

destination:
host: infra-kafka-0.infra-kafka-headless.infra.svc.cluster.local
port:
number: 9092
EOF

kubectl apply -n infra -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: infra-1
namespace: infra
spec:
gateways:

mesh
hosts:
infra-kafka-1.infra-kafka-headless.infra.svc.cluster.local
http:
route:

destination:
host: infra-kafka-1.infra-kafka-headless.infra.svc.cluster.local
port:
number: 9092
EOF

kubectl apply -n infra -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: infra-2
namespace: infra
spec:
gateways:

mesh
hosts:
infra-kafka-2.infra-kafka-headless.infra.svc.cluster.local
http:
route:

destination:
host: infra-kafka-2.infra-kafka-headless.infra.svc.cluster.local
port:
number: 9092
EOF

--- rules on routing
kubectl apply -n infra -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: infra-rule1
namespace: infra
spec:
host: infra-kafka-0.infra-kafka-headless.infra.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
EOF
kubectl apply -n infra -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: infra-rule2
namespace: infra
spec:
host: infra-kafka-1.infra-kafka-headless.infra.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
EOF
kubectl apply -n infra -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: infra-rule3
namespace: infra
spec:
host: infra-kafka-2.infra-kafka-headless.infra.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
EOF

2. rvice[headless],

Hi, brother.
Your case give me a successful example about how to make authencation for headless service, in which pod are called directly. I wonder why it does not need DestinationRule configuration, since DestinationRule is required, according to the formal documentantion, in order to tell client use mtls protocol.  I haved configured my project as you did, it succeed for authencation and authorization. Colud you explain for me, or show me any documentantion to  eliminate my confusion ?
Thanks a lot.

@yibaoren I'm guessing your Istio installation included a default mesh-level DestinationRule that enables mTLS:
$ kubectl get destinationrule -n istio-system default -o yaml | kubectl neat
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: default
  namespace: istio-system
spec:
  host: '*.local'
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

I am glad to get your response.
I have checked the resource, however, no DestinationRule is found, as the following picture shows.

Hi @adamglt
Today, I reinstall my project, and find "0 NR filter_chain_not_found " error in envoy log.
Then I try to add DestinationRule as what you showed. It works. Maybe there are some bugs before in the system.
Thank you very much.

authors: adamglt andraxylia andrewjjenkins christianwoehrle cscetbon hugocortes hzxuzhonghu lambdai myidpt satishmane silentdai yibaoren


text:
Unable to authenticate JWT Token + CORS

I can't seem to wrap my head around how and where Istio is performing what type of authentication...
The reason we are not defining JWT authentication at the backend service level is because we do not want to authenticate our internal services just to send traffic to this particular service.  Only external communication should be authenticated.
TL;DR:

Can and should I create a Virtual Service for my istio-ingressgateway for CORS processing when my frontend localhost or app.example.com doesnt match the host designator on the  Gateway api.example.com?

I am placing my backend Gateway in the same namespace as the application I am serving, should it be in the istio-system namespace instead?

Is CORS processing performed before the JWT Check?

Is it possible to define a service level JWT authentication policy to only authenticate external communications, and still allow internal services access to said backend service?

Versions first:
#KUBERNETES
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-eks-7c34c0", GitCommit:"7c34c0d2f2d0f11f397d55a46945193a0e22d8f3", GitTreeState:"clean", BuildDate:"2019-03-01T22:49:39Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

#ISTIO
Server Version: 1.0.6

NOTE: I could not get ouput from istioctl as I deleted the folder by mistake and had to reinstall the tool (newer version), server side has not changed.
INGRESS-GATEWAY:

namespace: istio-system

istio-ingressgateway is default aside from being an AWS NLB as opposed to an AWS CLB.
POLICY (for JWT):

namespace: istio-system

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: core-api-jwt-auth
  namespace: istio-system
spec:
  targets:
  - name: istio-ingressgateway
    ports:
    - number: 443
  origins:
  - jwt:
      audiences:
      - 4i00tescmmXXXXXX6c1chlvhpq
      issuer: https://cognito-idp.ap-northeast-1.amazonaws.com/ap-northeast-1_XXXXXXX
      jwksUri: https://cognito-idp.ap-northeast-1.amazonaws.com/ap-northeast-1_XXXXXXX/.well-known/jwks.json
  principalBinding: USE_ORIGIN

GATEWAY:

namespace: custom

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: core-api-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "api.example.net"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "api.example.net"
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

VIRTUAL SERVICE:

namespace: custom

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: core-api-virtual-service
spec:
  hosts:
  - "api.example.net"
  gateways:
  - core-api-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: core-api-service
        port:
          number: 80
    corsPolicy:
      allowOrigin:
        - "*"
      allowMethods:
        - POST
        - GET
        - OPTIONS
      allowHeaders:
        - content-type
      maxAge: 300s
      allowCredentials: true

**JWT: **
{
  "sub": "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "aud": "4i00tescmmXXXXXX6c1chlvhpq",
  "email_verified": true,
  "token_use": "id",
  "auth_time": 1553610006,
  "iss": "https://cognito-idp.ap-northeast-1.amazonaws.com/ap-northeast-1_XXXXXXX",
  "cognito:username": "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "exp": 1553613606,
  "custom:initial_device_id": "123",
  "iat": 1553610006,
  "email": "procyclinsur@xxxxx.jp"
}

REQUEST HEADERS:
Host: api.example.net
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: POST
Access-Control-Request-Headers: authorization,content-type
Referer: http://localhost:8080/
Origin: http://localhost:8080
Connection: keep-alive
TE: Trailers

NOTE: I manually entered Authorization: apollo-token in subsequent runs with only change being to the response body shown below.
ERROR:

Web Console:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.example.net/graphql. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.example.net/graphql. (Reason: CORS request did not succeed)

Response Body (when Authorization header is manually entered):

When token is not manually entered the body is empty.
Origin authentication failed.

Response Headers

HTTP/2.0 401 Unauthorized
content-length: 29
content-type: text/plain
date: Tue, 26 Mar 2019 14:49:46 GMT
server: envoy
X-Firefox-Spdy: h2

Opening new request due to learning a bit more about what is wrong..

source: gharchive/issue

repo: istio/istio

license_type: permissive

license_source: github-api


text:
unmarshal fails for enum type

for latest master, unmarshal fails for enum value for unknown value error e.g.
outboundTrafficPolicy:
      mode: ALLOW_ANY

This does not happen for 1.5 though, is it possibly related to the recent changes about the marshal/unmarshal stuff? @ostromart @morvencao

@richardwxn I didn't see this with following IOP:
# cat values-global.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      outboundTrafficPolicy:
        mode: ALLOW_ANY

Would you share more details?

@morvencao Did you test with master or 1.5?

@richardwxn master

It may be due to gogo/golang proto, which I just updated last night on 1.5: https://github.com/istio/api/pull/1338

authors: morvencao ostromart richardwxn


text:
per_filter_config not allowing 'disabled: false' values

Bug description
I'm trying to configure ext_authz filter per virtual hosts. Currently I have ext_authz filter as follow:
kind: EnvoyFilter
metadata:
  name: authn-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.ext_authz
        config:
          http_service:
            server_uri:
              uri: http://authservice.istio-system.svc.cluster.local
              cluster: outbound|8080||authservice.istio-system.svc.cluster.local
              failure_mode_allow: false
              timeout: 10s
            authorization_request:
              allowed_headers:
                patterns:
                - exact: "cookie"
                - exact: "X-Auth-Token"
            authorization_response:
              allowed_upstream_headers:
                patterns:
                - exact: "kubeflow-userid"
          status_on_error:
            code: GatewayTimeout

I don't want to apply this filter to all virtual hosts so I'm excluding some virtual hosts as follow:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: bypass-auth-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      routeConfiguration:
        vhost:
          name: a.example.com:80
    patch:
        operation: MERGE
        value:
          per_filter_config:
            envoy.ext_authz:
              disabled: true

However, I would like if I can do it such that I apply the filter on specific virtual hosts rather than having to exclude every hosts that doesn't need auth (more like a whitelist solution rather than a blacklist)
So I tried doing this:
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: all-bypass-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      routeConfiguration:
        vhost:
          route:
            name: '/'
    patch:
        operation: MERGE
        value:
          per_filter_config:
            envoy.ext_authz:
              disabled: true
  - applyTo: VIRTUAL_HOST
    match:
      routeConfiguration:
        vhost:
          name: 'b.example.com:80'
    patch:
        operation: MERGE
        value:
          per_filter_config:
            envoy.ext_authz:
              disabled: false

Here I only want the authentication on b.example.com. So I tried disabling the filter on everything, then re-enable it only for that specific virtual hosts. But when applying this filter, I get the following error:
[Envoy (Epoch 0)] [2020-06-09 15:11:50.796][20][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:82] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration rejected: Proto constraint validation failed (ExtAuthzPerRouteValidationError.Disabled: ["value must equal " %!q(bool=true)]): disabled: false

Why it is not allowed to set disabled to false?
[x] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
ext_authz filter is disabled on all virtual hosts but b.example.com
Steps to reproduce the bug
Apply the envoyfilters mentioned above using kubectl apply -f file.yaml
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
istio version 1.4
How was Istio installed?
By admin as part of EKS cluster setup
Environment where bug was observed (cloud vendor, OS, etc)
Cloud - EKS cluster

I think this is the same problem as https://github.com/istio/istio/issues/18169?

Closing as duplicate of https://github.com/istio/istio/issues/18169

This is not a duplicate of #18169 because the value true is not a protobuf default value, see https://developers.google.com/protocol-buffers/docs/proto3#default
@abdulsalama you want something like:

  # Skip requiring Bearer tokens when logging in:
  # https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-EnvoyConfigObjectPatch
  - applyTo: VIRTUAL_HOST
    # https://istio.io/docs/reference/config/networking/envoy-filter/#EnvoyFilter-EnvoyConfigObjectMatch
    match:
      # https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-PatchContext
      context: GATEWAY
      # https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-RouteConfigurationMatch
      routeConfiguration:
        gateway: istio-system/ingress
        # https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-RouteConfigurationMatch-VirtualHostMatch
        vhost:
          name: accounts.example.com:443
          # route: ...
    patch:
      # https://istio.io/latest/docs/reference/config/networking/envoy-filter/#EnvoyFilter-Patch-Operation
      operation: MERGE
      # -------- Envoy record is merged with the value
      # https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#struct
      value:
        name: envoy.ext_authz_disabled
        # https://www.envoyproxy.io/docs/envoy/v1.16.1/api-v3/config/route/v3/route_components.proto.html?highlight=per_filter_config
        typed_per_filter_config:
          envoy.filters.http.ext_authz:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
            disabled: true

However, it would seem there are even more bugs lurking in istio: https://github.com/istio/istio/issues/29734 — I haven't made the above work properly yet; it removes the full Route of the Gateway it tries to disable the Filter for, rather than removing the Filter.

authors: abdulsalama haf howardjohn


text:
istio-ingressgateway is still unhealthy.

Bug description
I tried to install the istio via Istio Operator. Following below is my istio-operator manifest
# kubectl create ns istio-system
...
# istioctl operator init
...
# kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: mars-istiocontrolplane
spec:
  profile: default
EOF

And the istio-ingressgateway is stucked in 0/1 Running and the pod Event description is
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  33m                    default-scheduler  Successfully assigned istio-system/istio-ingressgateway-64cfb9d44b-wvg9x to 10.0.0.202
  Normal   Pulled     33m                    kubelet            Container image "docker.io/istio/proxyv2:1.7.3" already present on machine
  Normal   Created    33m                    kubelet            Created container istio-proxy
  Normal   Started    33m                    kubelet            Started container istio-proxy
  Warning  Unhealthy  3m42s (x900 over 33m)  kubelet            Readiness probe failed: Get "http://10.1.121.8:15021/healthz/ready": dial tcp 10.1.121.8:15021: connect: connection refused

The log of istio-ingressgateway pod:
2020-10-17T10:09:05.108110Z	info	FLAG: --concurrency="0"
2020-10-17T10:09:05.108132Z	info	FLAG: --disableInternalTelemetry="false"
2020-10-17T10:09:05.108136Z	info	FLAG: --domain="istio-system.svc.cluster.local"
2020-10-17T10:09:05.108138Z	info	FLAG: --help="false"
2020-10-17T10:09:05.108140Z	info	FLAG: --id=""
2020-10-17T10:09:05.108142Z	info	FLAG: --ip=""
2020-10-17T10:09:05.108145Z	info	FLAG: --log_as_json="false"
2020-10-17T10:09:05.108148Z	info	FLAG: --log_caller=""
2020-10-17T10:09:05.108151Z	info	FLAG: --log_output_level="default:info"
2020-10-17T10:09:05.108154Z	info	FLAG: --log_rotate=""
2020-10-17T10:09:05.108158Z	info	FLAG: --log_rotate_max_age="30"
2020-10-17T10:09:05.108160Z	info	FLAG: --log_rotate_max_backups="1000"
2020-10-17T10:09:05.108163Z	info	FLAG: --log_rotate_max_size="104857600"
2020-10-17T10:09:05.108165Z	info	FLAG: --log_stacktrace_level="default:none"
2020-10-17T10:09:05.108170Z	info	FLAG: --log_target="[stdout]"
2020-10-17T10:09:05.108172Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2020-10-17T10:09:05.108175Z	info	FLAG: --mixerIdentity=""
2020-10-17T10:09:05.108178Z	info	FLAG: --outlierLogPath=""
2020-10-17T10:09:05.108181Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2020-10-17T10:09:05.108194Z	info	FLAG: --proxyLogLevel="warning"
2020-10-17T10:09:05.108199Z	info	FLAG: --serviceCluster="istio-ingressgateway"
2020-10-17T10:09:05.108202Z	info	FLAG: --serviceregistry="Kubernetes"
2020-10-17T10:09:05.108205Z	info	FLAG: --stsPort="0"
2020-10-17T10:09:05.108207Z	info	FLAG: --templateFile=""
2020-10-17T10:09:05.108209Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2020-10-17T10:09:05.108211Z	info	FLAG: --trust-domain="cluster.local"
2020-10-17T10:09:05.108237Z	info	Version 1.7.3-9686754643d0939c1f4dd0ee20443c51183f3589-Clean
2020-10-17T10:09:05.108345Z	info	Obtained private IP [10.1.121.15]
...
2020-10-17T10:09:05.109330Z	info	JWT policy is first-party-jwt
2020-10-17T10:09:05.109357Z	info	PilotSAN []string{"istiod.istio-system.svc"}
2020-10-17T10:09:05.109362Z	info	MixerSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"}
2020-10-17T10:09:05.109387Z	info	sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2020-10-17T10:09:05.109392Z	info	Using user-configured CA istiod.istio-system.svc:15012
2020-10-17T10:09:05.109395Z	info	istiod uses self-issued certificate
2020-10-17T10:09:05.109423Z	info	the CA cert of istiod is: -----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxx
-----END CERTIFICATE-----

2020-10-17T10:09:05.138017Z	info	Starting gateway SDS
2020-10-17T10:09:05.239146Z	info	sds	SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS"

2020-10-17T10:09:05.239230Z	info	sds	SDS gRPC server for gateway controller starts, listening on "./var/run/ingress_gateway/sds"

2020-10-17T10:09:05.239261Z	info	Starting proxy agent
2020-10-17T10:09:05.239232Z	info	sds	Start SDS grpc server
2020-10-17T10:09:05.239292Z	info	Opening status port 15020

2020-10-17T10:09:05.239317Z	info	sds	Start SDS grpc server for ingress gateway proxy
2020-10-17T10:09:05.239341Z	info	Received new config, creating new Envoy epoch 0
2020-10-17T10:09:05.239368Z	info	Epoch 0 starting
2020-10-17T10:09:07.241224Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.1.121.15~istio-ingressgateway-64cfb9d44b-lrbt7.istio-system~istio-system.svc.cluster.local --local-address-ip-version v4 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error]
2020-10-17T10:09:07.271441Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.reloadable_features.activate_fds_next_event_loop
2020-10-17T10:09:07.300581Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:09:07.300596Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:09:07.760510Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:09:07.760536Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:09:10.775603Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:09:10.775631Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:09:14.834496Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:09:14.834522Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:09:40.874684Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:09:40.874709Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:09:42.977790Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:09:42.977815Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:09:55.285374Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:09:55.285399Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:10:04.418329Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:10:04.418354Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:10:09.964593Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:10:09.964621Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:10:16.524364Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:10:16.524389Z	warning	envoy config	Unable to establish new stream
2020-10-17T10:10:22.303369Z	info	sds	resource:default new connection
2020-10-17T10:10:22.303429Z	info	sds	Skipping waiting for gateway secret
2020-10-17T10:10:29.858021Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:29.908072Z	warn	cache	resource:default request:d185ffa6-9e9a-4961-b14b-31350491904c CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2020-10-17T10:10:29.908140Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:30.008242Z	warn	cache	resource:default request:d185ffa6-9e9a-4961-b14b-31350491904c CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2020-10-17T10:10:30.008329Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:30.208432Z	warn	cache	resource:default request:d185ffa6-9e9a-4961-b14b-31350491904c CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2020-10-17T10:10:30.208495Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:30.608598Z	warn	cache	resource:default request:d185ffa6-9e9a-4961-b14b-31350491904c CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2020-10-17T10:10:30.608694Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:31.408810Z	warn	cache	resource:default request:d185ffa6-9e9a-4961-b14b-31350491904c CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec
2020-10-17T10:10:31.408877Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"

[ ] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Expected behavior
istio-ingressgateway is Running normally.
Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)

istio: 1.7.3
kubernetes: v1.19.3

How was Istio installed?
IstioOperator
Environment where bug was observed (cloud vendor, OS, etc)
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

There are more logs:
2020-10-17T10:10:31.408877Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.009007Z     warn    cache   resource:default request:d185ffa6-9e9a-4961-b14b-31350491904c CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 1600 millisec
2020-10-17T10:10:33.009083Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.009092Z     error   cache   resource:default request:d185ffa6-9e9a-4961-b14b-31350491904c CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.009123Z     error   cache   resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.009135Z     error   sds     resource:default Close connection. Failed to get secret for proxy "router~10.1.121.15~istio-ingressgateway-64cfb9d44b-lrbt7.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.009198Z     info    sds     resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-10-17T10:10:33.009380Z     warning envoy config    StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.097182Z     info    sds     resource:default new connection
2020-10-17T10:10:33.097245Z     info    sds     Skipping waiting for gateway secret
2020-10-17T10:10:33.583826Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.633938Z     warn    cache   resource:default request:666f0bd8-6c4f-4778-b542-affe0afd2410 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2020-10-17T10:10:33.634013Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.734116Z     warn    cache   resource:default request:666f0bd8-6c4f-4778-b542-affe0afd2410 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2020-10-17T10:10:33.734198Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:33.934302Z     warn    cache   resource:default request:666f0bd8-6c4f-4778-b542-affe0afd2410 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2020-10-17T10:10:33.934377Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:34.334468Z     warn    cache   resource:default request:666f0bd8-6c4f-4778-b542-affe0afd2410 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2020-10-17T10:10:42.045042Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:10:42.045066Z     warning envoy config    Unable to establish new stream
2020-10-17T10:10:48.066540Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:10:48.066564Z     warning envoy config    Unable to establish new stream
2020-10-17T10:10:54.253569Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:54.253589Z     error   cache   resource:default request:666f0bd8-6c4f-4778-b542-affe0afd2410 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:54.253612Z     error   cache   resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:54.253631Z     error   sds     resource:default Close connection. Failed to get secret for proxy "router~10.1.121.15~istio-ingressgateway-64cfb9d44b-lrbt7.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:54.253670Z     info    sds     resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-10-17T10:10:54.253805Z     warning envoy config    StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:54.666046Z     info    sds     resource:default new connection
2020-10-17T10:10:54.666110Z     info    sds     Skipping waiting for gateway secret
2020-10-17T10:10:54.904063Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:54.954183Z     warn    cache   resource:default request:f09de890-1686-45b3-abf7-f478e3c1cfb7 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2020-10-17T10:10:54.954258Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:55.054365Z     warn    cache   resource:default request:f09de890-1686-45b3-abf7-f478e3c1cfb7 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2020-10-17T10:10:55.054439Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:10:55.254550Z     warn    cache   resource:default request:f09de890-1686-45b3-abf7-f478e3c1cfb7 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2020-10-17T10:10:55.254640Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
...

2020-10-17T10:12:53.059288Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-10-17T10:12:59.459403Z     warn    cache   resource:default request:8a3c054b-8540-461c-a3ba-d17cd4811fb5 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 6400 millisec
2020-10-17T10:13:14.109588Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:13:14.109615Z     warning envoy config    Unable to establish new stream
2020-10-17T10:13:25.634589Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:13:25.634617Z     warning envoy config    Unable to establish new stream
2020-10-17T10:13:26.736144Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:13:26.736168Z     warning envoy config    Unable to establish new stream
2020-10-17T10:13:29.771330Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-10-17T10:13:29.771355Z     warning envoy config    Unable to establish new stream
2020-10-17T10:13:36.437388Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.0.6.200:53: read udp 10.1.121.15:38949->10.0.6.200:53: i/o timeout"
2020-10-17T10:13:36.437407Z     error   cache   resource:default request:8a3c054b-8540-461c-a3ba-d17cd4811fb5 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.0.6.200:53: read udp 10.1.121.15:38949->10.0.6.200:53: i/o timeout"
2020-10-17T10:13:36.437428Z     error   cache   resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.0.6.200:53: read udp 10.1.121.15:38949->10.0.6.200:53: i/o timeout"
2020-10-17T10:13:36.437437Z     error   sds     resource:default Close connection. Failed to get secret for proxy "router~10.1.121.15~istio-ingressgateway-64cfb9d44b-lrbt7.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.0.6.200:53: read udp 10.1.121.15:38949->10.0.6.200:53: i/o timeout"
2020-10-17T10:13:36.437482Z     info    sds     resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-10-17T10:13:36.437629Z     warning envoy config    StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.0.6.200:53: read udp 10.1.121.15:38949->10.0.6.200:53: i/o timeout"
2020-10-17T10:13:43.258629Z     info    sds     resource:default new connection
2020-10-17T10:13:43.258709Z     info    sds     Skipping waiting for gateway secret
2020-10-17T10:13:43.480539Z     error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.0.6.200:53: read udp 10.1.121.15:38949->10.0.6.200:53: i/o timeout"

10.0.6.200 is the IP of CoreDNS. 10.1.121.15 is the IP of istio-ingressgateway pod. I am sure the CoreDNS is ok.
ping istiod.istio-system.svc
PING istiod.istio-system.svc.cluster.local (10.0.6.51) 56(84) bytes of data.
64 bytes from istiod.istio-system.svc.cluster.local (10.0.6.51): icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from istiod.istio-system.svc.cluster.local (10.0.6.51): icmp_seq=2 ttl=64 time=0.032 ms
^C
--- istiod.istio-system.svc.cluster.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms

This looks like you may have some lower level networking issues in your cluster possibly? It seems to fail to connected to istio many many times but on occasion it did connect. On other s it times out of even fails dns lookup
Do you have other pods in  this cluster that run successfully? and without DNS issues?

The DNS is ok, but the iptables rule has been changed by Docker. Docker also sets the policy for the FORWARD chain to DROP.
Thanks a lot. I'll close this issue as it is not related to Istio.

authors: Macrame howardjohn


text:
istioctl proxy-status command not working

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
Hi Team,
We are running 3 versions of Istiod Control plane (1.15, 1.16 & 1.18) in our clusters using revision tag and we have enabled hostNetwork=true for istiod deployment. So we had to modify few ports used by istiod.
So once we modify monitoringAddr from 15014 to 15114 or any other port, we are getting an error while running istioctl proxy-status command.
ERROR MESSAGE: 2023-08-23T06:48:52.036368Z     error   klog    an error occurred forwarding 61563 -> 15014: error forwarding port 15014 to pod 603a37aa17fead498e0581d1677d65579ec846e494e12585de8b814f9a3a45c5, uid : failed to execute portforward in network namespace "host": failed to connect to localhost:15014 inside namespace "603a37aa17fead498e0581d1677d65579ec846e494e12585de8b814f9a3a45c5", IPv4: dial tcp4 127.0.0.1:15014: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
Error: failure running port forward process: Get "http://localhost:61563//debug/syncz": EOF
We are not able to find any options are flags available with istioctl to change the monitoring port. Could you please let us know how to make it work, since we use this command extensively to check if our proxies are syncing
Version
2023-08-23T07:00:03.277753Z     error   klog    an error occurred forwarding 62990 -> 15014: error forwarding port 15014 to pod 603a37aa17fead498e0581d1677d65579ec846e494e12585de8b814f9a3a45c5, uid : failed to execute portforward in network namespace "host": failed to connect to localhost:15014 inside namespace "603a37aa17fead498e0581d1677d65579ec846e494e12585de8b814f9a3a45c5", IPv4: dial tcp4 127.0.0.1:15014: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
client version: 1.16.0
istiod version: 1.15.7
istiod version: 1.15.7
istiod version: 1.15.7
istiod version: 1.15.7
istiod version: 1.16.5
istiod version: 1.16.5
istiod version: 1.18.2
istiod version: 1.18.2
istiod version: 1.18.2
data plane version: 1.16.5 (83 proxies)
---
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.10", GitCommit:"5c3a0bd0f4d44c0f0200eef6eed403c48d99fcfa", GitTreeState:"clean", BuildDate:"2023-06-12T18:45:58Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Additional Information
bug-report.log
Affected product area

[ ] Ambient
[ ] Docs
[X] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[X] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Control Plane Revisions

I think currently it's assuming a fixed value 15014.

so we have to run multiple versions of istio control plane with the same port? since we are using hostNetwork scheduling pods with same host port will be hard. Is there a way around?

so we have to run multiple versions of istio control plane with the same port? since we are using hostNetwork scheduling pods with same host port will be hard. Is there a way around?

Maybe istioctl x ps with similar functionality?

so we have to run multiple versions of istio control plane with the same port? since we are using hostNetwork scheduling pods with same host port will be hard. Is there a way around?

@Jotheeswaran-Lakshmanan Maybe istioctl x ps with similar functionality?

Same error
./istioctl x ps
2023-08-23T09:00:18.966232Z     error   klog    an error occurred forwarding 52028 -> 15012: error forwarding port 15012 to pod 0c813b77586e4cc5329f4cc892321db4d374aa2e9169dc6b4aff356f5fe341ab, uid : failed to execute portforward in network namespace "host": failed to connect to localhost:15012 inside namespace "0c813b77586e4cc5329f4cc892321db4d374aa2e9169dc6b4aff356f5fe341ab", IPv4: dial tcp4 127.0.0.1:15012: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
Error: could not get XDS from discovery pod "istiod-1-15-5bd77bfb78-87x75": ADSC: failed running rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"

is running 3 versions of Istiod Control plane a regular status for you? Istio does not encourage such usage

Running two version is the regular state and I suppose istio supports canary deployment of control plane with revision tags and in few case we had to run 3 control planes

so we have to run multiple versions of istio control plane with the same port? since we are using hostNetwork scheduling pods with same host port will be hard. Is there a way around?

@Jotheeswaran-Lakshmanan Maybe istioctl x ps with similar functionality?

Same error
./istioctl x ps 2023-08-23T09:00:18.966232Z error klog an error occurred forwarding 52028 -> 15012: error forwarding port 15012 to pod 0c813b77586e4cc5329f4cc892321db4d374aa2e9169dc6b4aff356f5fe341ab, uid : failed to execute portforward in network namespace "host": failed to connect to localhost:15012 inside namespace "0c813b77586e4cc5329f4cc892321db4d374aa2e9169dc6b4aff356f5fe341ab", IPv4: dial tcp4 127.0.0.1:15012: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found Error: could not get XDS from discovery pod "istiod-1-15-5bd77bfb78-87x75": ADSC: failed running rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"

There's a flag to specify the xds port.

so we have to run multiple versions of istio control plane with the same port? since we are using hostNetwork scheduling pods with same host port will be hard. Is there a way around?

@Jotheeswaran-Lakshmanan Maybe istioctl x ps with similar functionality?

Same error
./istioctl x ps 2023-08-23T09:00:18.966232Z error klog an error occurred forwarding 52028 -> 15012: error forwarding port 15012 to pod 0c813b77586e4cc5329f4cc892321db4d374aa2e9169dc6b4aff356f5fe341ab, uid : failed to execute portforward in network namespace "host": failed to connect to localhost:15012 inside namespace "0c813b77586e4cc5329f4cc892321db4d374aa2e9169dc6b4aff356f5fe341ab", IPv4: dial tcp4 127.0.0.1:15012: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found Error: could not get XDS from discovery pod "istiod-1-15-5bd77bfb78-87x75": ADSC: failed running rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"

There's a flag to specify the xds port.

Thanks, specifying xds port along with revision fixed it
./istioctl x ps -r 1-15 --xds-port 15212
NAME                                                    CLUSTER          CDS        LDS        EDS        RDS        ECDS         ISTIOD                           VERSION
bakeryapp2-deployment-2-79f5645ccc-8ltf8.bakeryapp2     ecp-weu-poc1     SYNCED     SYNCED     SYNCED     SYNCED     NOT_SENT     istiod-1-15-5bd77bfb78-qpbvl     1.15.7
bakeryapp2-deployment-74c9d998c5-2rlhw.bakeryapp2       ecp-weu-poc1     SYNCED     SYNCED     SYNCED     SYNCED     NOT_SENT     istiod-1-15-5bd77bfb78-ksbtq     1.15.7

thanks for the help, will close the issue

For someone who comes searching about this later - its the secureGRPCAddr param value from istiod pods that we specify as --xds-port param to istioctl x ps


text:
Add GKE flag

Getting istio ingress with "kubectl get po" is not compatible with gke cluster, we want to have a flag to choose in which way we get ingress.

Jenkins job istio/presubmit passed

When manager#562 is merged, you no longer need this flag. You will need to do "kubectl get ingress", and get the ADRESS and PORT from there.

source: gharchive/pull-request

authors: andraxylia istio-testing yutongz


text:
Strip out the prefix 'docker.io' when loading images in kind

If an image starts with docker.io/, docker strips it from the result of
docker images. Then those images were not being loaded into kind cluster.

@jwendell: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
d1983104d8a93a18a66e683bc2654250bc8bb550
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@jwendell: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
d1983104d8a93a18a66e683bc2654250bc8bb550
link
/test lint_istio

integ-istioio-k8s-tests_istio
d1983104d8a93a18a66e683bc2654250bc8bb550
link
/test integ-istioio-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-istioio-k8s-tests_istio

@jwendell: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-istioio-k8s-tests_istio
ff83a9ab418bfe5b43368aefb6c39050c3e64566
link
/test integ-istioio-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-istioio-k8s-tests_istio

authors: brian-avery istio-testing jwendell