text:
Pilot sending repeated pushes due to listener conflicts when setting up secure egress to TCP services

Bug description
After upgrading from 1.3.3 to 1.4.0 we see that discover is using all avaiable cpu.
$ kubectl top pods -n istio-system
NAME                                      CPU(cores)   MEMORY(bytes)
istio-citadel-95ddcf76d-n4ldx             1m           10Mi
istio-galley-85547b9f85-45kpj             15m          23Mi
istio-galley-85547b9f85-r9k29             17m          25Mi
istio-ingressgateway-85f4b89644-85qsb     83m          43Mi
istio-ingressgateway-85f4b89644-9n6j2     84m          34Mi
istio-ingressgateway-85f4b89644-h5x7f     109m         45Mi
istio-ingressgateway-85f4b89644-h87mf     93m          43Mi
istio-ingressgateway-85f4b89644-lfxrv     121m         41Mi
istio-pilot-74d9bfc476-5v45n              2544m        73Mi
istio-policy-789b7c796f-zps4b             4m           45Mi
istio-sidecar-injector-784fd748df-44h99   16m          8Mi
istio-sidecar-injector-784fd748df-mqm6l   19m          6Mi
istio-telemetry-7d994c7db6-s5p4x          26m          53Mi
kiali-b8f5d89df-5pqzc                     65m          35Mi
prometheus-7f7d884cb5-rgp9s               43m          801Mi

Mostly it is constantly logging updates like the following. It is logging approximately 500 times per second:
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.423593Z	info	ads	CDS: PUSH for node:istio-ingressgateway-85f4b89644-h5x7f.istio-system clusters:188 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.427651Z	info	ads	RDS: PUSH for node:hapi-fhir-server-77d85768c7-b9kss.ehealth-public routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.427733Z	info	ads	Pushing sidecar~10.150.31.70~hapi-fhir-server-77d85768c7-b9kss.ehealth-public~ehealth-public.svc.cluster.local-21
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.429174Z	info	ads	CDS: PUSH for node:hapi-fhir-server-77d85768c7-b9kss.ehealth-public clusters:70 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.429742Z	info	ads	RDS: PUSH for node:mq-proxy-66877d7c49-z7c2t.ehealth-public routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.429829Z	info	ads	EDS: PUSH for node:hapi-fhir-server-77d85768c7-b9kss.ehealth-public clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.432677Z	info	ads	RDS: PUSH for node:keycloak-779d46c89f-9pbrz.ehealth-public routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.432978Z	info	ads	EDS: PUSH for node:istio-ingressgateway-85f4b89644-h5x7f.istio-system clusters:170 endpoints:200 empty:[outbound_.80_._.client-demo.ehealth-public.svc.cluster.local outbound_.80_._.topiclistener.ehealth-public.svc.cluster.local outbound|5432||fut-trifork-rds-aurora.cluster-circogc45ikj.eu-west-1.rds.amazonaws.com outbound|5432||fut-trifork-rds-aurora.cluster-ro-circogc45ikj.eu-west-1.rds.amazonaws.com outbound|5671||b-7f7f6301-eb2d-4eb6-a03a-bb4e08b8c609-1.mq.eu-west-1.amazonaws.com outbound|5671||b-7f7f6301-eb2d-4eb6-a03a-bb4e08b8c609-2.mq.eu-west-1.amazonaws.com outbound|61617||b-7f7f6301-eb2d-4eb6-a03a-bb4e08b8c609-1.mq.eu-west-1.amazonaws.com outbound|61617||b-7f7f6301-eb2d-4eb6-a03a-bb4e08b8c609-2.mq.eu-west-1.amazonaws.com outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.433341Z	info	ads	LDS: PUSH for node:istio-ingressgateway-85f4b89644-h5x7f.istio-system listeners:2
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.433783Z	info	ads	CDS: PUSH for node:careplan-67b4b9f85b-p928j.ehealth-public clusters:70 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.438814Z	info	ads	RDS: PUSH for node:istio-ingressgateway-85f4b89644-h5x7f.istio-system routes:2
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.438873Z	info	ads	LDS: PUSH for node:hapi-fhir-server-77d85768c7-b9kss.ehealth-public listeners:30
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.438983Z	info	ads	Pushing sidecar~10.150.83.176~keycloak-779d46c89f-9pbrz.ehealth-public~ehealth-public.svc.cluster.local-10
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.439842Z	info	ads	EDS: PUSH for node:careplan-67b4b9f85b-p928j.ehealth-public clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.440477Z	info	ads	CDS: PUSH for node:keycloak-779d46c89f-9pbrz.ehealth-public clusters:71 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.442508Z	info	ads	EDS: PUSH for node:keycloak-779d46c89f-9pbrz.ehealth-public clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.444033Z	info	ads	LDS: PUSH for node:auditlog-consumer-764f8dc74d-rnkqr.ehealth-private listeners:30
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.457111Z	info	ads	LDS: PUSH for node:keycloak-779d46c89f-9pbrz.ehealth-public listeners:31
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.457772Z	info	ads	RDS: PUSH for node:hapi-fhir-server-77d85768c7-b9kss.ehealth-public routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.468302Z	info	ads	LDS: PUSH for node:sosi-gateway-69ff9c549-2wp4f.ehealth-private listeners:31
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.468587Z	info	ads	RDS: PUSH for node:keycloak-779d46c89f-9pbrz.ehealth-public routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.471926Z	info	ads	RDS: PUSH for node:auditlog-consumer-764f8dc74d-rnkqr.ehealth-private routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.475655Z	info	ads	RDS: PUSH for node:sosi-gateway-69ff9c549-2wp4f.ehealth-private routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.477736Z	info	ads	Pushing sidecar~10.150.31.35~cpr2fhir-57b9956fb5-zv2df.ehealth-private~ehealth-private.svc.cluster.local-16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.478338Z	info	ads	CDS: PUSH for node:keyservice-76b6567746-62r92.ehealth-public clusters:70 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.479352Z	info	ads	CDS: PUSH for node:cpr2fhir-57b9956fb5-zv2df.ehealth-private clusters:70 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.479443Z	info	ads	EDS: PUSH for node:keyservice-76b6567746-62r92.ehealth-public clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.480514Z	info	ads	Pushing sidecar~10.150.35.205~patient-5b4bc7cfb-z92k2.ehealth-public~ehealth-public.svc.cluster.local-20
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.481115Z	info	ads	Pushing sidecar~10.150.13.115~sor-organization-import-6bc5fcf875-swjml.ehealth-public~ehealth-public.svc.cluster.local-27
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.483152Z	info	ads	EDS: PUSH for node:cpr2fhir-57b9956fb5-zv2df.ehealth-private clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.484626Z	info	ads	CDS: PUSH for node:sor-organization-import-6bc5fcf875-swjml.ehealth-public clusters:70 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.485679Z	info	ads	EDS: PUSH for node:sor-organization-import-6bc5fcf875-swjml.ehealth-public clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.489346Z	info	ads	LDS: PUSH for node:careplan-67b4b9f85b-p928j.ehealth-public listeners:30
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.489557Z	info	ads	LDS: PUSH for node:keyservice-76b6567746-62r92.ehealth-public listeners:30
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.497868Z	info	ads	CDS: PUSH for node:patient-5b4bc7cfb-z92k2.ehealth-public clusters:70 services:92 version:2019-11-27T09:43:55Z/6719
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.499072Z	info	ads	EDS: PUSH for node:patient-5b4bc7cfb-z92k2.ehealth-public clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.500944Z	info	ads	LDS: PUSH for node:cpr2fhir-57b9956fb5-zv2df.ehealth-private listeners:30
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.501206Z	info	ads	RDS: PUSH for node:keyservice-76b6567746-62r92.ehealth-public routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.501386Z	info	ads	RDS: PUSH for node:careplan-67b4b9f85b-p928j.ehealth-public routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.504988Z	info	ads	RDS: PUSH for node:cpr2fhir-57b9956fb5-zv2df.ehealth-private routes:16
istio-pilot-74d9bfc476-5v45n discovery 2019-11-27T09:43:55.505709Z	info	ads	EDS: PUSH for node:vdx2fhir-bbc4958b9-bxtr8.ehealth-private clusters:42 endpoints:56 empty:[outbound|80||client-demo.ehealth-public.svc.cluster.local outbound|80||topiclistener.ehealth-public.svc.cluster.local]

Expected behavior
Steps to reproduce the bug
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
$ istioctl version --remote
client version: 1.2.4
citadel version: 1.4.0
galley version: 1.4.0
galley version: 1.4.0
ingressgateway version: 1.4.0
ingressgateway version: 1.4.0
ingressgateway version: 1.4.0
ingressgateway version: 1.4.0
ingressgateway version: 1.4.0
pilot version: 1.4.0
policy version: 1.4.0
sidecar-injector version: 1.4.0
sidecar-injector version: 1.4.0
telemetry version: 1.4.0

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T16:54:35Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-eks-b7174d", GitCommit:"b7174db5ee0e30c94a0b9899c20ac980c0850fc8", GitTreeState:"clean", BuildDate:"2019-10-18T17:56:01Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

$ helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

How was Istio installed?
Helm.
Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS.

The problem seems to be a misconfiguration:
The logs contains a number of pilot_conflict_outbound_listener_tcp_over_current_tcp , pilot_duplicate_envoy_clusters and pilot_eds_no_instances instances.
We've configured TCP egress like so:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: amq-tcp
spec:
  hosts:
  - activemq-hash-1.mq.eu-west-1.amazonaws.com
  - activemq-hash-2.mq.eu-west-1.amazonaws.com
  tls:
  - match:
    - port: 61617
      sniHosts:
      - activemq-hash-1.mq.eu-west-1.amazonaws.com
    route:
    - destination:
        host: activemq-hash-1.mq.eu-west-1.amazonaws.com
        port:
          number: 61617
  - match:
    - port: 61617
      sniHosts:
      - activemq-hash-2.mq.eu-west-1.amazonaws.com
    route:
    - destination:
        host: activemq-hash-2.mq.eu-west-1.amazonaws.com
        port:
          number: 61617
  - match:
    - port: 5671
      sniHosts:
      - activemq-hash-1.mq.eu-west-1.amazonaws.com
    route:
    - destination:
        host: activemq-hash-1.mq.eu-west-1.amazonaws.com
        port:
          number: 5671
  - match:
    - port: 5671
      sniHosts:
      - activemq-hash-2.mq.eu-west-1.amazonaws.com
    route:
    - destination:
        host: activemq-hash-2.mq.eu-west-1.amazonaws.com
        port:
          number: 5671

---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: amq-tcp
spec:
  addresses:
  - 10.0.160.0/21
  - 10.0.168.0/21
  - 10.0.176.0/21
  hosts:
  - activemq-hash-1.mq.eu-west-1.amazonaws.com
  - activemq-hash-2.mq.eu-west-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: tcp-61617
    number: 61617
    protocol: tcp
  - name: tcp-5671
    number: 5671
    protocol: tcp
  resolution: NONE

We reached this configuration after a lot of trial and error.
Is there a better way to configure secure egress to TCP services? Ideally I would like to not have to specify the specific CIDR ranges.

@vadimeisenbergibm can you help with the above config?

@lhotrifork cc @howardjohn
It seems there is a confusion between TCP and TLS routing. If the traffic is TLS with SNI, you should configure it as TLS. Use tls- in the names of the ports, and define the hosts only in the service entries, without addresses. Then you can omit the VirtualService. The configuration will be the same as https://istio.io/docs/tasks/traffic-management/egress/egress-control/#access-an-external-https-service, just use tls in the name of the port and in the protocol.

Hi @vadimeisenbergibm,
Thanks for your suggestion. I tried the following configuration, but had no success.
I had to revert to my previous configuration using ServiceEntry + VirtualService which triggered the CPU spike in Pilot again.
# Source: proxy/templates/serviceentry-virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: amq-tls
spec:
  hosts:
    - activemq-hash-1.mq.myregion.amazonaws.com
    - activemq-hash-2.mq.myregion.amazonaws.com
  ports:
  - number: 61617
    name: tls-61617
    protocol: TLS
  - number: 5671
    name: tls-5671
    protocol: TLS
  location: MESH_EXTERNAL
  resolution: DNS

We're running with mTLS enabled - do we need DestinationRules for our ServiceEntries to disable mTLS for egress traffic to these TLS services?

Hi @lhotrifork. The ServiceEntry-for-TLS configuration should work, let's debug it. First, I would like to verify: the applications talk to the activemq services by TLS, right?

Please split the service entry into two, a service entry per host
Define a DestinationRule to disable mTLS. I do not think it is required, just to be on the safe side.

If it still does not work, please dump the Envoy configuration of a pod that you want to access the active MQ. See https://istio.io/docs/ops/diagnostic-tools/proxy-cmd/#deep-dive-into-envoy-configuration . Paste the listeners on the ports 61617 and the clusters referenced by these listeners.

Hi @vadimeisenbergibm,
Thanks for taking the time to help debug this problem.
I've disabled mTLS for the implicated service using a Policy (server-side) and DestinationRule (client-side).
I've split the ServiceEntry into two and deleted the previous VirtualService.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: amq-tcp-0
  namespace: istio-system
spec:
  hosts:
  - HASH-1.mq.eu-west-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: tls-61617
    number: 61617
    protocol: TLS
  - name: tls-5671
    number: 5671
    protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: amq-tcp-1
  namespace: istio-system
spec:
  hosts:
  - HASH-2.mq.eu-west-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: tls-61617
    number: 61617
    protocol: TLS
  - name: tls-5671
    number: 5671
    protocol: TLS
  resolution: DNS

Validating TLS
I've validated TLS on both ports:
$ openssl s_client -crlf -connect HASH1-1.mq.eu-west-1.amazonaws.com:61617
CONNECTED(00000003)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = *.mq.eu-west-1.amazonaws.com
verify return:1
---
Certificate chain
 0 s:CN = *.mq.eu-west-1.amazonaws.com
   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
   i:C = US, O = Amazon, CN = Amazon Root CA 1
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFgjCCBGqgAwIBAgIQA5CpZGgz6XVZfbV6Dj547DANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg
Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0xOTEwMTEwMDAwMDBaFw0yMDExMTEx
MjAwMDBaMCcxJTAjBgNVBAMMHCoubXEuZXUtd2VzdC0xLmFtYXpvbmF3cy5jb20w
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCzMmLRV3TdwC7w2UiE0s2G
GCMWI480y53gdRY249c8wfSuBlVRnKBce06ArrNjDiTdFXdJbRRs0yfoeMIopiyu
NvVaCtO8SD6Yc7wjulrH4+9yz+GFlKAi8aSIhqB0AQMg0+I8E50c0jxi1xveXiuW
tYZsSzxVKSAuJStmzldhCaJ6y00bqmhraWP98/48NFhLcucBSpC57IqhYd7cbv8u
S1r0GZcdN5liFFHWu8xVdU3FgVmuZjeIEpl2Eowi7T4wU+ARJqY/5nRVa3HWDxEL
BGS33KGxPAfnsIbfZDEggLMBaZOWtxvDqti+79Dxx2fMbmX/AP2z1X57pcUqXmYH
AgMBAAGjggKJMIIChTAfBgNVHSMEGDAWgBRZpGYGUqB7lZI8o5QHJ5Z0W/k90DAd
BgNVHQ4EFgQUpbHmv0B5E1oKK6aFR65+0V8Da34wJwYDVR0RBCAwHoIcKi5tcS5l
dS13ZXN0LTEuYW1hem9uYXdzLmNvbTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMDsGA1UdHwQ0MDIwMKAuoCyGKmh0dHA6Ly9j
cmwuc2NhMWIuYW1hem9udHJ1c3QuY29tL3NjYTFiLmNybDAgBgNVHSAEGTAXMAsG
CWCGSAGG/WwBAjAIBgZngQwBAgEwdQYIKwYBBQUHAQEEaTBnMC0GCCsGAQUFBzAB
hiFodHRwOi8vb2NzcC5zY2ExYi5hbWF6b250cnVzdC5jb20wNgYIKwYBBQUHMAKG
Kmh0dHA6Ly9jcnQuc2NhMWIuYW1hem9udHJ1c3QuY29tL3NjYTFiLmNydDAMBgNV
HRMBAf8EAjAAMIIBBQYKKwYBBAHWeQIEAgSB9gSB8wDxAHYApLkJkLQYWBSHuxOi
zGdwCjw1mAT5G9+443fNDsgN3BAAAAFtuDQqHwAABAMARzBFAiAdZ8HKP//w1pWV
iuXtJDJvetLQdWJE57o/Ykc93W7ySAIhALSvmAOulBnBvn9/jv0pQ1Zkp8ya3LNV
+yeGHCBJjGOxAHcAh3W/51l8+IxDmV+9827/Vo1HVjb/SrVgwbTq/16ggw8AAAFt
uDQqgQAABAMASDBGAiEAh6KjbMGB++9ZD2/xfp8j/6gUwVYaUCyyjXDjoRJSddEC
IQC5QtmsHy6MmN+tw9rNE/e6JCssXKk9of3TUsmQHDAISTANBgkqhkiG9w0BAQsF
AAOCAQEAtbfdUlABDWpND8YdM7e5r/XSL7soAkj4IHk7hEVKryVqg8OWZe1sBY8U
Hkg0naBjbvaRNN64nLEEg/HlzJcxfal7fXbkmCC+damLI9zI4m8YywnSoMur+kXj
9JbMV8GYL8J5FqTuFc4QGeVf2Ztb40DNQ4uUxJ+u1Tr1HP0ubO5j+kc6u52kbgMa
T45nXsPEcGt1uIuP5jcrjG1diUuDnyQyeU6vTQcuKXP9ane8o7AK0KnMfZgRwBDY
LnNn5DK8eZKDrpYPHYmCEO3JbIHc3cTYOQd+Q4pRYV/UYK1iohP5Unic7hblm/ZL
TyTmjAIiLbP5McMEbkb/BPKh5irJVA==
-----END CERTIFICATE-----
subject=CN = *.mq.eu-west-1.amazonaws.com

issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5331 bytes and written 495 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: SESSIONID-HAS
    Session-ID-ctx:
    Master-Key: MASTERKEY-HASH
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: REDACTED
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
[ SNIP of Openwire connection initializing ]

Repeated with same results for port 5671
Logs from istio-proxy while this is ongoing:
 {
    "bytes_sent": "5710",
    "upstream_cluster": "outbound|61617||HASH1-1.mq.eu-west-1.amazonaws.com",
    "downstream_remote_address": "10.150.9.12:60846",
    "authority": "-",
    "path": "-",
    "protocol": "-",
    "upstream_service_time": "-",
    "upstream_local_address": "10.150.9.12:60848",
    "duration": "26291",
    "downstream_local_address": "10.150.160.14:61617",
    "upstream_transport_failure_reason": "-",
    "route_name": "-",
    "response_code": "0",
    "user_agent": "-",
    "response_flags": "-",
    "start_time": "2019-12-13T13:58:18.803Z",
    "method": "-",
    "request_id": "-",
    "upstream_host": "10.150.160.14:61617",
    "x_forwarded_for": "-",
    "requested_server_name": "-",
    "bytes_received": "495",
    "istio_policy_status": "-"
}
{
    "requested_server_name": "-",
    "bytes_received": "566",
    "istio_policy_status": "-",
    "bytes_sent": "5557",
    "upstream_cluster": "outbound|5671||HASH1-1.mq.eu-west-1.amazonaws.com",
    "downstream_remote_address": "10.150.9.12:41406",
    "authority": "-",
    "path": "-",
    "protocol": "-",
    "upstream_service_time": "-",
    "upstream_local_address": "10.150.9.12:41408",
    "duration": "5166",
    "downstream_local_address": "10.150.160.14:5671",
    "upstream_transport_failure_reason": "-",
    "route_name": "-",
    "response_code": "0",
    "user_agent": "-",
    "response_flags": "-",
    "start_time": "2019-12-13T13:58:53.466Z",
    "method": "-",
    "request_id": "-",
    "upstream_host": "10.150.160.14:5671",
    "x_forwarded_for": "-"
}

So I am (apparently) able to connect to the service using my temporary pod (docker image nicolaka/netshoot, but not my actual service.
Logs from a failing connection:
{
    "bytes_sent": "0",
    "upstream_cluster": "-",
    "downstream_remote_address": "10.150.35.205:59506",
    "authority": "-",
    "path": "-",
    "protocol": "-",
    "upstream_service_time": "-",
    "upstream_local_address": "-",
    "duration": "0",
    "downstream_local_address": "10.150.171.27:61617",
    "upstream_transport_failure_reason": "-",
    "route_name": "-",
    "response_code": "0",
    "user_agent": "-",
    "response_flags": "NR",
    "start_time": "2019-12-13T14:56:49.714Z",
    "method": "-",
    "request_id": "-",
    "upstream_host": "-",
    "x_forwarded_for": "-",
    "requested_server_name": "-",
    "bytes_received": "0",
    "istio_policy_status": "-"
}

Proxy config
$ istioctl proxy-config cluster -n namespace pod-6c866bbb47-jb5wg
SERVICE FQDN                                                                   PORT      SUBSET         DIRECTION     TYPE
HASH-1.mq.eu-west-1.amazonaws.com                                              8162      -              outbound      STRICT_DNS
HASH-2.mq.eu-west-1.amazonaws.com                                              8162      -              outbound      STRICT_DNS
$ istioctl proxy-config listeners -n  namespace pod-6c866bbb47-jb5wg
ADDRESS            PORT      TYPE
10.150.65.41       8080      HTTP
10.150.65.41       15020     TCP
172.20.167.127     15011     TCP
0.0.0.0            443       TCP
0.0.0.0            5432      TCP
0.0.0.0            8200      TCP
0.0.0.0            61617     TCP
172.20.52.112      443       TCP
172.20.220.164     443       TCP
0.0.0.0            5671      TCP
172.20.27.172      443       TCP
0.0.0.0            8162      TCP
0.0.0.0            7600      TCP
0.0.0.0            8060      TCP
0.0.0.0            15010     TCP
0.0.0.0            9090      TCP
172.20.154.67      80        TCP
0.0.0.0            15004     TCP
172.20.0.67        42422     TCP
0.0.0.0            80        TCP
172.20.224.45      44134     TCP
0.0.0.0            15014     TCP
0.0.0.0            9901      TCP
0.0.0.0            8080      TCP
0.0.0.0            9091      TCP
172.20.36.229      44134     TCP
0.0.0.0            20001     TCP
0.0.0.0            24224     TCP
0.0.0.0            15001     TCP
0.0.0.0            15006     TCP
0.0.0.0            15090     HTTP

Envoy config dump
istioctl proxy-config listeners -n namespace pod-6c866bbb47-5w6jg -o json --address 0.0.0.0 --port 61617
[
    {
        "name": "0.0.0.0_61617",
        "address": {
            "socketAddress": {
                "address": "0.0.0.0",
                "portValue": 61617
            }
        },
        "filterChains": [
            {
                "filterChainMatch": {
                    "prefixRanges": [
                        {
                            "addressPrefix": "10.150.76.196",
                            "prefixLen": 32
                        }
                    ]
                },
                "filters": [
                    {
                        "name": "mixer",
                        "typedConfig": {
                            "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                            "transport": {
                                "networkFailPolicy": {
                                    "policy": "FAIL_CLOSE",
                                    "baseRetryWait": "0.080s",
                                    "maxRetryWait": "1s"
                                },
                                "checkCluster": "outbound|15004||istio-policy.istio-system.svc.cluster.local",
                                "reportCluster": "outbound|15004||istio-telemetry.istio-system.svc.cluster.local",
                                "reportBatchMaxEntries": 100,
                                "reportBatchMaxTime": "1s"
                            },
                            "mixerAttributes": {
                                "attributes": {
                                    "context.proxy_version": {
                                        "stringValue": "1.4.0"
                                    },
                                    "context.reporter.kind": {
                                        "stringValue": "outbound"
                                    },
                                    "context.reporter.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    },
                                    "destination.service.host": {
                                        "stringValue": "HASH-1.mq.eu-west-1.amazonaws.com"
                                    },
                                    "destination.service.name": {
                                        "stringValue": "HASH-1.mq.eu-west-1.amazonaws.com"
                                    },
                                    "destination.service.namespace": {
                                        "stringValue": "istio-system"
                                    },
                                    "source.namespace": {
                                        "stringValue": "namespace"
                                    },
                                    "source.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    }
                                }
                            },
                            "disableCheckCalls": true
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "BlackHoleCluster",
                            "cluster": "BlackHoleCluster"
                        }
                    }
                ]
            },
            {
                "filterChainMatch": {
                    "serverNames": [
                        "HASH-1.mq.eu-west-1.amazonaws.com"
                    ]
                },
                "filters": [
                    {
                        "name": "mixer",
                        "typedConfig": {
                            "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                            "transport": {
                                "networkFailPolicy": {
                                    "policy": "FAIL_CLOSE",
                                    "baseRetryWait": "0.080s",
                                    "maxRetryWait": "1s"
                                },
                                "checkCluster": "outbound|15004||istio-policy.istio-system.svc.cluster.local",
                                "reportCluster": "outbound|15004||istio-telemetry.istio-system.svc.cluster.local",
                                "reportBatchMaxEntries": 100,
                                "reportBatchMaxTime": "1s"
                            },
                            "mixerAttributes": {
                                "attributes": {
                                    "context.proxy_version": {
                                        "stringValue": "1.4.0"
                                    },
                                    "context.reporter.kind": {
                                        "stringValue": "outbound"
                                    },
                                    "context.reporter.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    },
                                    "destination.service.host": {
                                        "stringValue": "HASH-1.mq.eu-west-1.amazonaws.com"
                                    },
                                    "destination.service.name": {
                                        "stringValue": "HASH-1.mq.eu-west-1.amazonaws.com"
                                    },
                                    "destination.service.namespace": {
                                        "stringValue": "istio-system"
                                    },
                                    "source.namespace": {
                                        "stringValue": "namespace"
                                    },
                                    "source.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    }
                                }
                            },
                            "disableCheckCalls": true
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "outbound|61617||HASH-1.mq.eu-west-1.amazonaws.com",
                            "cluster": "outbound|61617||HASH-1.mq.eu-west-1.amazonaws.com",
                            "accessLog": [
                                {
                                    "name": "envoy.file_access_log",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                        "path": "/dev/stdout",
                                        "jsonFormat": {
                                            "authority": "%REQ(:AUTHORITY)%",
                                            "bytes_received": "%BYTES_RECEIVED%",
                                            "bytes_sent": "%BYTES_SENT%",
                                            "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
                                            "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
                                            "duration": "%DURATION%",
                                            "istio_policy_status": "%DYNAMIC_METADATA(istio.mixer:status)%",
                                            "method": "%REQ(:METHOD)%",
                                            "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
                                            "protocol": "%PROTOCOL%",
                                            "request_id": "%REQ(X-REQUEST-ID)%",
                                            "requested_server_name": "%REQUESTED_SERVER_NAME%",
                                            "response_code": "%RESPONSE_CODE%",
                                            "response_flags": "%RESPONSE_FLAGS%",
                                            "route_name": "%ROUTE_NAME%",
                                            "start_time": "%START_TIME%",
                                            "upstream_cluster": "%UPSTREAM_CLUSTER%",
                                            "upstream_host": "%UPSTREAM_HOST%",
                                            "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
                                            "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
                                            "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
                                            "user_agent": "%REQ(USER-AGENT)%",
                                            "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%"
                                        }
                                    }
                                }
                            ]
                        }
                    }
                ]
            },
            {
                "filterChainMatch": {},
                "filters": [
                    {
                        "name": "mixer",
                        "typedConfig": {
                            "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                            "transport": {
                                "networkFailPolicy": {
                                    "policy": "FAIL_CLOSE",
                                    "baseRetryWait": "0.080s",
                                    "maxRetryWait": "1s"
                                },
                                "checkCluster": "outbound|15004||istio-policy.istio-system.svc.cluster.local",
                                "reportCluster": "outbound|15004||istio-telemetry.istio-system.svc.cluster.local",
                                "reportBatchMaxEntries": 100,
                                "reportBatchMaxTime": "1s"
                            },
                            "mixerAttributes": {
                                "attributes": {
                                    "context.proxy_version": {
                                        "stringValue": "1.4.0"
                                    },
                                    "context.reporter.kind": {
                                        "stringValue": "outbound"
                                    },
                                    "context.reporter.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    },
                                    "destination.service.host": {
                                        "stringValue": "BlackHoleCluster"
                                    },
                                    "destination.service.name": {
                                        "stringValue": "BlackHoleCluster"
                                    },
                                    "source.namespace": {
                                        "stringValue": "namespace"
                                    },
                                    "source.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    }
                                }
                            },
                            "disableCheckCalls": true
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "BlackHoleCluster",
                            "cluster": "BlackHoleCluster"
                        }
                    }
                ]
            },
            {
                "filterChainMatch": {
                    "serverNames": [
                        "HASH-2.mq.eu-west-1.amazonaws.com"
                    ]
                },
                "filters": [
                    {
                        "name": "mixer",
                        "typedConfig": {
                            "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                            "transport": {
                                "networkFailPolicy": {
                                    "policy": "FAIL_CLOSE",
                                    "baseRetryWait": "0.080s",
                                    "maxRetryWait": "1s"
                                },
                                "checkCluster": "outbound|15004||istio-policy.istio-system.svc.cluster.local",
                                "reportCluster": "outbound|15004||istio-telemetry.istio-system.svc.cluster.local",
                                "reportBatchMaxEntries": 100,
                                "reportBatchMaxTime": "1s"
                            },
                            "mixerAttributes": {
                                "attributes": {
                                    "context.proxy_version": {
                                        "stringValue": "1.4.0"
                                    },
                                    "context.reporter.kind": {
                                        "stringValue": "outbound"
                                    },
                                    "context.reporter.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    },
                                    "destination.service.host": {
                                        "stringValue": "HASH-2.mq.eu-west-1.amazonaws.com"
                                    },
                                    "destination.service.name": {
                                        "stringValue": "HASH-2.mq.eu-west-1.amazonaws.com"
                                    },
                                    "destination.service.namespace": {
                                        "stringValue": "istio-system"
                                    },
                                    "source.namespace": {
                                        "stringValue": "namespace"
                                    },
                                    "source.uid": {
                                        "stringValue": "kubernetes://pod-6c866bbb47-5w6jg.namespace"
                                    }
                                }
                            },
                            "disableCheckCalls": true
                        }
                    },
                    {
                        "name": "envoy.tcp_proxy",
                        "typedConfig": {
                            "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                            "statPrefix": "outbound|61617||HASH-2.mq.eu-west-1.amazonaws.com",
                            "cluster": "outbound|61617||HASH-2.mq.eu-west-1.amazonaws.com",
                            "accessLog": [
                                {
                                    "name": "envoy.file_access_log",
                                    "typedConfig": {
                                        "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                        "path": "/dev/stdout",
                                        "jsonFormat": {
                                            "authority": "%REQ(:AUTHORITY)%",
                                            "bytes_received": "%BYTES_RECEIVED%",
                                            "bytes_sent": "%BYTES_SENT%",
                                            "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
                                            "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
                                            "duration": "%DURATION%",
                                            "istio_policy_status": "%DYNAMIC_METADATA(istio.mixer:status)%",
                                            "method": "%REQ(:METHOD)%",
                                            "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%",
                                            "protocol": "%PROTOCOL%",
                                            "request_id": "%REQ(X-REQUEST-ID)%",
                                            "requested_server_name": "%REQUESTED_SERVER_NAME%",
                                            "response_code": "%RESPONSE_CODE%",
                                            "response_flags": "%RESPONSE_FLAGS%",
                                            "route_name": "%ROUTE_NAME%",
                                            "start_time": "%START_TIME%",
                                            "upstream_cluster": "%UPSTREAM_CLUSTER%",
                                            "upstream_host": "%UPSTREAM_HOST%",
                                            "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
                                            "upstream_service_time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%",
                                            "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
                                            "user_agent": "%REQ(USER-AGENT)%",
                                            "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%"
                                        }
                                    }
                                }
                            ]
                        }
                    }
                ]
            }
        ],
        "deprecatedV1": {
            "bindToPort": false
        },
        "listenerFilters": [
            {
                "name": "envoy.listener.tls_inspector"
            }
        ],
        "listenerFiltersTimeout": "0.100s",
        "continueOnListenerFiltersTimeout": true,
        "trafficDirection": "OUTBOUND"
    }
]

istioctl proxy-config cluster -n namespace pod-6c866bbb47-5w6jg --fqdn HASH-1.mq.eu-west-1.amazonaws.com  -o json
[
    {
        "name": "outbound|8162||HASH-1.mq.eu-west-1.amazonaws.com",
        "type": "STRICT_DNS",
        "connectTimeout": "10s",
        "loadAssignment": {
            "clusterName": "outbound|8162||HASH-1.mq.eu-west-1.amazonaws.com",
            "endpoints": [
                {
                    "lbEndpoints": [
                        {
                            "endpoint": {
                                "address": {
                                    "socketAddress": {
                                        "address": "HASH-1.mq.eu-west-1.amazonaws.com",
                                        "portValue": 8162
                                    }
                                }
                            },
                            "loadBalancingWeight": 1
                        }
                    ],
                    "loadBalancingWeight": 1
                }
            ]
        },
        "circuitBreakers": {
            "thresholds": [
                {
                    "maxConnections": 4294967295,
                    "maxPendingRequests": 4294967295,
                    "maxRequests": 4294967295,
                    "maxRetries": 4294967295
                }
            ]
        },
        "dnsRefreshRate": "300s",
        "respectDnsTtl": true,
        "dnsLookupFamily": "V4_ONLY"
    }
]

I've also tried dumping the endpoints, but don't know if there should be anything listed here, or if this expected behaviour for ServiceEntries.
$ istioctl proxy-config endpoints -n namespace pod-6c866bbb47-5w6jg --cluster HASH-1.mq.eu-west-1.amazonaws.com  -o json
[]
$ istioctl proxy-config endpoints -n namespace pod-6c866bbb47-5w6jg --cluster "outbound|61617||HASH-1.mq.eu-west-1.amazonaws.com"  -o json
[]

It just occurred to me that we also open up for this host on port 8162 in a different ServiceEntry with protocol HTTPS, but using the same two hostnames. Could this maybe the reason why it's not working?
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: amq-console-https
  namespace: istio-system
spec:
  hosts:
  - HASH-1.mq.eu-west-1.amazonaws.com
  - HASH-2.mq.eu-west-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: https-8162
    number: 8162
    protocol: https
  resolution: DNS

If I delete that rule I'm getting this in my istio-proxy logs instead, but still the service doesnt work.
After deletion my clusters look like this:
$ istioctl proxy-config clusters -n namespace pod-68fd84b9cf-2vkch
SERVICE FQDN                                                   PORT      SUBSET         DIRECTION     TYPE
HASH-1.mq.eu-west-1.amazonaws.com            5671      -              outbound      STRICT_DNS
HASH-1.mq.eu-west-1.amazonaws.com            61617     -              outbound      STRICT_DNS
HASH-2.mq.eu-west-1.amazonaws.com            5671      -              outbound      STRICT_DNS
HASH-2.mq.eu-west-1.amazonaws.com            61617     -              outbound      STRICT_DNS

And endpoints:
$ istioctl proxy-config endpoints -n namespace pod-68fd84b9cf-2vkch  --cluster "outbound|61617||HASH-1.mq.eu-west-1.amazonaws.com"  -o json
[
    {
        "name": "outbound|61617||HASH-1.mq.eu-west-1.amazonaws.com",
        "addedViaApi": true,
        "hostStatuses": [
            {
                "address": {
                    "socketAddress": {
                        "address": "10.150.160.14",
                        "portValue": 61617
                    }
                },
                "stats": [
                    {
                        "name": "cx_connect_fail"
                    },
                    {
                        "value": "12",
                        "name": "cx_total"
                    },
                    {
                        "name": "rq_error"
                    },
                    {
                        "name": "rq_success"
                    },
                    {
                        "name": "rq_timeout"
                    },
                    {
                        "value": "12",
                        "name": "rq_total"
                    },
                    {
                        "type": "GAUGE",
                        "name": "cx_active"
                    },
                    {
                        "type": "GAUGE",
                        "name": "rq_active"
                    }
                ],
                "healthStatus": {
                    "edsHealthStatus": "HEALTHY"
                },
                "weight": 1,
                "hostname": "HASH-1.mq.eu-west-1.amazonaws.com"
            }
        ]
    }
]

Logs from the istio-proxy on the service pod.
{
    "protocol": "-",
    "upstream_service_time": "-",
    "upstream_local_address": "10.150.45.8:36162",
    "duration": "3515",
    "downstream_local_address": "10.150.160.14:61617",
    "upstream_transport_failure_reason": "-",
    "route_name": "-",
    "response_code": "0",
    "user_agent": "-",
    "response_flags": "-",
    "start_time": "2019-12-13T15:08:54.758Z",
    "method": "-",
    "request_id": "-",
    "upstream_host": "10.150.160.14:61617",
    "x_forwarded_for": "-",
    "requested_server_name": "HASH-1.mq.eu-west-1.amazonaws.com",
    "bytes_received": "1641",
    "istio_policy_status": "-",
    "bytes_sent": "6112",
    "upstream_cluster": "outbound|61617||HASH-1.mq.eu-west-1.amazonaws.com",
    "downstream_remote_address": "10.150.45.8:36160",
    "authority": "-",
    "path": "-"
}

@lhotrifork OK, so I think we have some progress.
The log entries for the failed requests have "-" in the upstream cluster field, and "NR" in the "response_field". NR means Not Routed, and it usually means that the upstream cluster is missing in the cluster configuration. In your case, the cluster for port 61617 was indeed missing, the dump of clusters for HASH-1.mq.eu-west-1.amazonaws.com showed a cluster for port 8162 but not for 61617.
Once you deleted the service entry for the port 8162, the cluster for port 61617 appeared in the cluster dump, and it seems your requests according to the logs were successful.
So, you should define a single service entry per host, and list there all the ports: 61617, 5671 and 8162.

Hi @vadimeisenbergibm
Thanks for your help.
The ServiceEntry configuration now works for our ActiveMQ. For future reference this was the configuration that worked for us.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: amq-1-tls
spec:
  hosts:
  - HASH-1.mq.eu-west-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: tls-61617
    number: 61617
    protocol: TLS
  - name: tls-5671
    number: 5671
    protocol: TLS
  - name: tls-8162
    number: 8162
    protocol: TLS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: amq-2-tls
spec:
  hosts:
  - HASH-2.mq.eu-west-1.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: tls-61617
    number: 61617
    protocol: TLS
  - name: tls-5671
    number: 5671
    protocol: TLS
  - name: tls-8162
    number: 8162
    protocol: TLS
  resolution: DNS

Sadly this was not possible for Postgresql.
From what I've gathered on the internet the Postgresql protocol does not implement SNI Headers (or does something non-standard).
Using openssl s_client -connect I believe this to be the case
$ openssl s_client -crlf -connect my-rds-cluster.amazonaws.com:5432 -servername my-rds-cluster.eu-west-1.rds.amazonaws.com
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 367 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

However using the psql command we're successfully able to connect using TLS (as long as I include the CA cert).
$ PGSSLMODE=verify-full psql -h my-rds-cluster.eu-west-1.rds.amazonaws.com -U root postgres
psql: error: could not connect to server: root certificate file "/root/.postgresql/root.crt" does not exist
Either provide the file or change sslmode to disable server certificate verification.
$ PGSSLROOTCERT=root.pem PGSSLMODE=verify-full psql -h my-rds-cluster.rds.amazonaws.com -U username postgres
Password for user username:
psql (12.1, server 10.7)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=>

Final question: In this case Istio does require both ServiceEntry and VirtualService?

@lhotrifork For TLS-with-SNI traffic, Istio does not require ServiceEntry and VirtualService, SeviceEntry is enough.
For TLS-without-SNI, you can control the traffic by treating it as TCP. See https://istio.io/blog/2018/egress-tcp/.

authors: howardjohn lhotrifork vadimeisenbergibm

repo: istio/istio


text:
Communication issues from VM using mesh expansion to mesh cluster

Bug description
Communication from a VM with mesh expansion installed can't connect to a remote service inside a kubernetes cluster with Istio's service mesh.
The main problem identified is that the iptables are not set.
On Kubernetes:

Installed Istio 1.4.3
Enabled mesh expansion
Enabled MTLS cluster wide

On the VM:

Installed the .deb package using dpkg
Extracted the systemd config to the right path
CSR signing using node_agent is ok
Envoy has started
/etc/hosts has been updated with the IP of the target service
etc/hosts has been updated with the IP of istio-citadel + pilot

What has been tested
Connectivity status

Traffic is being intercepted but its Envoy's admin console that responds.
There seems to be no proxification toward the target service.

Identified problem
Even when executing the /usr/local/bin/istio-iptables.sh manualy, no iptables are created.
Checking sudo iptables -L returns nothing:
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

With no rules set, incomming requests can't be redirected to Envoy. Is this normal ?
Envoy not listening
In /var/lib/istio/envoy/sidecar.env updating ISTIO_INBOUD_INTERCEPTION_MODE from REDIRECT (default) to TPROXY allows Envoy to listen on 127.0.0.1:8080 and mapping to the external service is ok.
$ curl 127.0.0.1:15000/clusters
...
outbound|80||access-prometheus.cattle-prometheus.svc.cluster.local::10.233.68.146:8080::canary::false
outbound|80||access-prometheus.cattle-prometheus.svc.cluster.local::10.233.68.146:8080::success_rate::-1
inbound|8080http|vmhttp.myns.svc.cluster.local::default_priority::max_connections::4294967295
inbound|8080http|vmhttp.myns.svc.cluster.local::default_priority::max_pending_requests::4294967295
...

Affected product area (please put an X in all that apply)
Expected behavior
Iptables should be set.
Communication from the VM should reach the service in the mesh cluster.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
$ istioctl version --remote
client version: 1.4.3
control plane version: 1.4.3
data plane version:  (1 proxies), 1.4.3 (6 proxies)

How was Istio installed?
Kubernetes: Using Helm
VM: Using dpkg
Environment where bug was observed (cloud vendor, OS, etc)
On premise
Kubernetes : CentOs 7
VM : Ubuntu 14.04 (CSR signing is not working on Debian 9)

Do you have logs from istio-iptables.sh possibly? That seems to be the root cause here

$ sudo /usr/local/bin/istio-iptables.sh
Environment:
------------
ENVOY_PORT=15000
ISTIO_INBOUND_INTERCEPTION_MODE=TPROXY
ISTIO_INBOUND_TPROXY_MARK=1337
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=133
ISTIO_INBOUND_PORTS=8080
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=10.233.0.0/18
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15000
INBOUND_CAPTURE_PORT=15000
PROXY_UID=106,0
INBOUND_INTERCEPTION_MODE=TPROXY
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=8080
INBOUND_PORTS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=10.233.0.0/18
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=

+ iptables -t nat -N ISTIO_REDIRECT
+ iptables -t nat -A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15000
+ iptables -t nat -N ISTIO_IN_REDIRECT
+ iptables -t nat -A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15000
+ '[' -n 8080 ']'
+ '[' TPROXY = TPROXY ']'
+ iptables -t mangle -N ISTIO_DIVERT
+ iptables -t mangle -A ISTIO_DIVERT -j MARK --set-mark 1337
+ iptables -t mangle -A ISTIO_DIVERT -j ACCEPT
+ ip -f inet rule add fwmark 1337 lookup 133
+ ip -f inet route add local default dev lo table 133
RTNETLINK answers: File exists
+ ip route show table all
local default dev lo  table 133  scope host
default via 10.10.5.1 dev eth1
10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.15
10.10.5.0/24 dev eth1  proto kernel  scope link  src 10.10.5.250
broadcast 10.0.2.0 dev eth0  table local  proto kernel  scope link  src 10.0.2.15
local 10.0.2.15 dev eth0  table local  proto kernel  scope host  src 10.0.2.15
broadcast 10.0.2.255 dev eth0  table local  proto kernel  scope link  src 10.0.2.15
broadcast 10.10.5.0 dev eth1  table local  proto kernel  scope link  src 10.10.5.250
local 10.10.5.250 dev eth1  table local  proto kernel  scope host  src 10.10.5.250
broadcast 10.10.5.255 dev eth1  table local  proto kernel  scope link  src 10.10.5.250
broadcast 127.0.0.0 dev lo  table local  proto kernel  scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo  table local  proto kernel  scope host  src 127.0.0.1
local 127.0.0.1 dev lo  table local  proto kernel  scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo  table local  proto kernel  scope link  src 127.0.0.1
local ::1 dev lo  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev eth1  proto kernel  metric 256
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101
local ::1 dev lo  table local  proto none  metric 0
local fe80::a00:27ff:fe1f:d2eb dev lo  table local  proto none  metric 0
local fe80::a00:27ff:fe61:a085 dev lo  table local  proto none  metric 0
ff00::/8 dev eth0  table local  metric 256
ff00::/8 dev eth1  table local  metric 256
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101
+ iptables -t mangle -N ISTIO_TPROXY
+ iptables -t mangle -A ISTIO_TPROXY '!' -d 127.0.0.1/32 -p tcp -j TPROXY --tproxy-mark 1337/0xffffffff--on-port 15000
+ table=mangle
+ iptables -t mangle -N ISTIO_INBOUND
+ iptables -t mangle -A PREROUTING -p tcp -j ISTIO_INBOUND
+ '[' 8080 == '*' ']'
+ for port in '${INBOUND_PORTS_INCLUDE}'
+ '[' TPROXY = TPROXY ']'
+ iptables -t mangle -A ISTIO_INBOUND -p tcp --dport 8080 -m socket -j ISTIO_DIVERT
+ iptables -t mangle -A ISTIO_INBOUND -p tcp --dport 8080 -m socket -j ISTIO_DIVERT
+ iptables -t mangle -A ISTIO_INBOUND -p tcp --dport 8080 -j ISTIO_TPROXY
+ iptables -t nat -N ISTIO_OUTPUT
+ iptables -t nat -A OUTPUT -p tcp -j ISTIO_OUTPUT
+ '[' -n '' ']'
+ '[' -z '' ']'
+ iptables -t nat -A ISTIO_OUTPUT -o lo '!' -d 127.0.0.1/32 -j ISTIO_REDIRECT
+ for uid in '${PROXY_UID}'
+ iptables -t nat -A ISTIO_OUTPUT -m owner --uid-owner 106 -j RETURN
+ for uid in '${PROXY_UID}'
+ iptables -t nat -A ISTIO_OUTPUT -m owner --uid-owner 0 -j RETURN
+ for gid in '${PROXY_GID}'
+ iptables -t nat -A ISTIO_OUTPUT -m owner --gid-owner 106 -j RETURN
+ for gid in '${PROXY_GID}'
+ iptables -t nat -A ISTIO_OUTPUT -m owner --gid-owner 0 -j RETURN
+ iptables -t nat -A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
+ '[' 0 -gt 0 ']'
+ '[' 1 -gt 0 ']'
+ '[' 10.233.0.0/18 == '*' ']'
+ for cidr in '"${ipv4_ranges_include[@]}"'
+ iptables -t nat -A ISTIO_OUTPUT -d 10.233.0.0/18 -j ISTIO_REDIRECT
+ iptables -t nat -A ISTIO_OUTPUT -j RETURN
+ set +o nounset
+ '[' -n '' ']'
+ ip6tables -F INPUT
+ ip6tables -A INPUT -m state --state ESTABLISHED -j ACCEPT
+ ip6tables -A INPUT -i lo -d ::1 -j ACCEPT
+ ip6tables -A INPUT -j REJECT
+ dump
+ iptables-save
# Generated by iptables-save v1.4.21 on Mon Feb 24 15:53:56 2020
*filter
:INPUT ACCEPT [52102:67338527]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [40884:15597731]
COMMIT
# Completed on Mon Feb 24 15:53:56 2020
# Generated by iptables-save v1.4.21 on Mon Feb 24 15:53:56 2020
*mangle
:PREROUTING ACCEPT [69:2760]
:INPUT ACCEPT [69:2760]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [70:5352]
:POSTROUTING ACCEPT [70:5352]
:ISTIO_DIVERT - [0:0]
:ISTIO_INBOUND - [0:0]
:ISTIO_TPROXY - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_DIVERT -j MARK --set-xmark 0x539/0xffffffff
-A ISTIO_DIVERT -j ACCEPT
-A ISTIO_INBOUND -p tcp -m tcp --dport 8080 -m socket -j ISTIO_DIVERT
-A ISTIO_INBOUND -p tcp -m tcp --dport 8080 -m socket -j ISTIO_DIVERT
-A ISTIO_INBOUND -p tcp -m tcp --dport 8080 -j ISTIO_TPROXY
-A ISTIO_TPROXY ! -d 127.0.0.1/32 -p tcp -j TPROXY --on-port 15000 --on-ip 0.0.0.0 --tproxy-mark 0x539/0xffffffff
COMMIT
# Completed on Mon Feb 24 15:53:56 2020
# Generated by iptables-save v1.4.21 on Mon Feb 24 15:53:56 2020
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15000
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 106 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 0 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 106 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 0 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -d 10.233.0.0/18 -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15000
COMMIT
# Completed on Mon Feb 24 15:53:56 2020
+ ip6tables-save
# Generated by ip6tables-save v1.4.21 on Mon Feb 24 15:53:56 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED -j ACCEPT
-A INPUT -d ::1/128 -i lo -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-port-unreachable
COMMIT
# Completed on Mon Feb 24 15:53:56 2020

There's two errors hidding in there...
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101
unreachable default dev lo  table unspec  proto kernel  metric 4294967295  error -101

But still no iptables anywhere to be seen
$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

@rlenglet your expertise would be useful here if you have some time

@Docteur-RS
The output of iptables-save before the script returns clearly shows that rules are indeed installed.
Executing iptables -L without -t <table> outputs the rules in table filter (default table), which is not used in Istio. Istio uses tables nat and mangle.
You need to explicitly pass iptables -L -t nat and iptables -L -t mangle. Or just run iptables-save to dump all tables.

One way to debug iptables is to dump the rules with the counters, try to open some connections, dump again with counters, and compare to find which rules are being hit:
iptables -Z -t nat
iptables -L -n -x -v -t nat > before.txt
iptables -Z -t mangle
iptables -L -n -x -v -t mangle >> before.txt
... # Try to open a connection. Do it a unique number of times, e.g. 3 or 7 times, to make it easier to find the rules that are being hit.
iptables -L -n -x -v -t nat > after.txt
iptables -L -n -x -v -t mangle >> after.txt
diff -Naurd before.txt after.txt

@Docteur-RS can you try this and post the result?

Thanks for the test you provided.

Gist of before.txt here

Gist of after.txt here

Gist of the diff between the two here

I made 3 curls from a pod in the mesh (in Kubernetes) toward the Debian VM:
 curl 10.10.5.240:8080 -H "Host: vmhttp.myns.svc.cluster.local" -vv

source: gharchive/issue

authors: Docteur-RS howardjohn rlenglet

license_type: permissive

license_source: github-api


text:
Istio-ingressgateway pod memory and CPU increase slowly

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities/
To ask questions about how to use Istio, please visit https://discuss.istio.io
)
Bug description
istio-ingressgateway pods memory/cpu usage becomes high after some time(usually serveral hours). The issue could happen randomly. We have 10 pods running but usually can see 2 or 3 of them become abnormal. The traffic is light to the pods.
Affected product area (please put an X in all that apply)
[X] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Affected features (please put an X in all that apply)
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane
Expected behavior
Istio ingressgate pods memory/cpu usage is stable.
Steps to reproduce the bug
Difficult to reproduce. It happens to 2 of our 3 production clusters. Make the deployment and the memory/cpu usage will increase by time until OOM.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
Istio 1.6.3
How was Istio installed?
istioctl
Environment where bug was observed (cloud vendor, OS, etc)
Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.
I've added the profiler info to this issue: https://www.dropbox.com/s/eq9fnnsia71i4ch/envoy.tar.gz?dl=0
Not sure if this is related, during our investigation, we also found following errors:
2020-07-13T11:57:34.765656Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x7
2020-07-13T11:57:34.765670Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
2020-07-13T11:57:34.765672Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: 9085616b434c0e0b54cc677cfd3b97a0fed71c0c/1.14.1/Clean/RELEASE/BoringSSL
2020-07-13T11:57:34.765792Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __restore_rt [0x7fabe3993890]
2020-07-13T11:57:34.769236Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: Envoy::Extensions::TransportSockets::Tls::ContextConfigImpl::~ContextConfigImpl() [0x55e5a68bbf94]
2020-07-13T11:57:34.772747Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: Envoy::Extensions::TransportSockets::Tls::ClientContextConfigImpl::~ClientContextConfigImpl() [0x55e5a68bda62]
2020-07-13T11:57:34.776189Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: Envoy::Extensions::TransportSockets::Tls::ClientSslSocketFactory::~ClientSslSocketFactory() [0x55e5a68b8307]
2020-07-13T11:57:34.779578Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: std::__1::allocator<>::destroy() [0x55e5a7f8946a]
2020-07-13T11:57:34.782959Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: Envoy::Upstream::TransportSocketMatcherImpl::~TransportSocketMatcherImpl() [0x55e5a7f8934f]
2020-07-13T11:57:34.786275Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() [0x55e5a7f518d2]
2020-07-13T11:57:34.789557Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() [0x55e5a7f5192e]
2020-07-13T11:57:34.792739Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Upstream::HostDescriptionImpl::~HostDescriptionImpl() [0x55e5a693e262]
2020-07-13T11:57:34.795961Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Upstream::HostImpl::~HostImpl() [0x55e5a6945dfd]
2020-07-13T11:57:34.799245Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: std::__1::__shared_ptr_pointer<>::__on_zero_shared() [0x55e5a693ec3e]
2020-07-13T11:57:34.802531Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: Envoy::Upstream::HostSetImpl::~HostSetImpl() [0x55e5a6fa9236]
2020-07-13T11:57:34.805907Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::Upstream::HostSetImpl::~HostSetImpl() [0x55e5a7f5148e]
2020-07-13T11:57:34.809127Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Upstream::PrioritySetImpl::~PrioritySetImpl() [0x55e5a6932cf0]
2020-07-13T11:57:34.812473Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: Envoy::Upstream::ClusterManagerImpl::ThreadLocalClusterManagerImpl::ClusterEntry::~ClusterEntry() [0x55e5a7e0002e]
2020-07-13T11:57:34.815650Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: absl::container_internal::raw_hash_set<>::erase() [0x55e5a7e06a2a]
2020-07-13T11:57:34.818815Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #16: Envoy::Event::DispatcherImpl::runPostCallbacks() [0x55e5a7db4d26]
2020-07-13T11:57:34.822068Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #17: event_process_active_single_queue [0x55e5a81fdd16]
2020-07-13T11:57:34.825244Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #18: event_base_loop [0x55e5a81fc89e]
2020-07-13T11:57:34.828599Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #19: Envoy::Server::WorkerImpl::threadRoutine() [0x55e5a7daa084]
2020-07-13T11:57:34.831969Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #20: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke() [0x55e5a82b2a83]
2020-07-13T11:57:34.831990Z	critical	envoy backtrace	[bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #21: start_thread [0x7fabe39886db]
2020-07-13T11:57:35.474232Z	info	sds	resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-07-13T11:57:35.474226Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-07-13T11:57:35.474234Z	info	sds	resource:ROOTCA connection is terminated: rpc error: code = Canceled desc = context canceled
2020-07-13T11:57:35.474245Z	error	sds	Remote side closed connection
2020-07-13T11:57:35.474252Z	error	sds	Remote side closed connection
2020-07-13T11:57:35.474301Z	warn	Envoy proxy is NOT ready: failed to get readiness stats: Get "http://127.0.0.1:15000/stats?usedonly&filter=^(server.state|listener_manager.workers_started)": read tcp 127.0.0.1:44734->127.0.0.1:15000: read: connection reset by peer
2020-07-13T11:57:35.474578Z	error	Epoch 0 exited with error: signal: segmentation fault (core dumped)
2020-07-13T11:57:35.474583Z	info	No more active epochs, terminating

cc @lambdai I tried to visualise the pprof following the doc but did not get the expected result. If the pprof info in the link does not work for you neither, feel free to ask me to regenerate.

A kindly ping @lambdai

https://www.dropbox.com/s/n00h5hbbsq4alm2/profile003.pdf?dl=0
I put the profile result here as a reference.


text:
Ingress objects are only registered in Ingress Gateway instance in kube-system when multiple ingress gateways are run in different namespaces

Bug description
We've deployed ingress gateways into individual namespaces and have noticed that when Ingress objects are created, only the ingress gateway in istio-system gets updated about them. Looking at the configuration, it looks like running a setup like ours probably wasn't a scenario considered, as it only allows specification of a single ingress gateway. We have a workaround in using the Istio provided CRDs, which work fine, but wanted to raise the issue for awareness.
[ ] Docs
[ ] Installation
[ x] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Steps to reproduce the bug
Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
$ istioctl version --remote
client version: 1.6.3
control plane version: 1.6.3
data plane version: 1.6.3 (53 proxies)
$ kubectl version --short
Client Version: v1.19.2
Server Version: v1.17.9-eks-4c6976

How was Istio installed?
Istio Operator
Environment where bug was observed (cloud vendor, OS, etc)

Only a single gateway deployment will handle Ingress by design. This can be configured via IngressSelector/IngressService in mesh config (https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/, istio configmap), but only one may be selected

@howardjohn Ok, just wanted to verify that this was by design and not oversight, thank you for clarifying.

authors: howardjohn jeffherald


text:
istio-nodeagent failed to get secret for proxy sidecar from secret cache: failed to get root cert

kubeflow 1.1.0
pod istio-nodeagent-br5j5 image: gcr.io/istio-release/node-agent-k8s:release-1.3-latest-daily
centos7.6
When I start a deploy x-0, sidecar istio-proxy will not be ready.
kb logs -f x-0 -c istio-proxy -n nvidia-gpu
...
info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
...

I try to get the logs of the istio-nodeagent on the same kubelet node with x-0
kubectl logs -f istio-nodeagent-br5j5 -n istio-system
...
error  sdsServiceLog  CONNECTION ID: sidecar~10.253.12.32~x-0.nvidia-gpu.svc.cluster.local-7538, RESOURCE NAME： ROOTCA，EVENT：close connection. Failed to get secret for proxy "sidecar~10.253.12.32~x-0.nvidia-gpu.svc.cluster.loca" from secret cache: failed to get root cert.
...

It seems that only the node with x-0 pod has the problem. Pod distributed to other node work fine.

Istio 1.3 has not been supported for quite a while, so we are not able to help with this issue. Supported versions do not even have istio-nodeagent so this bug will not exist in the current code.
See https://istio.io/latest/news/support/announcing-1.3-eol-final/ for more info, thanks!

authors: howardjohn pretendhigh


text:
Liveness and readiness probes fail for Istio if AWS CNI custom network is used.

We configured AWS CNI Custom networks for pods.
kubectl get ENIconfig
NAME     AGE
us-west-2a   5h35m
us-west-2b   5h35m
us-west-2c   5h35m

kubectl get ENIconfig  us-west-2a -o yaml
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: us-west-2a
spec:
  securityGroups:
  - sg-XXXXXX
  subnet: subnet-XXXXX

kubectl get pods –o wide
istio-system        istio-ingressgateway-59cf75bf7-4s557         0/1 Running        0      2m58s   100.64.157.163   ip-10-0-146-62.us-west-2.compute.internal <none>       <none>
istio-system        istiod-6965c56995-d8jph                      0/1 Running        0      2m58s   100.64.148.223   ip-10-0-146-62.us-west-2.compute.internal <none>       <none>
kube-system         aws-node-5jfmb                               1/1 Running        0      16m   10.0.101.133 ip-10-0-101-133.us-west-2.compute.internal   <none>       <none>
kube-system         aws-node-qxhd4                               1/1 Running        0      16m   10.0.146.62  ip-10-0-146-62.us-west-2.compute.internal <none>       <none>
kube-system     aws-node-xtkmc                               1/1 Running        0      15m   10.0.38.17   ip-10-0-38-17.us-west-2.compute.internal <none>       <none>

kubectl –n istio-system get events
53s         Warning   Unhealthy             pod/istiod-6965c56995-d8jph                Readiness probe failed: Get http://100.64.148.223:8080/ready: dial tcp 100.64.148.223:8080: connect: connection refused

Kubelet can’t connect to istiod from node ip to 100.64.x.x
Also, we have the same issue with all applications which used Health Checks.
From pod with ip 100.64.x.x we can ping pods with ip 100.64.x.x
We have this issue with the following components and versions:
AWS CNI plugin versions 1.7.4, 1.7.5 and 1.7.6.
Istio versions: 1.7.3, 1.7.4, 1.8.0
But if try to deploy an easy application pod with Readiness and LIveness probes - it works  (for  example Nginx)
kubectl -n kube-system get daemonset aws-node -o yaml
   containers:
   - env:
     - name: ADDITIONAL_ENI_TAGS
       value: '{}'
     - name: AWS_VPC_CNI_NODE_PORT_SUPPORT
       value: "true"
     - name: AWS_VPC_ENI_MTU
       value: "9001"
     - name: AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER
       value: "false"
     - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
       value: "true"
     - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
       value: "true"
     - name: AWS_VPC_K8S_CNI_LOGLEVEL
       value: DEBUG
     - name: AWS_VPC_K8S_CNI_LOG_FILE
       value: /host/var/log/aws-routed-eni/ipamd.log
     - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
       value: prng
     - name: AWS_VPC_K8S_CNI_VETHPREFIX
       value: eni
     - name: AWS_VPC_K8S_PLUGIN_LOG_FILE
       value: /var/log/aws-routed-eni/plugin.log
     - name: AWS_VPC_K8S_PLUGIN_LOG_LEVEL
       value: DEBUG
     - name: DISABLE_INTROSPECTION
       value: "false"
     - name: DISABLE_METRICS
       value: "false"
     - name: ENABLE_POD_ENI
       value: "true"
     - name: MY_NODE_NAME
       valueFrom:
         fieldRef:
           apiVersion: v1
           fieldPath: spec.nodeName
     - name: WARM_ENI_TARGET
       value: "1"
     - name: ENI_CONFIG_LABEL_DEF
       value: failure-domain.beta.kubernetes.io/zone
     image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.7.6-eksbuild.1
     imagePullPolicy: Always
     livenessProbe:
       exec:
         command:
         - /app/grpc-health-probe
         - -addr=:50051
       failureThreshold: 3
       initialDelaySeconds: 60
       periodSeconds: 10
       successThreshold: 1
       timeoutSeconds: 1
     name: aws-node
     ports:
     - containerPort: 61678
       hostPort: 61678
       name: metrics
       protocol: TCP
     readinessProbe:
       exec:
         command:
         - /app/grpc-health-probe
         - -addr=:50051
       failureThreshold: 3
       initialDelaySeconds: 1
       periodSeconds: 10
       successThreshold: 1
       timeoutSeconds: 1
     resources:
       requests:
         cpu: 10m
     securityContext:
       capabilities:
         add:
         - NET_ADMIN
     terminationMessagePath: /dev/termination-log
     terminationMessagePolicy: File
     volumeMounts:
     - mountPath: /host/opt/cni/bin
       name: cni-bin-dir
     - mountPath: /host/etc/cni/net.d
       name: cni-net-dir
     - mountPath: /host/var/log/aws-routed-eni
       name: log-dir
     - mountPath: /var/run/aws-node
       name: run-dir
     - mountPath: /var/run/dockershim.sock
       name: dockershim
     - mountPath: /run/xtables.lock
       name: xtables-lock
   dnsPolicy: ClusterFirst
   hostNetwork: true
   initContainers:
   - env:
     - name: DISABLE_TCP_EARLY_DEMUX
       value: "true"
     image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init:v1.7.6-eksbuild.1
     imagePullPolicy: Always
     name: aws-vpc-cni-init

We have this issue with the following components and versions:
AWS CNI plugin versions 1.7.4, 1.7.5 and 1.7.6.
Istio versions: 1.7.3, 1.7.4, 1.8.0

Jumped on a call with @SaranBalaji90 we found out that istio pods were not able to communicate to Kubernetes APIServer because SecurityGroup applied on the Kubernetes APIServer didn't allow access to istio pods Security Group. After fixing that istio came up as expected.


text:
KubernetesResourcesSpec marshaling for volumes is wrong

What works:
      k8s:
        volumes:
        - name: config-volume
          volumeSource:
            configMap:
              localObjectReference:
                name: istio

What should work:
      k8s:
        volumes:
        - name: config-volume
          configMap:
            name: istio

cc @morvencao

The volumeSource  is another inline field, see: https://github.com/kubernetes/api/blob/master/core/v1/types.go#L46
which results in error marshal/unmarshal with jsonpb, we may consider duplicating the volume field in our operator api, similar to what we do for the readinessProbe and livenessProbe fields(https://github.com/istio/api/blob/master/operator/v1alpha1/operator.proto#L518)
But there are too many different volumeSources, I'm not sure if this is a good idea, wdyt? @howardjohn
/cc @ostromart

Is it not possible to use k8s library marshalling logic?

AFIK, it's not.
We encountered similar issue for other fields long time ago, eg. Handler field in Probe, see: https://github.com/kubernetes/api/blob/master/core/v1/types.go#L2098
The way to resolve that is duplicate the field in operator api.

@howardjohn I filed another PR to fix this: https://github.com/istio/api/pull/1925
Tested this case locally, it works. I may also need to update the test cases in https://github.com/istio/istio/pull/31683 afterward.

Not stale

It looks like besides wrong marshalling there is another underlying issue with Volumes and Volume Mounts pointed out in #34046 and #36066

authors: hi-artem howardjohn morvencao


text:
Test CNI ambient flow with integration tests

We now have the ability to create various namespaces without root; we do this in tools/istio-iptables/pkg/dependencies/implementation_linux_test.go already.
It may be valuable to test more end-to-end flows of the CNI plugin + ambient + ztunnel lifecycle. I am not 100% convinced this is will improve coverage, but my intuition is that it will
cc @bleggett

xref: https://github.com/istio/istio/issues/50991
It would be good to (re)enable coverage for istio-cni before we did this, to actually make sure we were covering new scenarios/branches/hitting the least-covered flows first.

authors: bleggett howardjohn


text:
WIP:  run tests before sleep

#13266
#13264
Signed-off-by: Yuchen Dai silentdai@gmail.com

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: silentdai
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: geeknoid
If they are not already assigned, you can assign the PR to them by writing /assign @geeknoid in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@duderino @kyessenov FYI let's see if this pr succeeds

@silentdai: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
392802df1ed3cafc97b6e12d090831d9308aa57d
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silentdai: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
392802df1ed3cafc97b6e12d090831d9308aa57d
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
392802df1ed3cafc97b6e12d090831d9308aa57d
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silentdai: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
392802df1ed3cafc97b6e12d090831d9308aa57d
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
392802df1ed3cafc97b6e12d090831d9308aa57d
link
/test istio-pilot-multicluster-e2e

prow/e2e_pilotv2_auth_sds.sh
392802df1ed3cafc97b6e12d090831d9308aa57d
link
/test istio_auth_sds_e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@silentdai: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

abadon

source: gharchive/pull-request

authors: istio-testing silentdai


text:
lint fixes for pilot

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #1439
Special notes for your reviewer:
Release note:

none

/assign @kyessenov


text:
use global.network to set ISTIO_META_NETWORK for sidecars and ingressgateway

See also https://github.com/istio/installer/pull/425

@ayj: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

unit-tests_istio
33b2b4dfb4b36d2f147c49f4544401d31a44e33a
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

unit-tests_istio
33b2b4dfb4b36d2f147c49f4544401d31a44e33a
link
/test unit-tests_istio

integ-security-k8s-tests_istio
33b2b4dfb4b36d2f147c49f4544401d31a44e33a
link
/test integ-security-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test unit-tests_istio

/test integ-security-k8s-tests_istio

/hold

@ayj: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-mixer-no_auth_istio
72d443487f69d2cde0f4d8bce592b9f2b977ee1e
link
/test e2e-mixer-no_auth_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test e2e-mixer-no_auth_istio

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-mixer-no_auth_istio
72d443487f69d2cde0f4d8bce592b9f2b977ee1e
link
/test e2e-mixer-no_auth_istio

integ-security-k8s-tests_istio
72d443487f69d2cde0f4d8bce592b9f2b977ee1e
link
/test integ-security-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@ayj: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-mixer-no_auth_istio
72d443487f69d2cde0f4d8bce592b9f2b977ee1e
link
/test e2e-mixer-no_auth_istio

integ-security-k8s-tests_istio
72d443487f69d2cde0f4d8bce592b9f2b977ee1e
link
/test integ-security-k8s-tests_istio

unit-tests_istio
72d443487f69d2cde0f4d8bce592b9f2b977ee1e
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-security-k8s-tests_istio

authors: ayj istio-testing


text:
remove old operator ref doc

otherwise it would be grabbed and rendered on istio.io, instead the reference on istio.io should point to the one in istio/api
related change on istio.io: https://github.com/istio/istio.io/pull/6600

@ostromart can you help approve this one


text:
Deepequal instead of pointer ==

Istio operator pointer address is compared instead of the value itself. Every call to getOrCreateReconciler would have a different iop address and the would always result in equality failing when this is compared with reconciler.GetInstance

Hi @deva26. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

I this change is ok but I'd like @rcernich to take a look since he wrote the code and I may be missing something.

I think this change is ok but I'd like @rcernich to take a look since he wrote the code. I actually think you'll get a new reconciler anyway because Reconcile is only called when there's a change in the CR which makes me think I may be missing something.

A reconciler is also called when one of the watched resource is deleted. So this check makes the code tighter.

ping @istio/wg-environments-maintainers

@rcernich -- WDYT?

@deva26: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@deva26: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This PR need has been done away with the recent changes in the master. So closing this. Thanks for the review comments from istio team.


text:
expose values.tracing.jaeger.storageSize

Currently manifests/charts/istio-telemetry/tracing/templates/pvc.yaml is hard-coded to 5Gi, and some scenarios require more storage space
[ ] Configuration Infrastructure
[ ] Docs
[X] Installation
[ ] Networking
[ ] Performance and Scalability
[X] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Hi @potterhe. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@potterhe: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
79eb713f0e185b1f2739e1d7a3a473c0cc7da44d
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@potterhe: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
79eb713f0e185b1f2739e1d7a3a473c0cc7da44d
link
/test gencheck_istio

unit-tests_istio
79eb713f0e185b1f2739e1d7a3a473c0cc7da44d
link
/test unit-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Ok i got it thanks

authors: istio-testing potterhe