text:
k8s jobs seem to fail as envoy proxy is not ready

Bug description
When we deploy a job with automatic istio sidecar injection we see that the job seems to be stuck as the istio envoy proxy is not ready and it terminates abruptly.
istio logs:
drainDuration: 45s
holdApplicationUntilProxyStarts: true
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: 00000-minio-make-bucket-job.dev
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
sampling: 10
zipkin:
address: jaeger-operator-jaeger-collector.dev-metrics.svc.cluster.local:9411

2021-04-16T14:27:42.393663Z info Proxy role ips=[10.244.1.172 fe80::a0e9:dbff:fe79:2abb] type=sidecar id=00000-minio-make-bucket-job-7mg6j.dev domain=dev.svc.cluster.local
2021-04-16T14:27:42.393704Z info JWT policy is first-party-jwt
2021-04-16T14:27:42.393764Z info Pilot SAN: [istiod.istio-system.svc]
2021-04-16T14:27:42.393799Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-04-16T14:27:42.393969Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-16T14:27:42.394365Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-04-16T14:27:42.645773Z info ads All caches have been synced up in 373.3233ms, marking server ready
2021-04-16T14:27:42.651450Z info sds SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-04-16T14:27:42.653318Z info sds Start SDS grpc server
2021-04-16T14:27:42.653670Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-04-16T14:27:42.655790Z info Starting proxy agent
2021-04-16T14:27:42.659604Z info Opening status port 15020
2021-04-16T14:27:42.660720Z info Received new config, creating new Envoy epoch 0
2021-04-16T14:27:42.660949Z info Epoch 0 starting
2021-04-16T14:27:44.709621Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster 00000-minio-make-bucket-job.dev --service-node sidecar~10.244.1.172~00000-minio-make-bucket-job-7mg6j.dev~dev.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2021-04-16T14:27:45.023810Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T14:27:45.023954Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T14:27:45.026974Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T14:27:45.027304Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T14:28:02.365432Z warn ca ca request failed, starting attempt 1 in 95.862284ms
2021-04-16T14:28:02.462371Z warn ca ca request failed, starting attempt 2 in 188.263306ms
2021-04-16T14:28:02.651992Z warn ca ca request failed, starting attempt 3 in 429.226801ms
2021-04-16T14:28:03.082641Z warn ca ca request failed, starting attempt 4 in 831.475066ms
2021-04-16T14:28:05.361934Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-04-16T14:28:22.380693Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.172:60008->10.96.0.10:53: i/o timeout"
2021-04-16T14:28:22.385375Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.172:60008->10.96.0.10:53: i/o timeout"
2021-04-16T14:28:43.156402Z info Agent draining Proxy
2021-04-16T14:28:43.156875Z info Watcher has successfully terminated
2021-04-16T14:28:43.157045Z info Status server has successfully terminated
2021-04-16T14:28:43.157192Z error accept tcp [::]:15020: use of closed network connection
2021-04-16T14:28:43.164136Z info Graceful termination period is 5s, starting...
2021-04-16T14:28:43.308033Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-04-16T14:28:43.847791Z warn ca ca request failed, starting attempt 1 in 95.641619ms
2021-04-16T14:28:43.944209Z warn ca ca request failed, starting attempt 2 in 211.544196ms
2021-04-16T14:28:44.156925Z warn ca ca request failed, starting attempt 3 in 388.944438ms
2021-04-16T14:28:44.548070Z warn ca ca request failed, starting attempt 4 in 860.886899ms
2021-04-16T14:28:45.410152Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2021-04-16T14:28:48.165225Z info Graceful termination period complete, terminating remaining proxies.
2021-04-16T14:28:48.165389Z warn Aborting epoch 0...
2021-04-16T14:28:48.165440Z warn Aborted all epochs
2021-04-16T14:28:48.165486Z info Agent has successfully terminated
2021-04-16T14:28:48.170603Z warning envoy main caught ENVOY_SIGTERM
2021-04-16T14:28:48.183664Z warning envoy config StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
Logs from 4/16/2021, 9:27:42 AM
Show timestamps
Show previous terminated container

It is a spordiac error, some of service and josb get deployed but some seem to get stuck depending if envoy is not ready or not.
What we see is for the jobs that pass following seem to be the logs
2021-04-13T08:40:43.038362Z info sds SDS: PUSH resource=ROOTCA
2021-04-13T08:40:43.038445Z info sds SDS: PUSH resource=default
2021-04-13T08:40:44.824525Z info Initialization took 3.559888677s
2021-04-13T08:40:44.824555Z info Envoy proxy is ready
{"method":"GET","user_agent":null,"response_code":200,"upstream_local_address":"10.40.129.235:44760","downstream_local_address":"10.0.108.130:9000","response_flags":"-","authority":"dev-argo-minio.argo.svc.cluster.local:9000","route_name":"default","downstream_remote_address":"10.40.129.235:45240","start_time":"2021-04-13T08:40:47.018Z","upstream_transport_failure_reason":null,"bytes_sent":128,"duration":17,"requested_server_name":null,"path":"/run-1-5bbe59b7f1574f239f95624e7c5f3d78?location=","response_code_details":"via_upstream","protocol":"HTTP/1.1","upstream_cluster":"outbound|9000||dev-argo-minio.argo.svc.cluster.local","x_forwarded_for":null,"request_id":"82ffbf9d-2240-4e3c-827b-69df1c76152c","bytes_received":0,"upstream_service_time":"16","upstream_host":"10.40.136.14:9000","connection_termination_details":null}
2021-04-13T08:40:50.204151Z info Agent draining Proxy

and for the ones that fail we see the following error
2021-04-13T09:03:13.492423Z info sds SDS: PUSH resource=ROOTCA
2021-04-13T09:03:13.492464Z info sds SDS: PUSH resource=default
2021-04-13T09:04:08.028960Z info Agent draining Proxy

[ ] Docs
[ ] Installation
[ X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
[X ] Multi Cluster
[] Virtual Machine
[ ] Multi Control Plane
Expected behavior
the envoy proxy should not be terminated abruptly
Steps to reproduce the bug

install istio on a kind cluster/ or any cluster
create a separate namespace and label istio injection true (automatic sidecar injection)
deploy the job and monitor the logs for istio sidecar

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)
$ istio-1.9.3 % istioctl version --remote
client version: 1.9.3
control plane version: 1.9.1
data plane version: 1.9.1 (11 proxies), 1.7.3 (2 proxies)

helm
$ istio-1.9.3 % helm version --short
v3.5.3+g041ce5a

How was Istio installed?
kubectl apply -f https://docs.projectcalico.org/$CALICO_VERSION/manifests/calico.yaml
ISTIO_VERSION=1.9.1
ISTIONAME="istio-$ISTIO_VERSION"
URL="https://github.com/istio/istio/releases/download/${ISTIO_VERSION}/istio-${ISTIO_VERSION}-${OSEXT}.tar.gz"
  curl -L "$URL" | tar xz
$ISTIONAME/bin/istioctl operator init --operatorNamespace istio-system

sed '/serviceAnnotations:/,/
kubectl apply -f operator-deployment.yaml -n istio-sy

stem
kubectl wait deployment/istiod -n istio-system --for condition=available --timeout=-1s
kubectl wait deployment/istio-ingressgateway -n istio-system --for condition=available --timeout=-1s

# istio ingress gateway mapped to 30443
kubectl patch -n istio-system service istio-ingressgateway --patch '{"spec":{"ports":[{"name":"https","nodePort":30443,"port":443}]}}'

#MTLS enforced
kubectl apply -f peer.yaml
kubectl apply -f gateway.yaml

operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: default
  addonComponents:
    prometheus:
      enabled: false
    grafana:
      enabled: false
    tracing:
      enabled: false
    telemetry:
      enabled: false
    kiali:
      enabled: false
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogEncoding: JSON
    defaultConfig:
      holdApplicationUntilProxyStarts: true
      tracing:
        zipkin:
          address: jaeger-operator-jaeger-collector.dev-metrics.svc.cluster.local:9411
        sampling: 10
  components:
    pilot:
      k8s:
        podDisruptionBudget:
          minAvailable: 1
        resources:
          limits:
            memory: 3Gi
          requests:
            cpu: "1"
            memory: 1Gi
        hpaSpec:
          maxReplicas: 10
          minReplicas: 3
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  topologyKey: topology.kubernetes.io/zone
                weight: 100
    egressGateways:
      - name: istio-egressgateway
        enabled: true
        k8s:
          podDisruptionBudget:
            minAvailable: 1
          hpaSpec:
            maxReplicas: 3
            minReplicas: 2
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - podAffinityTerm:
                    topologyKey: topology.kubernetes.io/zone
                  weight: 100
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          podDisruptionBudget:
            minAvailable: 1
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
                - podAffinityTerm:
                    topologyKey: topology.kubernetes.io/zone
                  weight: 100
          hpaSpec:
            maxReplicas: 10
            minReplicas: 3
          serviceAnnotations:
            service.beta.kubernetes.io/azure-load-balancer-resource-group: dev-uk
          service:
            loadBalancerIP: 20.49.240.148
      # Ingress specifically for cluster local traffic
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          hpaSpec:
            maxReplicas: 10
            minReplicas: 3
          service:
            type: ClusterIP
            ports:
              - port: 15020
                name: status-port
              - port: 80
                targetPort: 8080
                name: http2
              - port: 443
                targetPort: 8443
                name: https
  values:
    global:
      proxy:
        tracer: zipkin
      mountMtlsCerts: true

peer.yaml
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

gateway.yaml
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: https-dns
        number: 443
        protocol: HTTPS
[bug-report.zip](https://github.com/istio/istio/files/6326227/bug-report.zip)

      tls:
        credentialName: prod-wildcard-dev-tls
        mode: SIMPLE

Environment where the bug was observed (cloud vendor, OS, etc)
kind cluster and also in aks cluster
attached the bug-report as well

Looks like the proxy has trouble looking up istiod from DNS: envoy config StreamAggregatedResources gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.1.172:60008->10.96.0.10:53: i/o timeout"
Seems like there is something wrong with your cluster DNS instead of an Istio issue.

cluster DNS seems to be ok as we see some of the services seems to be working correctly
eg this is the logs from envoy sidecar for Nginx service:
2021-04-15T20:33:36.205477Z info Proxy role ips=[10.244.1.137 fe80::9c1e:c6ff:fe07:b706] type=sidecar id=nginx-6d48c9bcb8-bfksg.default domain=default.svc.cluster.local
2021-04-15T20:33:36.205503Z info JWT policy is first-party-jwt
2021-04-15T20:33:36.205519Z info Pilot SAN: [istiod.istio-system.svc]
2021-04-15T20:33:36.205536Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-04-15T20:33:36.205666Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-15T20:33:36.205793Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-04-15T20:33:36.237628Z info ads All caches have been synced up in 37.6944ms, marking server ready
2021-04-15T20:33:36.238056Z info sds SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-04-15T20:33:36.238097Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-04-15T20:33:36.238390Z info Starting proxy agent
2021-04-15T20:33:36.238897Z info sds Start SDS grpc server
2021-04-15T20:33:36.239094Z info Opening status port 15020
2021-04-15T20:33:36.239248Z info Received new config, creating new Envoy epoch 0
2021-04-15T20:33:36.239358Z info Epoch 0 starting
2021-04-15T20:33:36.268372Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster nginx.default --service-node sidecar~10.244.1.137~nginx-6d48c9bcb8-bfksg.default~default.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2021-04-15T20:33:36.310673Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-15T20:33:36.310737Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-15T20:33:36.311114Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-15T20:33:36.311153Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-15T20:33:36.358099Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-15T20:33:36.382873Z info ads ADS: new connection for node:sidecar~10.244.1.137~nginx-6d48c9bcb8-bfksg.default~default.svc.cluster.local-1
2021-04-15T20:33:36.383151Z info ads ADS: new connection for node:sidecar~10.244.1.137~nginx-6d48c9bcb8-bfksg.default~default.svc.cluster.local-2
2021-04-15T20:33:36.491718Z info cache Root cert has changed, start rotating root cert
2021-04-15T20:33:36.491760Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2021-04-15T20:33:36.491816Z info cache generated new workload certificate latency=253.847ms ttl=23h59m59.5082014s
2021-04-15T20:33:36.491920Z info cache returned delayed workload certificate from cache ttl=23h59m59.5080901s
2021-04-15T20:33:36.492134Z info sds SDS: PUSH resource=default
2021-04-15T20:33:36.583886Z info sds SDS: PUSH resource=ROOTCA
2021-04-15T20:33:36.584425Z info sds SDS: PUSH resource=ROOTCA
2021-04-15T20:33:36.788857Z info Initialization took 600.2021ms
2021-04-15T20:33:36.788911Z info Envoy proxy is ready
{"bytes_received":1600,"x_forwarded_for":null,"downstream_local_address":"10.244.1.137:80","duration":65072,"upstream_local_address":"127.0.0.1:51286","response_code_details":null,"protocol":null,"upstream_cluster":"inbound|80||","request_id":null,"path":null,"start_time":"2021-04-15T20:33:52.607Z","route_name":null,"response_flags":"-","method":null,"response_code":0,"bytes_sent":851,"upstream_host":"127.0.0.1:80","requested_server_name":"outbound_.80_._.nginx.default.svc.cluster.local","authority":null,"downstream_remote_address":"10.244.1.135:54462","upstream_service_time":null,"upstream_transport_failure_reason":null,"connection_termination_details":null,"user_agent":null}
2021-04-15T21:01:38.698473Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-15T21:01:38.975382Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-15T21:30:31.468850Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-15T21:30:31.568427Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-15T21:58:07.739210Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-15T21:58:07.920747Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-15T22:29:06.218544Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-15T22:29:06.260153Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-15T22:59:08.101953Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-15T22:59:08.393028Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-15T23:27:41.895663Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-15T23:27:42.053222Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-15T23:54:52.174203Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-15T23:54:52.468088Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T00:25:24.502884Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-16T00:25:24.522801Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T00:57:51.541702Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-16T00:57:51.897610Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T01:26:54.518094Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-16T01:26:54.647871Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T01:59:09.789439Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-16T01:59:10.064931Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T02:30:47.101288Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-16T02:30:47.399154Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T12:59:37.589130Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-16T12:59:37.616750Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T13:29:10.128271Z warning envoy config StreamAggregatedResources gRPC config stream closed: 0,
2021-04-16T13:29:10.370899Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012

so some of the services are running great but some have issue with envoy. If it was core DNS then none of the services would run, right?
Although I do see i/o timeout in core DNS but not related to the above jobs
core DNS logs:
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:39636->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:47030->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:33771->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:59642->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:43081->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:56091->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:50865->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:56831->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:54200->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:38388->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:38799->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:49164->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 zipkin.istio-system. A: read udp 10.244.222.65:44546->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:49421->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:50650->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:45466->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:34988->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:37987->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 zipkin.istio-system. A: read udp 10.244.222.65:36314->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:36026->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:37740->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:53790->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:41638->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:54651->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:52371->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:53001->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:54074->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:53570->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:54967->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:54203->192.168.65.2:53: i/o timeout
[ERROR] plugin/errors: 2 api.github.com. A: read udp 10.244.222.65:53772->192.168.65.2:53: i/o timeout

so some of the services are running great but some have issue with envoy. If it was core DNS then none of the services would run, right?

timeout could be intermittent, due to over loaded etc.

when I try on the aks cluster I don't see i/o timeout errors...but still envoy fails..
envoy side-car logs for an upload job
2021-04-16T19:15:23.214294Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
holdApplicationUntilProxyStarts: true
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: upload-dyn-div-strategy-to-mongo.dev
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
sampling: 10
zipkin:
address: jaeger-operator-jaeger-collector.dev-metrics.svc.cluster.local:9411

2021-04-16T19:15:23.214323Z info Proxy role ips=[10.40.136.233 fe80::bca1:1aff:fe79:e087] type=sidecar id=upload-dyn-div-strategy-to-mongo-jrzsk.dev domain=dev.svc.cluster.local
2021-04-16T19:15:23.214330Z info JWT policy is third-party-jwt
2021-04-16T19:15:23.214338Z info Pilot SAN: [istiod.istio-system.svc]
2021-04-16T19:15:23.214341Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-04-16T19:15:23.214382Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-16T19:15:23.214475Z info citadelclient Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-04-16T19:15:23.276867Z info ads All caches have been synced up in 65.70697ms, marking server ready
2021-04-16T19:15:23.277228Z info sds SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-04-16T19:15:23.277250Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-04-16T19:15:23.277439Z info Starting proxy agent
2021-04-16T19:15:23.277707Z info Opening status port 15020
2021-04-16T19:15:23.277710Z info sds Start SDS grpc server
2021-04-16T19:15:23.277774Z info Received new config, creating new Envoy epoch 0
2021-04-16T19:15:23.277820Z info Epoch 0 starting
2021-04-16T19:15:23.466855Z info cache Root cert has changed, start rotating root cert
2021-04-16T19:15:23.466905Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-04-16T19:15:23.466920Z info cache generated new workload certificate latency=189.576211ms ttl=23h59m59.533089303s
2021-04-16T19:15:24.290100Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster upload-dyn-div-strategy-to-mongo.dev --service-node sidecar~10.40.136.233~upload-dyn-div-strategy-to-mongo-jrzsk.dev~dev.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2021-04-16T19:15:24.333732Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T19:15:24.333780Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T19:15:24.334394Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T19:15:24.334441Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-16T19:15:24.381273Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-16T19:15:24.747390Z info ads ADS: new connection for node:sidecar~10.40.136.233~upload-dyn-div-strategy-to-mongo-jrzsk.dev~dev.svc.cluster.local-1
2021-04-16T19:15:24.747463Z info ads ADS: new connection for node:sidecar~10.40.136.233~upload-dyn-div-strategy-to-mongo-jrzsk.dev~dev.svc.cluster.local-2
2021-04-16T19:15:24.747474Z info cache returned workload certificate from cache ttl=23h59m58.252532334s
2021-04-16T19:15:24.747783Z info sds SDS: PUSH resource=default
2021-04-16T19:15:24.747870Z info sds SDS: PUSH resource=ROOTCA
2021-04-16T19:16:23.472791Z info Agent draining Proxy
2021-04-16T19:16:23.472825Z info Status server has successfully terminated
2021-04-16T19:16:23.472937Z error accept tcp [::]:15020: use of closed network connection
2021-04-16T19:16:23.472958Z info Watcher has successfully terminated
2021-04-16T19:16:23.473912Z info Graceful termination period is 5s, starting...
2021-04-16T19:16:28.474042Z info Graceful termination period complete, terminating remaining proxies.
2021-04-16T19:16:28.474078Z warn Aborting epoch 0...
2021-04-16T19:16:28.474082Z warn Aborted all epochs
2021-04-16T19:16:28.474086Z info Agent has successfully terminated
2021-04-16T19:16:28.474093Z warn Aborting epoch 0

Also there is no i/o timeout on core DNS
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[ERROR] plugin/errors: 2 argoproj.github.io.hr3ftajhce1ezlxcnldzrc3qdh.zx.internal.cloudapp.net. A: read udp 10.40.128.88:50833->168.63.129.16:53: i/o timeout
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override
[WARNING] No files matching import glob pattern: custom/*.server
[WARNING] No files matching import glob pattern: custom/*.override

So I don't think it is an issue with DNS

Thanks! there is clearly something wrong. The agent stuck for a minute:
2021-04-16T19:15:24.747870Z info sds SDS: PUSH resource=ROOTCA
2021-04-16T19:16:23.472791Z info Agent draining Proxy

A log from istiod during that time frame would also be useful. I guess the enable xdsproxy debug log will also be helpful (by adding sidecar.istio.io/agentLogLevel: xdsproxy:debug to your job pod template).
@howardjohn @ramaraochavali @hzxuzhonghu Does this ring a bell or is there more debug signal that we can gather?

What makes you think it is stuck? It doesn't log that much, I wouldn't expect any logs during that minute?

There is no Envoy does not become ready and initialization done log line.

Ah of course, thanks. Might check the k8s events, it will show if its getting connection refused on port 15021, which would mean envoy didn't fully start up so we don't get the probes

so I have like 6 replicas of istiod... out of those that I can gather logs related to the above job
021-04-16T19:15:19.995994Z info Sidecar injection request for ravnml-dev/upload-dyn-div-strategy-to-mongo-***** (actual name not yet known)
2021-04-16T19:15:22.538858Z info ads ADS: "10.40.134.254:40380" sidecar~10.40.134.254~ku-solr-exporter-56df4c6c48-k5g2d.ku~ku.svc.cluster.local-2776 terminated rpc error: code = Canceled desc = context canceled
2021-04-16T19:15:37.827304Z info ads Push debounce stable[15010] 1: 100.106589ms since last change, 100.106389ms since last push, full=false
2021-04-16T19:15:37.827395Z info ads XDS: Incremental Pushing:2021-04-16T19:11:34Z/1647 ConnectedEndpoints:22 Version:2021-04-16T19:11:34Z/1647
2021-04-16T19:15:39.052732Z info ads ADS: new connection for node:sidecar~10.40.136.235~kube-monkey-847b9cfd74-dpps4.ravnml-inttest~ravnml-inttest.svc.cluster.local-2798
2021-04-16T19:15:39.069329Z info ads CDS: PUSH for node:kube-monkey-847b9cfd74-dpps4.ravnml-inttest resources:669 size:681.8kB
2021-04-16T19:15:39.070538Z info ads EDS: PUSH for node:kube-monkey-847b9cfd74-dpps4.ravnml-inttest resources:541 size:229.2kB empty:1 cached:537/541
2021-04-16T19:15:39.091147Z info ads LDS: PUSH for node:kube-monkey-847b9cfd74-dpps4.ravnml-inttest resources:238 size:1.4MB
2021-04-16T19:15:39.286595Z info ads RDS: PUSH for node:kube-monkey-847b9cfd74-dpps4.ravnml-inttest resources:134 size:1.1MB
2021-04-16T19:15:41.707825Z info ads Incremental push, service ku-search-fe.ku.svc.cluster.local has no endpoints
2021-04-16T19:15:41.808022Z info ads Push debounce stable[15011] 1: 100.137289ms since last change, 100.137189ms since last push, full=false
2021-04-16T19:15:41.808120Z info ads XDS: Incremental Pushing:2021-04-16T19:11:34Z/1647 ConnectedEndpoints:23 Version:2021-04-16T19:11:34Z/1647
2021-04-16T19:15:41.818926Z info ads ADS: new connection for node:sidecar~10.40.135.247~ku-laminar-solr-0.ku-laminar~ku-laminar.svc.cluster.local-2799
2021-04-16T19:15:41.836097Z info ads CDS: PUSH for node:ku-laminar-solr-0.ku-laminar resources:670 size:682.2kB
2021-04-16T19:15:41.837709Z info ads EDS: PUSH for node:ku-laminar-solr-0.ku-laminar resources:541 size:229.0kB empty:0 cached:541/541
2021-04-16T19:15:41.855785Z info ads LDS: PUSH for node:ku-laminar-solr-0.ku-laminar resources:237 size:1.4MB
2021-04-16T19:15:42.001661Z info ads RDS: PUSH for node:ku-laminar-solr-0.ku-laminar resources:134 size:1.1MB
2021-04-16T19:15:42.742022Z info ads Incremental push, service ml-data-service.ravnml-inttest.svc.cluster.local has no endpoints
2021-04-16T19:15:42.842196Z info ads Push debounce stable[15012] 1: 100.109992ms since last change, 100.109892ms since last push, full=false
2021-04-16T19:15:42.842279Z info ads XDS: Incremental Pushing:2021-04-16T19:11:34Z/1647 ConnectedEndpoints:24 Version:2021-04-16T19:11:34Z/1647
2021-04-16T19:15:59.092512Z info ads ADS: "10.40.136.236:50544" sidecar~10.40.136.236~ml-data-service-9954698ff-9b6lv.ravnml~ravnml.svc.cluster.local-2774 terminated rpc error: code = Canceled desc = context canceled
2021-04-16T19:16:18.033377Z info ads ADS: "10.40.137.197:41062" sidecar~10.40.137.197~ravnml-pegasus-minio-674bd77b84-l6k54.ravnml-pegasus~ravnml-pegasus.svc.cluster.local-2775 terminated rpc error: code = Canceled desc = context canceled
2021-04-16T19:16:46.374377Z info ads ADS: "10.40.137.35:41820" sidecar~10.40.137.35~ku-cloud-migration-d8748f9b4-l55fj.ku~ku.svc.cluster.local-2773 terminated rpc error: code = Canceled desc = context canceled
2021-04-16T19:16:51.012918Z info klog admissionregistration.k8s.io/v1beta1 MutatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+; use admissionregistration.k8s.io/v1 MutatingWebhookConfiguration
2021-04-16T19:17:11.956920Z info klog networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
2021-04-16T19:17:36.954383Z info klog networking.k8s.io/v1beta1 IngressClass is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 IngressClassList
2021-04-16T19:17:52.643649Z info rootcertrotator Check and rotate root cert.
2021-04-16T19:17:52.648824Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2021-04-16T19:17:59.014969Z info ads Push debounce stable[15013] 1: 100.122545ms since last change, 100.122345ms since last push, full=false
2021-04-16T19:17:59.015070Z info ads XDS: Incremental Pushing:2021-04-16T19:11:34Z/1647 ConnectedEndpoints:21 Version:2021-04-16T19:11:34Z/1647
2021-04-16T19:18:02.930013Z info ads Incremental push, service ml-data-service.ravnml-inttest.svc.cluster.local has no endpoints
2021-04-16T19:18:03.030191Z info ads Push debounce stable[15014] 1: 100.117446ms since last change, 100.117046ms since last push, full=false
2021-04-16T19:18:03.030278Z info ads XDS: Incremental Pushing:2021-04-16T19:11:34Z/1647 ConnectedEndpoints:21 Version:2021-04-16T19:11:34Z/1647
2021-04-16T19:18:07.540751Z info ads ADS: new connection for node:sidecar~10.40.137.187~dc-adb2cd784b199b889a5032d4bade950224356953-00001-deploymetljln.ravnml~ravnml.svc.cluster.local-2800
2021-04-16T19:18:07.558782Z info ads CDS: PUSH for node:dc-adb2cd784b199b889a5032d4bade950224356953-00001-deploymetljln.ravnml resources:673 size:683.1kB

Also logs for the upload job, we can see connection refused error. I use scuttle but it waits for the envoy to start which never starts and the envoy terminates so the job is stuck waiting for it...
2021-04-16T21:38:55Z scuttle: Polling Envoy (154), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:40:22Z scuttle: Polling Envoy (155), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:41:40Z scuttle: Polling Envoy (156), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:42:17Z scuttle: Polling Envoy (157), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:43:34Z scuttle: Polling Envoy (158), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:44:27Z scuttle: Polling Envoy (159), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:45:05Z scuttle: Polling Envoy (160), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:45:47Z scuttle: Polling Envoy (161), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:47:01Z scuttle: Polling Envoy (162), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:48:11Z scuttle: Polling Envoy (163), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:48:47Z scuttle: Polling Envoy (164), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:49:48Z scuttle: Polling Envoy (165), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:50:24Z scuttle: Polling Envoy (166), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:51:03Z scuttle: Polling Envoy (167), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:51:38Z scuttle: Polling Envoy (168), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:52:27Z scuttle: Polling Envoy (169), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:53:06Z scuttle: Polling Envoy (170), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:53:44Z scuttle: Polling Envoy (171), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:54:34Z scuttle: Polling Envoy (172), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:55:36Z scuttle: Polling Envoy (173), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:56:41Z scuttle: Polling Envoy (174), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:57:41Z scuttle: Polling Envoy (175), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T21:58:53Z scuttle: Polling Envoy (176), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:00:02Z scuttle: Polling Envoy (177), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:01:03Z scuttle: Polling Envoy (178), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:02:13Z scuttle: Polling Envoy (179), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:03:26Z scuttle: Polling Envoy (180), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:04:34Z scuttle: Polling Envoy (181), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:05:05Z scuttle: Polling Envoy (182), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:05:37Z scuttle: Polling Envoy (183), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:06:13Z scuttle: Polling Envoy (184), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:07:05Z scuttle: Polling Envoy (185), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:08:24Z scuttle: Polling Envoy (186), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:09:15Z scuttle: Polling Envoy (187), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:10:06Z scuttle: Polling Envoy (188), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:10:51Z scuttle: Polling Envoy (189), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:11:34Z scuttle: Polling Envoy (190), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:12:38Z scuttle: Polling Envoy (191), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:13:32Z scuttle: Polling Envoy (192), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:14:32Z scuttle: Polling Envoy (193), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:15:12Z scuttle: Polling Envoy (194), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:16:02Z scuttle: Polling Envoy (195), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:17:22Z scuttle: Polling Envoy (196), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:18:34Z scuttle: Polling Envoy (197), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:19:08Z scuttle: Polling Envoy (198), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:20:38Z scuttle: Polling Envoy (199), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:21:32Z scuttle: Polling Envoy (200), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:22:09Z scuttle: Polling Envoy (201), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:23:26Z scuttle: Polling Envoy (202), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:24:02Z scuttle: Polling Envoy (203), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:24:35Z scuttle: Polling Envoy (204), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:25:48Z scuttle: Polling Envoy (205), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:26:33Z scuttle: Polling Envoy (206), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:27:54Z scuttle: Polling Envoy (207), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:29:22Z scuttle: Polling Envoy (208), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:30:05Z scuttle: Polling Envoy (209), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:30:37Z scuttle: Polling Envoy (210), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:32:03Z scuttle: Polling Envoy (211), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:32:39Z scuttle: Polling Envoy (212), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:33:48Z scuttle: Polling Envoy (213), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:34:37Z scuttle: Polling Envoy (214), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:35:34Z scuttle: Polling Envoy (215), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused
2021-04-16T22:36:33Z scuttle: Polling Envoy (216), error: internal_service: dial tcp 127.0.0.1:15000: connect: connection refused

Any ideas on how to resolve it?

seems like it got resolved when we increase the replicas of istiod to 10 in our cluster. I don't know if it was memory constraint but we are no longer seeing this issue

I still have the issue for an argo workflow jobs, because the envoy is not ready I get connection refused error from main application..
istio proxy logs
2021-04-20T21:37:52.745567Z info Epoch 0 starting
2021-04-20T21:37:52.746060Z info Opening status port 15020
2021-04-20T21:37:53.024006Z info cache Root cert has changed, start rotating root cert
2021-04-20T21:37:53.024033Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-04-20T21:37:53.024046Z info cache generated new workload certificate latency=295.635545ms ttl=23h59m59.975961542s
2021-04-20T21:37:53.753728Z warn HTTP request unsuccessful with status: 400 Bad Request
2021-04-20T21:37:53.761401Z info Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster training.ravnml-dev --service-node sidecar~10.40.129.126~ravnml-w8857-3116787102.ravnml-dev~ravnml-dev.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2]
2021-04-20T21:37:53.801617Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T21:37:53.801662Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T21:37:53.802207Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T21:37:53.802261Z warning envoy runtime Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T21:37:58.847091Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-20T21:37:59.175870Z info ads ADS: new connection for node:sidecar~10.40.129.126~ravnml-w8857-3116787102.ravnml-dev~ravnml-dev.svc.cluster.local-2
2021-04-20T21:37:59.175974Z info cache returned workload certificate from cache ttl=23h59m53.824032267s
2021-04-20T21:37:59.176011Z info ads ADS: new connection for node:sidecar~10.40.129.126~ravnml-w8857-3116787102.ravnml-dev~ravnml-dev.svc.cluster.local-1
2021-04-20T21:37:59.176209Z info sds SDS: PUSH resource=default
2021-04-20T21:37:59.176309Z info sds SDS: PUSH resource=ROOTCA
2021-04-20T21:38:52.875059Z info Agent draining Proxy
2021-04-20T21:38:52.875071Z info Status server has successfully terminated
2021-04-20T21:38:52.875105Z info Watcher has successfully terminated
2021-04-20T21:38:52.875148Z error accept tcp [::]:15020: use of closed network connection
2021-04-20T21:38:52.876048Z info Graceful termination period is 5s, starting...
2021-04-20T21:38:57.876310Z info Graceful termination period complete, terminating remaining proxies.
2021-04-20T21:38:57.876345Z warn Aborting epoch 0...
2021-04-20T21:38:57.876351Z warn Aborted all epochs
2021-04-20T21:38:57.876354Z info Agent has successfully terminated
2021-04-20T21:38:57.876636Z warning envoy main caught ENVOY_SIGTERM

pod
akshay.uppal@CHI-C02CD4SAMD6P commons % kc describe pod ravnml-wjhr8-2019476748 -n ravnml-dev
Name:         ravnml-wjhr8-2019476748
Namespace:    ravnml-dev
Priority:     0
Node:         aks-mleff3-11574466-vmss000006/10.40.129.64
Start Time:   Tue, 20 Apr 2021 17:17:09 -0500
Labels:       app=training
              istio.io/rev=default
              pipelines.kubeflow.org/pipeline-sdk-type=kfp
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=training
              service.istio.io/canonical-revision=latest
              strategy=dyn-div-1
              workflows.argoproj.io/completed=true
              workflows.argoproj.io/controller-instanceid=ravnml-dev
              workflows.argoproj.io/workflow=ravnml-wjhr8
Annotations:  pipelines.kubeflow.org/component_spec:
                {"description": "Preprocess the data before training.", "inputs": [{"description": "Path to the data file(s)", "name": "data_path", "type"...
              pipelines.kubeflow.org/max_cache_staleness: P0D
              prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true
              sidecar.istio.io/inject: true
              sidecar.istio.io/status:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-...
              workflows.argoproj.io/node-name: ravnml-wjhr8[0].call-pipeline-template.exit-handler-1.preprocessing
              workflows.argoproj.io/template:
                {"name":"preprocessing","arguments":{},"inputs":{"parameters":[{"name":"data_path","value":"pq1"},{"name":"label_column","value":"label"},...
Status:       Failed
IP:           10.40.129.93
IPs:
  IP:           10.40.129.93
Controlled By:  Workflow/ravnml-wjhr8
Init Containers:
  istio-init:
    Container ID:  containerd://89000f7882b4585097642fa6d1f3a88e70d783f489c58ee6cb7198d679f87a8e
    Image:         docker.io/istio/proxyv2:1.9.1
    Image ID:      docker.io/istio/proxyv2@sha256:361ce2cb5b30094e9a08346aec2178a753f5844a3e7ed873c0deff2b4cae3607
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 20 Apr 2021 17:17:12 -0500
      Finished:     Tue, 20 Apr 2021 17:17:12 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
Containers:
  istio-proxy:
    Container ID:  containerd://66c864a6d654e9e68322958df3f39335b53e04be7aad73dc6afc1264dd1711ac
    Image:         docker.io/istio/proxyv2:1.9.1
    Image ID:      docker.io/istio/proxyv2@sha256:361ce2cb5b30094e9a08346aec2178a753f5844a3e7ed873c0deff2b4cae3607
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      training.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
      --concurrency
      2
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 20 Apr 2021 17:17:13 -0500
      Finished:     Tue, 20 Apr 2021 17:18:18 -0500
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    third-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      ravnml-wjhr8-2019476748 (v1:metadata.name)
      POD_NAMESPACE:                 ravnml-dev (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      CANONICAL_SERVICE:              (v1:metadata.labels['service.istio.io/canonical-name'])
      CANONICAL_REVISION:             (v1:metadata.labels['service.istio.io/canonical-revision'])
      PROXY_CONFIG:                  {"tracing":{"zipkin":{"address":"jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411"},"sampling":10},"holdApplicationUntilProxyStarts":true}

      ISTIO_META_POD_PORTS:          [
                                     ]
      ISTIO_META_APP_CONTAINERS:     wait,main
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_METAJSON_ANNOTATIONS:    {"pipelines.kubeflow.org/component_spec":"{\"description\": \"Preprocess the data before training.\", \"inputs\": [{\"description\": \"Path to the data file(s)\", \"name\": \"data_path\", \"type\": \"String\"}, {\"description\": \"Minio source bucket name.\", \"name\": \"source_bucket\", \"type\": \"String\"}, {\"description\": \"Minio target bucket name.\", \"name\": \"target_bucket\", \"type\": \"String\"}, {\"description\": \"Text column names in the dataset.\", \"name\": \"text_columns\", \"type\": \"String\"}, {\"description\": \"Only train selected labels.\", \"name\": \"selected_labels\", \"type\": \"String\"}, {\"description\": \"Label column name.\", \"name\": \"label_column\", \"type\": \"String\"}, {\"description\": \"Enable SVD dimensionality reduction.\", \"name\": \"svd_enabled\", \"type\": \"Boolean\"}, {\"default\": \"1000\", \"description\": \"number of result components in svd\", \"name\": \"svd_components\", \"type\": \"Integer\"}, {\"default\": \"200\", \"description\": \"number of iterations for svd\", \"name\": \"svd_iterations\", \"type\": \"Integer\"}, {\"default\": \"1, 2, 3, 5\", \"description\": \"ngram sizes\", \"name\": \"ngrams\", \"type\": \"String\"}, {\"default\": \"10000, 10000, 2000, 2000\", \"description\": \"ngram vocabulary sizes\", \"name\": \"ngrams_voc_size\", \"type\": \"String\"}], \"name\": \"Preprocessing\", \"outputs\": [{\"description\": \"List of discovered names (corresponding to Y columns)\", \"name\": \"labels\", \"type\": \"List\"}]}","pipelines.kubeflow.org/max_cache_staleness":"P0D","sidecar.istio.io/inject":"true","workflows.argoproj.io/node-name":"ravnml-wjhr8[0].call-pipeline-template.exit-handler-1.preprocessing","workflows.argoproj.io/template":"{\"name\":\"preprocessing\",\"arguments\":{},\"inputs\":{\"parameters\":[{\"name\":\"data_path\",\"value\":\"pq1\"},{\"name\":\"label_column\",\"value\":\"label\"},{\"name\":\"minio_endpoint\",\"value\":\"ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000\"},{\"name\":\"minio_secret\",\"value\":\"ravnml-dev-minio\"},{\"name\":\"ngrams\",\"value\":\"1, 2\"},{\"name\":\"ngrams_voc_size\",\"value\":\"100, 100\"},{\"name\":\"run_urn\",\"value\":\"urn:ravnml:us:1:run:39b1018d54ec44caab818653744a60f1\"},{\"name\":\"selected_labels\",\"value\":\"true,false\"},{\"name\":\"source_bucket\",\"value\":\"dataset-1-d0b8dd98df3740f0931c76ba23d8f81c\"},{\"name\":\"svd_components\",\"value\":\"1000\"},{\"name\":\"svd_enabled\",\"value\":\"false\"},{\"name\":\"svd_iterations\",\"value\":\"200\"},{\"name\":\"target_bucket\",\"value\":\"run-1-39b1018d54ec44caab818653744a60f1\"},{\"name\":\"text_columns\",\"value\":\"text\"}]},\"outputs\":{\"parameters\":[{\"name\":\"preprocessing-labels\",\"valueFrom\":{\"path\":\"/tmp/outputs/labels/data\"}}],\"artifacts\":[{\"name\":\"preprocessing-labels\",\"path\":\"/tmp/outputs/labels/data\"}]},\"nodeSelector\":{\"role\":\"training-cpu\"},\"metadata\":{\"annotations\":{\"pipelines.kubeflow.org/component_spec\":\"{\\\"description\\\": \\\"Preprocess the data before training.\\\", \\\"inputs\\\": [{\\\"description\\\": \\\"Path to the data file(s)\\\", \\\"name\\\": \\\"data_path\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Minio source bucket name.\\\", \\\"name\\\": \\\"source_bucket\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Minio target bucket name.\\\", \\\"name\\\": \\\"target_bucket\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Text column names in the dataset.\\\", \\\"name\\\": \\\"text_columns\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Only train selected labels.\\\", \\\"name\\\": \\\"selected_labels\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Label column name.\\\", \\\"name\\\": \\\"label_column\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Enable SVD dimensionality reduction.\\\", \\\"name\\\": \\\"svd_enabled\\\", \\\"type\\\": \\\"Boolean\\\"}, {\\\"default\\\": \\\"1000\\\", \\\"description\\\": \\\"number of result components in svd\\\", \\\"name\\\": \\\"svd_components\\\", \\\"type\\\": \\\"Integer\\\"}, {\\\"default\\\": \\\"200\\\", \\\"description\\\": \\\"number of iterations for svd\\\", \\\"name\\\": \\\"svd_iterations\\\", \\\"type\\\": \\\"Integer\\\"}, {\\\"default\\\": \\\"1, 2, 3, 5\\\", \\\"description\\\": \\\"ngram sizes\\\", \\\"name\\\": \\\"ngrams\\\", \\\"type\\\": \\\"String\\\"}, {\\\"default\\\": \\\"10000, 10000, 2000, 2000\\\", \\\"description\\\": \\\"ngram vocabulary sizes\\\", \\\"name\\\": \\\"ngrams_voc_size\\\", \\\"type\\\": \\\"String\\\"}], \\\"name\\\": \\\"Preprocessing\\\", \\\"outputs\\\": [{\\\"description\\\": \\\"List of discovered names (corresponding to Y columns)\\\", \\\"name\\\": \\\"labels\\\", \\\"type\\\": \\\"List\\\"}]}\",\"pipelines.kubeflow.org/max_cache_staleness\":\"P0D\",\"sidecar.istio.io/inject\":\"true\"},\"labels\":{\"app\":\"training\",\"pipelines.kubeflow.org/pipeline-sdk-type\":\"kfp\",\"strategy\":\"dyn-div-1\"}},\"container\":{\"name\":\"\",\"image\":\"docker.imanage.com/dyn-doc-preprocess:1.1.20\",\"command\":[\"python\",\"-m\",\"main\",\"--data_path\",\"pq1\",\"--source_bucket\",\"dataset-1-d0b8dd98df3740f0931c76ba23d8f81c\",\"--target_bucket\",\"run-1-39b1018d54ec44caab818653744a60f1\",\"--text_columns\",\"text\",\"--label_column\",\"label\",\"--svd_enabled\",\"false\",\"--svd_components\",\"1000\",\"--svd_iterations\",\"200\",\"--ngrams\",\"1, 2\",\"--ngrams_voc_size\",\"100, 100\",\"--selected_labels\",\"true,false\",\"--labels\",\"/tmp/outputs/labels/data\"],\"env\":[{\"name\":\"MINIO_ENDPOINT\",\"value\":\"ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000\"},{\"name\":\"MINIO_ACCESSKEY\",\"valueFrom\":{\"secretKeyRef\":{\"name\":\"ravnml-dev-minio\",\"key\":\"accesskey\"}}},{\"name\":\"MINIO_SECRETKEY\",\"valueFrom\":{\"secretKeyRef\":{\"name\":\"ravnml-dev-minio\",\"key\":\"secretkey\"}}},{\"name\":\"ravnml_runId\",\"value\":\"urn:ravnml:us:1:run:39b1018d54ec44caab818653744a60f1\"},{\"name\":\"EXCEPTIONS_BUCKET\",\"value\":\"run-1-39b1018d54ec44caab818653744a60f1\"},{\"name\":\"EXCEPTIONS_FOLDER\",\"value\":\"exceptions/\"}],\"resources\":{\"requests\":{\"cpu\":\"450m\"}},\"imagePullPolicy\":\"Always\"},\"archiveLocation\":{\"s3\":{\"endpoint\":\"ravnml-dev-minio:9000\",\"bucket\":\"argo-artifacts\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"ravnml-dev-minio\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"ravnml-dev-minio\",\"key\":\"secretkey\"},\"key\":\"ravnml-wjhr8/ravnml-wjhr8-2019476748\"}},\"tolerations\":[{\"key\":\"role\",\"operator\":\"Equal\",\"value\":\"training-cpu\",\"effect\":\"NoSchedule\"}]}"}

      ISTIO_META_WORKLOAD_NAME:      ravnml-wjhr8-2019476748
      ISTIO_META_OWNER:              kubernetes://apis/v1/namespaces/ravnml-dev/pods/ravnml-wjhr8-2019476748
      ISTIO_META_MESH_ID:            cluster.local
      TRUST_DOMAIN:                  cluster.local
    Mounts:
      /etc/certs/ from istio-certs (ro)
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
      /var/run/secrets/tokens from istio-token (rw)
  wait:
    Container ID:  containerd://4f195c93dd0b8128874998fda25b021d58ed92597e80fe3e9fc7b0fe108f3ae1
    Image:         argoproj/argoexec:v2.12.5
    Image ID:      docker.io/argoproj/argoexec@sha256:1841ef4c44e582f66af82091fd15eeb493faddb359a54fb75087c993b5404e0b
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Terminated
      Reason:       Error
      Message:      Failed to establish pod watch: timed out waiting for the condition
      Exit Code:    1
      Started:      Tue, 20 Apr 2021 17:18:18 -0500
      Finished:     Tue, 20 Apr 2021 17:18:53 -0500
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:                    ravnml-wjhr8-2019476748 (v1:metadata.name)
      ARGO_CONTAINER_RUNTIME_EXECUTOR:  pns
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /argo/secret/ravnml-dev-minio from ravnml-dev-minio (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
  main:
    Container ID:  containerd://42d60370658810dc0c6ed3eaea4d64a737874eab4952cec5d5c94c04d4aeaa6a
    Image:         docker.imanage.com/dyn-doc-preprocess:1.1.20
    Image ID:      docker.imanage.com/dyn-doc-preprocess@sha256:b644d2b1ef6d33b00f17c37ef0587d09d18722210b1d897b1d97a819de21a700
    Port:          <none>
    Host Port:     <none>
    Command:
      python
      -m
      main
      --data_path
      pq1
      --source_bucket
      dataset-1-d0b8dd98df3740f0931c76ba23d8f81c
      --target_bucket
      run-1-39b1018d54ec44caab818653744a60f1
      --text_columns
      text
      --label_column
      label
      --svd_enabled
      false
      --svd_components
      1000
      --svd_iterations
      200
      --ngrams
      1, 2
      --ngrams_voc_size
      100, 100
      --selected_labels
      true,false
      --labels
      /tmp/outputs/labels/data
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 20 Apr 2021 17:18:20 -0500
      Finished:     Tue, 20 Apr 2021 17:18:33 -0500
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:  450m
    Environment:
      MINIO_ENDPOINT:     ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000
      MINIO_ACCESSKEY:    <set to the key 'accesskey' in secret 'ravnml-dev-minio'>  Optional: false
      MINIO_SECRETKEY:    <set to the key 'secretkey' in secret 'ravnml-dev-minio'>  Optional: false
      ravnml_runId:       urn:ravnml:us:1:run:39b1018d54ec44caab818653744a60f1
      EXCEPTIONS_BUCKET:  run-1-39b1018d54ec44caab818653744a60f1
      EXCEPTIONS_FOLDER:  exceptions/
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
      limits.cpu -> cpu-limit
      requests.cpu -> cpu-request
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.argo-runner
    Optional:    true
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  ravnml-dev-minio:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ravnml-dev-minio
    Optional:    false
  argo-runner-token-6zm8g:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  argo-runner-token-6zm8g
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  role=training-cpu
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                 role=training-cpu:NoSchedule
Events:
  Type     Reason               Age    From               Message
  ----     ------               ----   ----               -------
  Normal   Scheduled            4m13s  default-scheduler  Successfully assigned ravnml-dev/ravnml-wjhr8-2019476748 to aks-mleff3-11574466-vmss000006
  Normal   Pulling              4m11s  kubelet            Pulling image "docker.io/istio/proxyv2:1.9.1"
  Normal   Pulled               4m11s  kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.1" in 799.05246ms
  Normal   Started              4m10s  kubelet            Started container istio-init
  Normal   Pulling              4m10s  kubelet            Pulling image "docker.io/istio/proxyv2:1.9.1"
  Normal   Created              4m10s  kubelet            Created container istio-init
  Normal   Pulled               4m9s   kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.1" in 795.790203ms
  Normal   Created              4m9s   kubelet            Created container istio-proxy
  Normal   Started              4m9s   kubelet            Started container istio-proxy
  Normal   Killing              3m9s   kubelet            FailedPostStartHook
  Warning  FailedPostStartHook  3m9s   kubelet            Exec lifecycle hook ([pilot-agent wait]) for Container "istio-proxy" in Pod "ravnml-wjhr8-2019476748_ravnml-dev(2687a3ef-9443-4e4a-b1be-2fbcc8bb137c)" failed - error: command 'pilot-agent wait' exited with 255: Error: timeout waiting for Envoy proxy to become ready. Last error: Get "http://localhost:15021/healthz/ready": dial tcp [::1]:15021: connect: connection refused
, message: "2021-04-20T22:17:13.348816Z\tinfo\tWaiting for Envoy proxy to be ready (timeout: 60 seconds)...\n2021-04-20T22:18:13.434887Z\terror\ttimeout waiting for Envoy proxy to become ready. Last error: Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused\nError: timeout waiting for Envoy proxy to become ready. Last error: Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused\n"
  Normal  Pulled   3m4s  kubelet  Container image "argoproj/argoexec:v2.12.5" already present on machine
  Normal  Created  3m4s  kubelet  Created container wait
  Normal  Started  3m4s  kubelet  Started container wait
  Normal  Pulling  3m4s  kubelet  Pulling image "docker.imanage.com/dyn-doc-preprocess:1.1.20"
  Normal  Pulled   3m2s  kubelet  Successfully pulled image "docker.imanage.com/dyn-doc-preprocess:1.1.20" in 1.393212301s
  Normal  Created  3m2s  kubelet  Created container main
  Normal  Started  3m2s  kubelet  Started container main

logs of main container
akshay.uppal@CHI-C02CD4SAMD6P commons % kc logs ravnml-zbzcv-1034734040 istio-proxy -n ravnml-dev
2021-04-20T22:31:04.284952Z	info	FLAG: --concurrency="2"
2021-04-20T22:31:04.284978Z	info	FLAG: --domain="ravnml-dev.svc.cluster.local"
2021-04-20T22:31:04.284983Z	info	FLAG: --help="false"
2021-04-20T22:31:04.284986Z	info	FLAG: --log_as_json="false"
2021-04-20T22:31:04.284988Z	info	FLAG: --log_caller=""
2021-04-20T22:31:04.284992Z	info	FLAG: --log_output_level="default:info"
2021-04-20T22:31:04.284994Z	info	FLAG: --log_rotate=""
2021-04-20T22:31:04.284997Z	info	FLAG: --log_rotate_max_age="30"
2021-04-20T22:31:04.285001Z	info	FLAG: --log_rotate_max_backups="1000"
2021-04-20T22:31:04.285003Z	info	FLAG: --log_rotate_max_size="104857600"
2021-04-20T22:31:04.285006Z	info	FLAG: --log_stacktrace_level="default:none"
2021-04-20T22:31:04.285012Z	info	FLAG: --log_target="[stdout]"
2021-04-20T22:31:04.285015Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2021-04-20T22:31:04.285018Z	info	FLAG: --outlierLogPath=""
2021-04-20T22:31:04.285020Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2021-04-20T22:31:04.285023Z	info	FLAG: --proxyLogLevel="warning"
2021-04-20T22:31:04.285026Z	info	FLAG: --serviceCluster="training.ravnml-dev"
2021-04-20T22:31:04.285029Z	info	FLAG: --stsPort="0"
2021-04-20T22:31:04.285031Z	info	FLAG: --templateFile=""
2021-04-20T22:31:04.285034Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-04-20T22:31:04.285039Z	info	Version 1.9.1-2dd7b6207f02cec8b42f4263ac197434f4ec9b4a-Clean
2021-04-20T22:31:04.285318Z	info	Apply proxy config from env {"tracing":{"zipkin":{"address":"jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411"},"sampling":10},"holdApplicationUntilProxyStarts":true}

2021-04-20T22:31:04.286147Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
holdApplicationUntilProxyStarts: true
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: training.ravnml-dev
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  sampling: 10
  zipkin:
    address: jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411

2021-04-20T22:31:04.286176Z	info	Proxy role	ips=[10.40.129.125 fe80::6875:17ff:feb9:c4d3] type=sidecar id=ravnml-zbzcv-1034734040.ravnml-dev domain=ravnml-dev.svc.cluster.local
2021-04-20T22:31:04.286182Z	info	JWT policy is third-party-jwt
2021-04-20T22:31:04.286195Z	info	Pilot SAN: [istiod.istio-system.svc]
2021-04-20T22:31:04.286202Z	info	CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-04-20T22:31:04.286239Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-20T22:31:04.286322Z	info	citadelclient	Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-04-20T22:31:04.314863Z	info	ads	All caches have been synced up in 31.529774ms, marking server ready
2021-04-20T22:31:04.315145Z	info	sds	SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-04-20T22:31:04.315163Z	info	xdsproxy	Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-04-20T22:31:04.315254Z	info	sds	Start SDS grpc server
2021-04-20T22:31:04.315303Z	info	Starting proxy agent
2021-04-20T22:31:04.315342Z	info	Opening status port 15020
2021-04-20T22:31:04.315350Z	info	Received new config, creating new Envoy epoch 0
2021-04-20T22:31:04.315385Z	info	Epoch 0 starting
2021-04-20T22:31:04.414719Z	info	cache	Root cert has changed, start rotating root cert
2021-04-20T22:31:04.414752Z	info	ads	XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-04-20T22:31:04.414766Z	info	cache	generated new workload certificate	latency=99.652696ms ttl=23h59m59.585242684s
2021-04-20T22:31:05.326531Z	warn	HTTP request unsuccessful with status: 400 Bad Request
2021-04-20T22:31:05.332062Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster training.ravnml-dev --service-node sidecar~10.40.129.125~ravnml-zbzcv-1034734040.ravnml-dev~ravnml-dev.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error --concurrency 2]
2021-04-20T22:31:05.372852Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:31:05.372900Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:31:05.373449Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:31:05.373494Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:31:05.418922Z	info	xdsproxy	connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-20T22:31:05.733537Z	info	ads	ADS: new connection for node:sidecar~10.40.129.125~ravnml-zbzcv-1034734040.ravnml-dev~ravnml-dev.svc.cluster.local-1
2021-04-20T22:31:05.733617Z	info	cache	returned workload certificate from cache	ttl=23h59m58.266389188s
2021-04-20T22:31:05.733669Z	info	ads	ADS: new connection for node:sidecar~10.40.129.125~ravnml-zbzcv-1034734040.ravnml-dev~ravnml-dev.svc.cluster.local-2
2021-04-20T22:31:05.733922Z	info	sds	SDS: PUSH	resource=default
2021-04-20T22:31:05.734046Z	info	sds	SDS: PUSH	resource=ROOTCA
2021-04-20T22:31:06.891473Z	info	Initialization took 2.614116237s
2021-04-20T22:31:06.891505Z	info	Envoy proxy is ready
{"authority":"ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000","user_agent":"python-urllib3/1.26.3","bytes_sent":128,"duration":25,"upstream_transport_failure_reason":null,"start_time":"2021-04-20T22:31:09.827Z","x_forwarded_for":null,"downstream_remote_address":"10.40.129.125:34464","connection_termination_details":null,"protocol":"HTTP/1.1","response_flags":"-","response_code":200,"downstream_local_address":"10.0.86.46:9000","response_code_details":"via_upstream","path":"/run-1-6e7dae7041e54fc2b4c19487ef4f2a90?location=","method":"GET","upstream_service_time":"24","upstream_host":"10.40.136.100:9000","upstream_local_address":"10.40.129.125:51406","route_name":"default","request_id":"b095ad84-3f87-4f3a-b468-5a71e11e3d83","bytes_received":0,"upstream_cluster":"outbound|9000||ravnml-dev-minio.ravnml-dev.svc.cluster.local","requested_server_name":null}
2021-04-20T22:31:10.427840Z	info	Agent draining Proxy
2021-04-20T22:31:10.427859Z	info	Status server has successfully terminated
2021-04-20T22:31:10.427881Z	info	Watcher has successfully terminated
2021-04-20T22:31:10.427944Z	error	accept tcp [::]:15020: use of closed network connection
2021-04-20T22:31:10.428298Z	info	Graceful termination period is 5s, starting...
2021-04-20T22:31:15.428424Z	info	Graceful termination period complete, terminating remaining proxies.
2021-04-20T22:31:15.428463Z	warn	Aborting epoch 0...
2021-04-20T22:31:15.428471Z	warn	Aborted all epochs
2021-04-20T22:31:15.428476Z	info	Agent has successfully terminated
2021-04-20T22:31:15.428508Z	warn	Aborting epoch 0
akshay.uppal@CHI-C02CD4SAMD6P commons % kc logs ravnml-k25cx-2111753272 istio-proxy -n ravnml-dev
2021-04-20T22:34:24.452689Z	info	FLAG: --concurrency="2"
2021-04-20T22:34:24.452716Z	info	FLAG: --domain="ravnml-dev.svc.cluster.local"
2021-04-20T22:34:24.452720Z	info	FLAG: --help="false"
2021-04-20T22:34:24.452723Z	info	FLAG: --log_as_json="false"
2021-04-20T22:34:24.452726Z	info	FLAG: --log_caller=""
2021-04-20T22:34:24.452729Z	info	FLAG: --log_output_level="default:info"
2021-04-20T22:34:24.452731Z	info	FLAG: --log_rotate=""
2021-04-20T22:34:24.452734Z	info	FLAG: --log_rotate_max_age="30"
2021-04-20T22:34:24.452737Z	info	FLAG: --log_rotate_max_backups="1000"
2021-04-20T22:34:24.452739Z	info	FLAG: --log_rotate_max_size="104857600"
2021-04-20T22:34:24.452742Z	info	FLAG: --log_stacktrace_level="default:none"
2021-04-20T22:34:24.452749Z	info	FLAG: --log_target="[stdout]"
2021-04-20T22:34:24.452752Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2021-04-20T22:34:24.452754Z	info	FLAG: --outlierLogPath=""
2021-04-20T22:34:24.452757Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2021-04-20T22:34:24.452760Z	info	FLAG: --proxyLogLevel="warning"
2021-04-20T22:34:24.452762Z	info	FLAG: --serviceCluster="training.ravnml-dev"
2021-04-20T22:34:24.452765Z	info	FLAG: --stsPort="0"
2021-04-20T22:34:24.452768Z	info	FLAG: --templateFile=""
2021-04-20T22:34:24.452772Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-04-20T22:34:24.452776Z	info	Version 1.9.1-2dd7b6207f02cec8b42f4263ac197434f4ec9b4a-Clean
2021-04-20T22:34:24.453039Z	info	Apply proxy config from env {"tracing":{"zipkin":{"address":"jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411"},"sampling":10},"holdApplicationUntilProxyStarts":true}

2021-04-20T22:34:24.453958Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
holdApplicationUntilProxyStarts: true
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: training.ravnml-dev
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  sampling: 10
  zipkin:
    address: jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411

2021-04-20T22:34:24.453984Z	info	Proxy role	ips=[10.40.129.123 fe80::346e:75ff:fe36:6fb4] type=sidecar id=ravnml-k25cx-2111753272.ravnml-dev domain=ravnml-dev.svc.cluster.local
2021-04-20T22:34:24.453989Z	info	JWT policy is third-party-jwt
2021-04-20T22:34:24.454005Z	info	Pilot SAN: [istiod.istio-system.svc]
2021-04-20T22:34:24.454016Z	info	CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2021-04-20T22:34:24.454062Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-20T22:34:24.454193Z	info	citadelclient	Citadel client using custom root cert: istiod.istio-system.svc:15012
2021-04-20T22:34:24.479381Z	info	ads	All caches have been synced up in 28.143398ms, marking server ready
2021-04-20T22:34:24.480067Z	info	sds	SDS server for workload certificates started, listening on "./etc/istio/proxy/SDS"
2021-04-20T22:34:24.480115Z	info	xdsproxy	Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2021-04-20T22:34:24.480348Z	info	sds	Start SDS grpc server
2021-04-20T22:34:24.480495Z	info	Starting proxy agent
2021-04-20T22:34:24.480492Z	info	Opening status port 15020
2021-04-20T22:34:24.480568Z	info	Received new config, creating new Envoy epoch 0
2021-04-20T22:34:24.480749Z	info	Epoch 0 starting
2021-04-20T22:34:24.621851Z	info	cache	Root cert has changed, start rotating root cert
2021-04-20T22:34:24.621880Z	info	ads	XDS: Incremental Pushing:0 ConnectedEndpoints:0 Version:
2021-04-20T22:34:24.621891Z	info	cache	generated new workload certificate	latency=141.818412ms ttl=23h59m59.378115366s
2021-04-20T22:34:25.494960Z	warn	HTTP request unsuccessful with status: 400 Bad Request
2021-04-20T22:34:25.500829Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster training.ravnml-dev --service-node sidecar~10.40.129.123~ravnml-k25cx-2111753272.ravnml-dev~ravnml-dev.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error --concurrency 2]
2021-04-20T22:34:25.542125Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:34:25.542168Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:34:25.542728Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:34:25.542781Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-20T22:34:35.585461Z	info	xdsproxy	connected to upstream XDS server: istiod.istio-system.svc:15012
2021-04-20T22:34:35.934205Z	info	ads	ADS: new connection for node:sidecar~10.40.129.123~ravnml-k25cx-2111753272.ravnml-dev~ravnml-dev.svc.cluster.local-1
2021-04-20T22:34:35.934248Z	info	ads	ADS: new connection for node:sidecar~10.40.129.123~ravnml-k25cx-2111753272.ravnml-dev~ravnml-dev.svc.cluster.local-2
2021-04-20T22:34:35.934319Z	info	cache	returned workload certificate from cache	ttl=23h59m48.06568726s
2021-04-20T22:34:35.934559Z	info	sds	SDS: PUSH	resource=ROOTCA
2021-04-20T22:34:35.934616Z	info	sds	SDS: PUSH	resource=default
2021-04-20T22:35:24.587226Z	info	Agent draining Proxy
2021-04-20T22:35:24.587228Z	info	Watcher has successfully terminated
2021-04-20T22:35:24.587227Z	info	Status server has successfully terminated
2021-04-20T22:35:24.587332Z	error	accept tcp [::]:15020: use of closed network connection
2021-04-20T22:35:24.588169Z	info	Graceful termination period is 5s, starting...
2021-04-20T22:35:29.588310Z	info	Graceful termination period complete, terminating remaining proxies.
2021-04-20T22:35:29.588352Z	warn	Aborting epoch 0...
2021-04-20T22:35:29.588368Z	warn	Aborted all epochs
2021-04-20T22:35:29.588388Z	info	Agent has successfully terminated
2021-04-20T22:35:29.588399Z	warn	Aborting epoch 0
akshay.uppal@CHI-C02CD4SAMD6P commons % kc logs ravnml-k25cx-2111753272 main -n ravnml-dev
{"@timestamp": "2021-04-20T22:35:32.131531", "level": "INFO", "message": "Logging configured to JSON format.", "caller": "log.py::setup", "data": {"python.logger_name": "root", "python.module": "log", "python.funcName": "setup", "python.filename": "log.py", "python.lineno": 60, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.132975", "level": "DEBUG", "message": "Namespace(X='tmp/data/false/X', Y='tmp/data/false/Y', base='tmp/data/false/crossvalidate', folds='/tmp/outputs/folds/data', nfolds=3, target_bucket='run-1-9a6a5e78183747e5a9be7758e8a7f027')", "caller": "main.py::run", "data": {"python.logger_name": "__main__", "python.module": "main", "python.funcName": "run", "python.filename": "main.py", "python.lineno": 85, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.133428", "level": "DEBUG", "message": "X: tmp/data/false/X", "caller": "main.py::main", "data": {"python.logger_name": "__main__", "python.module": "main", "python.funcName": "main", "python.filename": "main.py", "python.lineno": 38, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.133765", "level": "DEBUG", "message": "Y: tmp/data/false/Y", "caller": "main.py::main", "data": {"python.logger_name": "__main__", "python.module": "main", "python.funcName": "main", "python.filename": "main.py", "python.lineno": 39, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.134091", "level": "INFO", "message": "Loading X / Y from Minio", "caller": "main.py::main", "data": {"python.logger_name": "__main__", "python.module": "main", "python.funcName": "main", "python.filename": "main.py", "python.lineno": 41, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.135333", "level": "DEBUG", "message": "Starting new HTTP connection (1): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.147387", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=4, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.147803", "level": "WARNING", "message": "Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f95bb70>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.148295", "level": "DEBUG", "message": "Starting new HTTP connection (2): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.162608", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=3, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.563677", "level": "WARNING", "message": "Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f983ba8>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.564397", "level": "DEBUG", "message": "Starting new HTTP connection (3): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:32.587316", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=2, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:33.388794", "level": "WARNING", "message": "Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f983d68>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:33.389499", "level": "DEBUG", "message": "Starting new HTTP connection (4): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:33.408207", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=1, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:35.010568", "level": "WARNING", "message": "Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f983e10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:35.011285", "level": "DEBUG", "message": "Starting new HTTP connection (5): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:35.029199", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=0, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.233082", "level": "WARNING", "message": "Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6bc128>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.233833", "level": "DEBUG", "message": "Starting new HTTP connection (6): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.248455", "level": "DEBUG", "message": "Starting new HTTP connection (1): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.258323", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=4, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.258724", "level": "WARNING", "message": "Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6bc860>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.259205", "level": "DEBUG", "message": "Starting new HTTP connection (2): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.270683", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=3, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.671823", "level": "WARNING", "message": "Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6bcb70>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.672533", "level": "DEBUG", "message": "Starting new HTTP connection (3): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:38.688501", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=2, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:39.490055", "level": "WARNING", "message": "Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6bccf8>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:39.490757", "level": "DEBUG", "message": "Starting new HTTP connection (4): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:39.500667", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=1, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:41.103085", "level": "WARNING", "message": "Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6bceb8>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:41.103786", "level": "DEBUG", "message": "Starting new HTTP connection (5): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:46.165979", "level": "DEBUG", "message": "Incremented Retry for (url='/run-1-9a6a5e78183747e5a9be7758e8a7f027?location='): Retry(total=0, connect=None, read=None, redirect=None, status=None)", "caller": "retry.py::increment", "data": {"python.logger_name": "urllib3.util.retry", "python.module": "retry", "python.funcName": "increment", "python.filename": "retry.py", "python.lineno": 575, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:49.370008", "level": "WARNING", "message": "Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6d21d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /run-1-9a6a5e78183747e5a9be7758e8a7f027?location=", "caller": "connectionpool.py::urlopen", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "urlopen", "python.filename": "connectionpool.py", "python.lineno": 781, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
{"@timestamp": "2021-04-20T22:35:49.370729", "level": "DEBUG", "message": "Starting new HTTP connection (6): ravnml-dev-minio.ravnml-dev.svc.cluster.local:9000", "caller": "connectionpool.py::_new_conn", "data": {"python.logger_name": "urllib3.connectionpool", "python.module": "connectionpool", "python.funcName": "_new_conn", "python.filename": "connectionpool.py", "python.lineno": 231, "python.thread": "MainThread", "python.pid": 107}, "runId": "urn:ravnml:us:1:run:9a6a5e78183747e5a9be7758e8a7f027"}
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f1b8f6bc2e8>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kftools/io/miniolib.py", line 119, in get
    http_response = self.get_object(bucket, name)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 879, in get_object
    extra_query_params=extra_query_params,
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 1766, in _get_partial_object
    query=extra_query_params,
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 2185, in _url_open
    region = self._get_bucket_region(bucket_name)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 2063, in _get_bucket_region
    region = self._get_bucket_location(bucket_name)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 2098, in _get_bucket_location
    headers=headers)
  File "/usr/local/lib/python3.6/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
    **response_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
    **response_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
    **response_kw
  [Previous line repeated 2 more times]
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='ravnml-dev-minio.ravnml-dev.svc.cluster.local', port=9000): Max retries exceeded with url: /run-1-9a6a5e78183747e5a9be7758e8a7f027?location= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6bc2e8>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/ravnml_exception_handler/persistence.py", line 41, in wrapper_debug
    value = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/main.py", line 93, in run
    folds_path=Path(args.folds)
  File "/usr/local/lib/python3.6/site-packages/main.py", line 44, in main
    X = minio.get_pkl(target_bucket, X_path)
  File "/usr/local/lib/python3.6/site-packages/kftools/io/miniolib.py", line 147, in get_pkl
    return self.get(bucket, name, lambda bs: pickle.loads(bs, encoding='utf-8'))
  File "/usr/local/lib/python3.6/site-packages/kftools/io/miniolib.py", line 123, in get
    raise MinioDownloadError(str(e))
kftools.io.miniolib.MinioDownloadError: HTTPConnectionPool(host='ravnml-dev-minio.ravnml-dev.svc.cluster.local', port=9000): Max retries exceeded with url: /run-1-9a6a5e78183747e5a9be7758e8a7f027?location= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6bc2e8>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.6/http/client.py", line 1287, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1333, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1282, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1042, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 980, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f1b8f6d2278>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kftools/io/miniolib.py", line 112, in store
    self.put_object(bucket, name, BytesIO(s_data), len(s_data))
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 1060, in put_object
    progress=progress)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 1830, in _do_put_object
    content_sha256=sha256_hex
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 2185, in _url_open
    region = self._get_bucket_region(bucket_name)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 2063, in _get_bucket_region
    region = self._get_bucket_location(bucket_name)
  File "/usr/local/lib/python3.6/site-packages/minio/api.py", line 2098, in _get_bucket_location
    headers=headers)
  File "/usr/local/lib/python3.6/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
    **response_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
    **response_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 796, in urlopen
    **response_kw
  [Previous line repeated 2 more times]
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='ravnml-dev-minio.ravnml-dev.svc.cluster.local', port=9000): Max retries exceeded with url: /run-1-9a6a5e78183747e5a9be7758e8a7f027?location= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6d2278>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/site-packages/main.py", line 98, in <module>
    run()
  File "/usr/local/lib/python3.6/site-packages/ravnml_exception_handler/persistence.py", line 51, in wrapper_debug
    minio.store_json(bucket, filename, exc_info)
  File "/usr/local/lib/python3.6/site-packages/kftools/io/miniolib.py", line 159, in store_json
    self.store(bucket, name, data, lambda d: json.dumps(d).encode('UTF-8'))
  File "/usr/local/lib/python3.6/site-packages/kftools/io/miniolib.py", line 114, in store
    raise MinioUploadError(f'Uploading [{name}] failed: [{str(e)}].')
kftools.io.miniolib.MinioUploadError: Uploading [exceptions/04-20-2021_22:35:38:247521.json] failed: [HTTPConnectionPool(host='ravnml-dev-minio.ravnml-dev.svc.cluster.local', port=9000): Max retries exceeded with url: /run-1-9a6a5e78183747e5a9be7758e8a7f027?location= (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1b8f6d2278>: Failed to establish a new connection: [Errno 111] Connection refused',))].

From the logs, i can see the sidecar is terminated with a SIG_TERM. Do you have liveness probe set for the pod container.

I dont have liveness probe. So what happens is that we get the error that the envoy is not ready and when main starts it fails as the envoy is not ready and we get the connection refused error.
adding the full description of the container
akshay.uppal@CHI-C02CD4SAMD6P commons % kc describe pod/****-98kdr-3549724327 -n ****-dev
Name:         ****-98kdr-3549724327
Namespace:    ****-dev
Priority:     0
Node:         aks-mleff3-11574466-vmss000006/10.40.129.64
Start Time:   Wed, 21 Apr 2021 13:55:28 -0500
Labels:       app=training
              istio.io/rev=default
              pipelines.kubeflow.org/pipeline-sdk-type=kfp
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=training
              service.istio.io/canonical-revision=latest
              strategy=dyn-div-1
              workflows.argoproj.io/completed=true
              workflows.argoproj.io/controller-instanceid=****-dev
              workflows.argoproj.io/workflow=****-98kdr
Annotations:  pipelines.kubeflow.org/component_spec:
                {"description": "Training the data", "inputs": [{"description": "Minio target bucket name.", "name": "target_bucket", "type": "String"}, {...
              pipelines.kubeflow.org/max_cache_staleness: P0D
              prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true
              sidecar.istio.io/inject: true
              sidecar.istio.io/status:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-...
              workflows.argoproj.io/node-name:
                ****-98kdr[0].call-pipeline-template.exit-handler-1.for-loop-for-loop-7e33a43d-2(1:X:tmp/data/true/X,label:true,y:tmp/data/true/Y).for-l...
              workflows.argoproj.io/template:
                {"name":"train-2","arguments":{},"inputs":{"parameters":[{"name":"crossvalidationsplit-folds-loop-item-subvar-fold","value":"2"},{"name":"...
Status:       Failed
IP:           10.40.129.169
IPs:
  IP:           10.40.129.169
Controlled By:  Workflow/****-98kdr
Init Containers:
  istio-init:
    Container ID:  containerd://0752b34a16c9981dc6f04afc52a2e3e44a3583ec4b500f80254b59c19691e3ce
    Image:         docker.io/istio/proxyv2:1.9.1
    Image ID:      docker.io/istio/proxyv2@sha256:361ce2cb5b30094e9a08346aec2178a753f5844a3e7ed873c0deff2b4cae3607
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 21 Apr 2021 13:55:31 -0500
      Finished:     Wed, 21 Apr 2021 13:55:31 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
Containers:
  istio-proxy:
    Container ID:  containerd://aba88c3565eebf1168c70ef23060076917ccc6d70504ae16c7df81b069753377
    Image:         docker.io/istio/proxyv2:1.9.1
    Image ID:      docker.io/istio/proxyv2@sha256:361ce2cb5b30094e9a08346aec2178a753f5844a3e7ed873c0deff2b4cae3607
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      training.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
      --concurrency
      2
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 21 Apr 2021 13:55:33 -0500
      Finished:     Wed, 21 Apr 2021 13:56:39 -0500
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    third-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      ****-98kdr-3549724327 (v1:metadata.name)
      POD_NAMESPACE:                 ****-dev (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      CANONICAL_SERVICE:              (v1:metadata.labels['service.istio.io/canonical-name'])
      CANONICAL_REVISION:             (v1:metadata.labels['service.istio.io/canonical-revision'])
      PROXY_CONFIG:                  {"tracing":{"zipkin":{"address":"jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411"},"sampling":10},"holdApplicationUntilProxyStarts":true}

      ISTIO_META_POD_PORTS:          [
                                     ]
      ISTIO_META_APP_CONTAINERS:     wait,main
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_META_WORKLOAD_NAME:      ****-98kdr-3549724327
      ISTIO_META_OWNER:              kubernetes://apis/v1/namespaces/****-dev/pods/****-98kdr-3549724327
      ISTIO_META_MESH_ID:            cluster.local
      TRUST_DOMAIN:                  cluster.local
    Mounts:
      /etc/certs/ from istio-certs (ro)
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
      /var/run/secrets/tokens from istio-token (rw)
  wait:
    Container ID:  containerd://67955234aa6ae5cfe6b84bb48f648e7c7f0da6ea4cdb6fd28877229734ba0d1d
    Image:         argoproj/argoexec:v2.12.5
    Image ID:      docker.io/argoproj/argoexec@sha256:1841ef4c44e582f66af82091fd15eeb493faddb359a54fb75087c993b5404e0b
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
    State:          Terminated
      Reason:       Error
      Message:      Failed to establish pod watch: timed out waiting for the condition
      Exit Code:    1
      Started:      Wed, 21 Apr 2021 13:56:39 -0500
      Finished:     Wed, 21 Apr 2021 13:57:13 -0500
    Ready:          False
    Restart Count:  0
    Environment:
      ARGO_POD_NAME:                    ****-98kdr-3549724327 (v1:metadata.name)
      ARGO_CONTAINER_RUNTIME_EXECUTOR:  pns
    Mounts:
      /argo/podmetadata from podmetadata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
  main:
    Container ID:  containerd://5911ca1c5e2f39d994bacf2c4384d5eed54c324aa97787581f40516dfb7dc8dd
    Image:         docker.imanage.com/dyn-div-train:0.2.8
    Image ID:      docker.imanage.com/dyn-div-train@sha256:76ee89da10cb4f97a22628856f6bab1ef652155c4df99a012449128eb831bfb0
    Port:          <none>
    Host Port:     <none>
    Command:
      python
      -m
      main
      --target_bucket
      run-1-1f2b808460e44cadb3adb34416f696c5
      --model_path
      tmp/model/true/crossvalidate/2/model.pkl
      --X
      tmp/data/true/crossvalidate/2/Xtrain
      --Y
      tmp/data/true/crossvalidate/2/ytrain
      --n_estimators
      -100
      --max_depth
      50
      --min_samples_leaf
      1
      --max_features
      auto
      --random_state
      42
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 21 Apr 2021 13:56:40 -0500
      Finished:     Wed, 21 Apr 2021 13:56:47 -0500
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:  450m
    Environment:
      MINIO_ENDPOINT:     ****-dev-minio.****-dev.svc.cluster.local:9000
      MINIO_ACCESSKEY:    <set to the key 'accesskey' in secret '****-dev-minio'>  Optional: false
      MINIO_SECRETKEY:    <set to the key 'secretkey' in secret '****-dev-minio'>  Optional: false
      ****_runId:       urn:****:us:1:run:1f2b808460e44cadb3adb34416f696c5
      EXCEPTIONS_BUCKET:  run-1-1f2b808460e44cadb3adb34416f696c5
      EXCEPTIONS_FOLDER:  exceptions/
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from argo-runner-token-6zm8g (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
      limits.cpu -> cpu-limit
      requests.cpu -> cpu-request
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  istio-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio.argo-runner
    Optional:    true
  podmetadata:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  argo-runner-token-6zm8g:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  argo-runner-token-6zm8g
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  role=training-cpu
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                 role=training-cpu:NoSchedule
Events:
  Type     Reason               Age    From               Message
  ----     ------               ----   ----               -------
  Normal   Scheduled            4m12s  default-scheduler  Successfully assigned ****-dev/****-98kdr-3549724327 to aks-mleff3-11574466-vmss000006
  Normal   Pulling              4m11s  kubelet            Pulling image "docker.io/istio/proxyv2:1.9.1"
  Normal   Pulled               4m9s   kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.1" in 2.196182182s
  Normal   Started              4m9s   kubelet            Started container istio-init
  Normal   Pulling              4m9s   kubelet            Pulling image "docker.io/istio/proxyv2:1.9.1"
  Normal   Created              4m9s   kubelet            Created container istio-init
  Normal   Pulled               4m7s   kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.1" in 1.835766822s
  Normal   Created              4m7s   kubelet            Created container istio-proxy
  Normal   Started              4m7s   kubelet            Started container istio-proxy
  Normal   Killing              3m6s   kubelet            FailedPostStartHook
  Warning  FailedPostStartHook  3m6s   kubelet            Exec lifecycle hook ([pilot-agent wait]) for Container "istio-proxy" in Pod "****-98kdr-3549724327_****-dev(f0140293-8505-4462-964b-7d6dcb43f93a)" failed - error: command 'pilot-agent wait' exited with 255: Error: timeout waiting for Envoy proxy to become ready. Last error: Get "http://localhost:15021/healthz/ready": dial tcp [::1]:15021: connect: connection refused
, message: "2021-04-21T18:55:34.026534Z\tinfo\tWaiting for Envoy proxy to be ready (timeout: 60 seconds)...\n2021-04-21T18:56:34.102096Z\terror\ttimeout waiting for Envoy proxy to become ready. Last error: Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused\nError: timeout waiting for Envoy proxy to become ready. Last error: Get \"http://localhost:15021/healthz/ready\": dial tcp [::1]:15021: connect: connection refused\n"
  Normal  Pulled   3m1s  kubelet  Container image "argoproj/argoexec:v2.12.5" already present on machine
  Normal  Created  3m1s  kubelet  Created container wait
  Normal  Started  3m1s  kubelet  Started container wait
  Normal  Pulling  3m1s  kubelet  Pulling image "docker.imanage.com/dyn-div-train:0.2.8"
  Normal  Pulled   3m    kubelet  Successfully pulled image "docker.imanage.com/dyn-div-train:0.2.8" in 1.005611534s
  Normal  Created  3m    kubelet  Created container main
  Normal  Started  3m    kubelet  Started container main

I also tried holdApplicationUntilProxyStarts for the operator but it doesn't seem to help for the case when envoy is not ready.
istiocontrolplane
$ kc get istiooperator --all-namespaces -o yaml
apiVersion: v1
items:
- apiVersion: install.istio.io/v1alpha1
  kind: IstioOperator
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"install.istio.io/v1alpha1","kind":"IstioOperator","metadata":{"annotations":{},"name":"istiocontrolplane","namespace":"istio-system"},"spec":{"addonComponents":{"grafana":{"enabled":false},"kiali":{"enabled":false},"prometheus":{"enabled":false},"telemetry":{"enabled":false},"tracing":{"enabled":false}},"components":{"egressGateways":[{"enabled":true,"k8s":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"topologyKey":"topology.kubernetes.io/zone"},"weight":100}]}},"hpaSpec":{"maxReplicas":3,"minReplicas":2},"podDisruptionBudget":{"minAvailable":1},"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"500m","memory":"500Mi"}}},"name":"istio-egressgateway"}],"ingressGateways":[{"enabled":true,"k8s":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"topologyKey":"topology.kubernetes.io/zone"},"weight":100}]}},"hpaSpec":{"maxReplicas":10,"minReplicas":3},"podDisruptionBudget":{"minAvailable":1},"resources":{"limits":{"memory":"1Gi"},"requests":{"cpu":"500m","memory":"500Mi"}},"service":{"loadBalancerIP":"20.49.240.148"},"serviceAnnotations":{"service.beta.kubernetes.io/azure-load-balancer-resource-group":"ravn-dev-uk"}},"name":"istio-ingressgateway"},{"enabled":true,"k8s":{"hpaSpec":{"maxReplicas":10,"minReplicas":3},"service":{"ports":[{"name":"status-port","port":15020},{"name":"http2","port":80,"targetPort":8080},{"name":"https","port":443,"targetPort":8443}],"type":"ClusterIP"}},"label":{"app":"cluster-local-gateway","istio":"cluster-local-gateway"},"name":"cluster-local-gateway"}],"pilot":{"k8s":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"topologyKey":"topology.kubernetes.io/zone"},"weight":100}]}},"hpaSpec":{"maxReplicas":10,"minReplicas":10},"podDisruptionBudget":{"minAvailable":1},"resources":{"limits":{"memory":"4Gi"},"requests":{"cpu":"1","memory":"4Gi"}}}}},"meshConfig":{"accessLogEncoding":"JSON","accessLogFile":"/dev/stdout","defaultConfig":{"holdApplicationUntilProxyStarts":true,"tracing":{"sampling":10,"zipkin":{"address":"jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411"}}}},"profile":"default","values":{"global":{"mountMtlsCerts":true,"proxy":{"tracer":"zipkin"}}}}}
    creationTimestamp: "2021-01-26T15:30:02Z"
    finalizers:
    - istio-finalizer.install.istio.io
    generation: 34
    managedFields:
      ***
    name: istiocontrolplane
    namespace: istio-system
    resourceVersion: "117745088"
    selfLink: /apis/install.istio.io/v1alpha1/namespaces/istio-system/istiooperators/istiocontrolplane
    uid: c129db54-005f-4d0d-a87f-6597e5605f63
  spec:
    addonComponents:
      grafana:
        enabled: false
      kiali:
        enabled: false
      prometheus:
        enabled: false
      telemetry:
        enabled: false
      tracing:
        enabled: false
    components:
      egressGateways:
      - enabled: true
        k8s:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  topologyKey: topology.kubernetes.io/zone
                weight: 100
          hpaSpec:
            maxReplicas: 3
            minReplicas: 2
          podDisruptionBudget:
            minAvailable: 1
          resources:
            limits:
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 500Mi
        name: istio-egressgateway
      ingressGateways:
      - enabled: true
        k8s:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  topologyKey: topology.kubernetes.io/zone
                weight: 100
          hpaSpec:
            maxReplicas: 10
            minReplicas: 3
          podDisruptionBudget:
            minAvailable: 1
          resources:
            limits:
              memory: 1Gi
            requests:
              cpu: 500m
              memory: 500Mi
          service:
            loadBalancerIP: 20.49.240.148
          serviceAnnotations:
            service.beta.kubernetes.io/azure-load-balancer-resource-group: ravn-dev-uk
        name: istio-ingressgateway
      - enabled: true
        k8s:
          hpaSpec:
            maxReplicas: 10
            minReplicas: 3
          service:
            ports:
            - name: status-port
              port: 15020
            - name: http2
              port: 80
              targetPort: 8080
            - name: https
              port: 443
              targetPort: 8443
            type: ClusterIP
        label:
          app: cluster-local-gateway
          istio: cluster-local-gateway
        name: cluster-local-gateway
      pilot:
        k8s:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  topologyKey: topology.kubernetes.io/zone
                weight: 100
          hpaSpec:
            maxReplicas: 10
            minReplicas: 10
          podDisruptionBudget:
            minAvailable: 1
          resources:
            limits:
              memory: 4Gi
            requests:
              cpu: "1"
              memory: 4Gi
    meshConfig:
      accessLogEncoding: JSON
      accessLogFile: /dev/stdout
      defaultConfig:
        holdApplicationUntilProxyStarts: true
        tracing:
          sampling: 10
          zipkin:
            address: jaeger-operator-jaeger-collector.ravn-metrics.svc.cluster.local:9411
    profile: default
    values:
      global:
        mountMtlsCerts: true
        proxy:
          tracer: zipkin
  status:
    componentStatus:
      Base:
        status: HEALTHY
      EgressGateways:
        status: HEALTHY
      IngressGateways:
        status: HEALTHY
      Pilot:
        status: HEALTHY
    status: HEALTHY
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

logs for other container
wait logs
$ kc logs ****-98kdr-3549724327 wait -n ****-dev
time="2021-04-21T18:56:39.390Z" level=info msg="Starting Workflow Executor" version=v2.12.5
time="2021-04-21T18:56:39.393Z" level=info msg="Creating PNS executor (namespace: ****-dev, pod: ****-98kdr-3549724327, pid: 85, hasOutputs: false)"
time="2021-04-21T18:56:39.393Z" level=info msg="Executor (version: v2.12.5, build_date: 2021-01-19T22:43:04Z) initialized (pod: ****-dev/****-98kdr-3549724327) with template:\n{\"name\":\"train-2\",\"arguments\":{},\"inputs\":{\"parameters\":[{\"name\":\"crossvalidationsplit-folds-loop-item-subvar-fold\",\"value\":\"2\"},{\"name\":\"crossvalidationsplit-folds-loop-item-subvar-xtrain\",\"value\":\"tmp/data/true/crossvalidate/2/Xtrain\"},{\"name\":\"crossvalidationsplit-folds-loop-item-subvar-ytrain\",\"value\":\"tmp/data/true/crossvalidate/2/ytrain\"},{\"name\":\"max_depth\",\"value\":\"50\"},{\"name\":\"max_features\",\"value\":\"auto\"},{\"name\":\"min_samples_leaf\",\"value\":\"1\"},{\"name\":\"minio_endpoint\",\"value\":\"****-dev-minio.****-dev.svc.cluster.local:9000\"},{\"name\":\"minio_secret\",\"value\":\"****-dev-minio\"},{\"name\":\"n_estimators\",\"value\":\"-100\"},{\"name\":\"preprocessing-labels-loop-item-subvar-label\",\"value\":\"true\"},{\"name\":\"run_urn\",\"value\":\"urn:****:us:1:run:1f2b808460e44cadb3adb34416f696c5\"},{\"name\":\"seed\",\"value\":\"42\"},{\"name\":\"target_bucket\",\"value\":\"run-1-1f2b808460e44cadb3adb34416f696c5\"}]},\"outputs\":{},\"nodeSelector\":{\"role\":\"training-cpu\"},\"metadata\":{\"annotations\":{\"pipelines.kubeflow.org/component_spec\":\"{\\\"description\\\": \\\"Training the data\\\", \\\"inputs\\\": [{\\\"description\\\": \\\"Minio target bucket name.\\\", \\\"name\\\": \\\"target_bucket\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Minio target model path.\\\", \\\"name\\\": \\\"model_path\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Minio location of X Matrix.\\\", \\\"name\\\": \\\"x\\\", \\\"type\\\": \\\"DataFrame\\\"}, {\\\"description\\\": \\\"Minio location of Y Matrix.\\\", \\\"name\\\": \\\"y\\\", \\\"type\\\": \\\"DataFrame\\\"}, {\\\"description\\\": \\\"Number of trees in forest\\\", \\\"name\\\": \\\"n_estimators\\\", \\\"type\\\": \\\"Integer\\\"}, {\\\"description\\\": \\\"Max depth of the tree\\\", \\\"name\\\": \\\"max_depth\\\", \\\"type\\\": \\\"Integer\\\"}, {\\\"description\\\": \\\"Min samples required to be a leaf node\\\", \\\"name\\\": \\\"min_samples_leaf\\\", \\\"type\\\": \\\"Integer\\\"}, {\\\"description\\\": \\\"features to consider when looking for the best split\\\", \\\"name\\\": \\\"max_features\\\", \\\"type\\\": \\\"String\\\"}, {\\\"description\\\": \\\"Controls the randomness for random forest ( split etc.)\\\", \\\"name\\\": \\\"random_state\\\", \\\"type\\\": \\\"Integer\\\"}], \\\"name\\\": \\\"Train\\\"}\",\"pipelines.kubeflow.org/max_cache_staleness\":\"P0D\",\"sidecar.istio.io/inject\":\"true\"},\"labels\":{\"app\":\"training\",\"pipelines.kubeflow.org/pipeline-sdk-type\":\"kfp\",\"strategy\":\"dyn-div-1\"}},\"container\":{\"name\":\"\",\"image\":\"docker.imanage.com/dyn-div-train:0.2.8\",\"command\":[\"python\",\"-m\",\"main\",\"--target_bucket\",\"run-1-1f2b808460e44cadb3adb34416f696c5\",\"--model_path\",\"tmp/model/true/crossvalidate/2/model.pkl\",\"--X\",\"tmp/data/true/crossvalidate/2/Xtrain\",\"--Y\",\"tmp/data/true/crossvalidate/2/ytrain\",\"--n_estimators\",\"-100\",\"--max_depth\",\"50\",\"--min_samples_leaf\",\"1\",\"--max_features\",\"auto\",\"--random_state\",\"42\"],\"env\":[{\"name\":\"MINIO_ENDPOINT\",\"value\":\"****-dev-minio.****-dev.svc.cluster.local:9000\"},{\"name\":\"MINIO_ACCESSKEY\",\"valueFrom\":{\"secretKeyRef\":{\"name\":\"****-dev-minio\",\"key\":\"accesskey\"}}},{\"name\":\"MINIO_SECRETKEY\",\"valueFrom\":{\"secretKeyRef\":{\"name\":\"****-dev-minio\",\"key\":\"secretkey\"}}},{\"name\":\"****_runId\",\"value\":\"urn:****:us:1:run:1f2b808460e44cadb3adb34416f696c5\"},{\"name\":\"EXCEPTIONS_BUCKET\",\"value\":\"run-1-1f2b808460e44cadb3adb34416f696c5\"},{\"name\":\"EXCEPTIONS_FOLDER\",\"value\":\"exceptions/\"}],\"resources\":{\"requests\":{\"cpu\":\"450m\"}},\"imagePullPolicy\":\"Always\"},\"tolerations\":[{\"key\":\"role\",\"operator\":\"Equal\",\"value\":\"training-cpu\",\"effect\":\"NoSchedule\"}]}"
time="2021-04-21T18:56:39.393Z" level=info msg="Waiting on main container"
time="2021-04-21T18:56:51.453Z" level=error msg="executor error: Failed to establish pod watch: timed out waiting for the condition\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapErrorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:78\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).waitMainContainerStart\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:1077\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).Wait\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:1030\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:40\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:13\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
time="2021-04-21T18:56:51.453Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-04-21T18:56:51.453Z" level=info msg="Capturing script exit code"
time="2021-04-21T18:56:51.453Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:56:52.666Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:56:54.816Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:56:57.461Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:57:01.878Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:57:01.878Z" level=error msg="executor error: Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).getPod\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:655\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).GetMainContainerStatus\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:727\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).GetMainContainerID\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:744\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).CaptureScriptExitCode\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:805\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:53\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:13\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
time="2021-04-21T18:57:01.878Z" level=info msg="Killing sidecars"
time="2021-04-21T18:57:01.879Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:57:02.928Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:57:04.769Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:57:07.990Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:57:13.752Z" level=warning msg="Failed to get pod '****-98kdr-3549724327': Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused"
time="2021-04-21T18:57:13.753Z" level=error msg="executor error: Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).getPod\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:655\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).KillSidecars\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:1249\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:33\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:56\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:13\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
time="2021-04-21T18:57:13.753Z" level=info msg="Alloc=4471 TotalAlloc=13129 Sys=71616 NumGC=5 Goroutines=5"
time="2021-04-21T18:57:13.753Z" level=fatal msg="Get https://10.0.0.1:443/api/v1/namespaces/****-dev/pods/****-98kdr-3549724327: dial tcp 10.0.0.1:443: connect: connection refused\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).getPod\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:655\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).GetMainContainerStatus\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:727\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).GetMainContainerID\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:744\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).CaptureScriptExitCode\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:805\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:53\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:13\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"

main logs as explained above

I think we'd need to have XdsProxy debug log (with sidecar.istio.io/agentLogLevel: xdsproxy:debug pod annotation) and istiod log around that time range to debug. istio bug report tool would be helpful to collect logs from all istiod instances.

Attaching bug report files
analyze.zip
bug-report.log
cluster.zip
istio.zip
operator.zip
proxies.zip
versions.zip

Is there a way to use liveness probe to start the pod only when istio proxy is ready @hzxuzhonghu ?

I enabled the debug logs with annotation sidecar.istio.io/logLevel: debug.
Looking at the logs it seems to be DNS issue
2021-04-23T12:57:25.006042Z	debug	envoy main	flushing stats
2021-04-23T12:57:25.006078Z	debug	envoy main	Envoy is not fully initialized, skipping histogram merge and flushing stats
2021-04-23T12:57:25.362166Z	debug	envoy upstream	DNS request timed out 1 times

Comparing with istio envoy logs  when envoy was in fact ready..
error_envoy.txt
error_envoy.log
Uploading enovy_ready.log…

authors: akshayuppal3 bianpengyuan howardjohn hzxuzhonghu