text:
Connection returns TLS / Certificate verification error in proxy after enabling SDS

Connection returns TLS / Certificate verification error in proxy after enabling SDS
Hi, I am having a problem with istio in my current production setup and would need your help to troubleshoot it.
Bug Description
Background
I am running Istio 1.1.7 in all our environments on kubernetes (amazon eks) 1.12.7 with mtls enable on application namespace, sds in both ingress gateway and sidecar.
There is no circuit breaker, no custom root CA for citadel.
Problem
The behaviour I saw is at first, all services in cluster are working fine, connection from ingress controller hit the services and return correctly.
But after a while, days or weeks, i haven’t been able to find the pattern, all connections from ingress to services return 503 UF, URX.
There are logs in istio-proxy container of ingress pod but no log in the upstream service’s istio-proxy container.
In example log (sorry for the format, i pull it out from elasticsearch)
"stream_name": "istio-ingressgateway-76749b4bb4-z6n78",
"istio_policy_status": "-",
"bytes_sent": "91",
"upstream_cluster": "outbound|8080||frontend.services.svc.cluster.local",
"downstream_remote_address": "172.23.24.174:30690",
"path": "/user",
"authority": "prod.example.com",
"protocol": "HTTP/1.1",
"upstream_service_time": "-",
"upstream_local_address": "-",
"duration": "69",
"downstream_local_address": "172.23.24.189:443",
"response_code": "503",
"user_agent": "Mozilla/5.0 (Linux; Android 8.0.0) ...",
"response_flags": "UF,URX",
"start_time": "2019-06-03T13:26:06.617Z",
"method": "GET",
"request_id": "320037db-601b-9c52-861f-bwoeifwoiegi",
"upstream_host": "172.23.24.143:80",
"x_forwarded_for": "218.186.146.112,172.23.24.174",
"requested_server_name": "prod.example.com",
"bytes_received": "0",

I tried to enable debug logging in proxy sidecar with
curl -XPOST localhost:15000/logging?connection=debug

then i found this in the isito-proxy container of the ingress controller:
[2019-05-21 08:18:36.878][33][debug][connection] [external/envoy/source/common/network/connection_impl.cc:644] [C79846] connecting to 172.23.14.229:80
[2019-05-21 08:18:36.878][33][debug][connection] [external/envoy/source/common/network/connection_impl.cc:517] [C79846] connected
[2019-05-21 08:18:36.878][33][debug][connection] [external/envoy/source/common/network/connection_impl.cc:653] [C79846] connection in progress
[2019-05-21 08:18:36.878][33][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C79846] handshake error: 2
[2019-05-21 08:18:36.883][33][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C79846] handshake error: 2
[2019-05-21 08:18:36.883][33][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C79846] handshake error: 2
[2019-05-21 08:18:36.885][33][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C79846] handshake error: 1
[2019-05-21 08:18:36.885][33][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:175] [C79846] TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED
[2019-05-21 08:18:36.885][33][debug][connection] [external/envoy/source/common/network/connection_impl.cc:183] [C79846] closing socket: 0

So it looks like there are some problem with the TLS cert. The cert in istio-ca-secret and istio.istio-ingressgateway-service-account look correct and are not expired yet. Same goes for the internal certificates for my upstream services.
And as far as I can tell, this only happens when the service pods runs for a few days without being restarted or deployed with a new version.
I also saw another instance of the problem, but these logs were found inside the upstream service’s istio-proxy container, and the TLS error is different from the one in the ingress controller:
[2019-06-04 01:18:58.029][32][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C400] handshake error: 2
[2019-06-04 01:18:58.029][32][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C400] handshake error: 2
[2019-06-04 01:18:58.031][32][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:142] [C400] handshake error: 1
[2019-06-04 01:18:58.031][32][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:175] [C400] TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
[2019-06-04 01:18:58.031][32][debug][connection] [external/envoy/source/common/network/connection_impl.cc:183] [C400] closing socket: 0

I am not sure what actually happened here; The citadel logs, node agent logs and the rest looked normal at that point in time.
Please let me know if there are any other logs/config you need to troubleshoot the problem.
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[X] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastrcture
Steps to reproduce the bug
As mentioned previously, we still don't know how exactly to reproduce it, but based these are the patterns we observed so far:

Enable SDS
Deploy applications and let them run for 2 - 4 days without any re-deployment / bouncing
Observe for 503 errors on the proxy logs

Version (include the output of istioctl version --remote and kubectl version)
Istio 1.1.7 in all our environments
Kubernetes (AWS EKS) 1.12.7
How was Istio installed?
We installed it using Helm, via Tiller

@JimmyCYJ @quanjielin @wattli @myidpt

@Janesee3 did you turn on sds on workload sidecar or ingress or both ?  could you pls share the install yaml file, and nodeagent logs ?

Hi, we turned on SDS for both the sidecar and ingress.
Here is the values file we used for our Istio installation:
# Top level istio values file has the following sections.
#
# global: This file is the authoritative and exhaustive source for the global section.
#
# chart sections: Every subdirectory inside the charts/ directory has a top level
#       configuration key in this file. This file overrides the values specified
#       by the charts/${chartname}/values.yaml.
#       Check the chart level values file for exhaustive list of configuration options.

#
# Gateways Configuration, refer to the charts/gateways/values.yaml
# for detailed configuration
#
gateways:
  enabled: true
  istio-ingressgateway:
    enabled: true
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    sds:
      enabled: true
    ports:
      - port: 15020
        targetPort: 15020
        name: status-port
      - port: 80
        targetPort: 80
        name: http2
        nodePort: 31380
      - port: 443
        name: https
        nodePort: 31390
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 2000m
        memory: 256Mi
  istio-egressgateway:
    enabled: true
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 2000m
        memory: 256Mi
  custom-ingressgateway:
    enabled: true
    #
    # Secret Discovery Service (SDS) configuration for ingress gateway.
    #
    sds:
      # If true, ingress gateway fetches credentials from SDS server to handle TLS connections.
      enabled: true
      # SDS server that watches kubernetes secrets and provisions credentials to ingress gateway.
      # This server runs in the same pod as ingress gateway.
      image: node-agent-k8s
      resources:
        requests:
          cpu: 50m
          memory: 128Mi
        limits:
          cpu: 300m
          memory: 128Mi

    labels:
      app: custom-ingressgateway
      istio: custom-ingressgateway
    autoscaleEnabled: true
    autoscaleMin: 1
    autoscaleMax: 5
    # specify replicaCount when autoscaleEnabled: false
    # replicaCount: 1
    resources:
      requests:
        cpu: 100m
        memory: 128Mi
      limits:
        cpu: 2000m
        memory: 256Mi
    cpu:
      targetAverageUtilization: 80
    loadBalancerIP: ""
    loadBalancerSourceRanges: []
    externalIPs: []
    serviceAnnotations:
      helm.sh/resource-policy: keep
      service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
    podAnnotations: {}
    type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be
    #externalTrafficPolicy: Local #change to Local to preserve source IP or Cluster for default behaviour or leave commented out
    ports:
      ## You can add custom gateway ports
      # Note that AWS ELB will by default perform health checks on the first port
      # on this list. Setting this to the health check port will ensure that health
      # checks always work. https://github.com/istio/istio/issues/12503
      - port: 15020
        targetPort: 15020
        name: status-port
      - port: 80
        targetPort: 80
        name: http2
        nodePort: 32380
      - port: 443
        name: https
        nodePort: 32390
    ##############
    secretVolumes:
      - name: ingressgateway-certs
        secretName: istio-ingressgateway-certs
        mountPath: /etc/istio/ingressgateway-certs
      - name: ingressgateway-ca-certs
        secretName: istio-ingressgateway-ca-certs
        mountPath: /etc/istio/ingressgateway-ca-certs
    ### Advanced options ############

    # Ports to explicitly check for readiness. If configured, the readiness check will expect a
    # listener on these ports. A comma separated list is expected, such as "80,443".
    #
    # Warning: If you do not have a gateway configured for the ports provided, this check will always
    # fail. This is intended for use cases where you always expect to have a listener on the port,
    # such as 80 or 443 in typical setups.
    applicationPorts: ""

    env:
      # A gateway with this mode ensures that pilot generates an additional
      # set of clusters for internal services but without Istio mTLS, to
      # enable cross cluster routing.
      ISTIO_META_ROUTER_MODE: "sni-dnat"
    nodeSelector: {}

    # Specify the pod anti-affinity that allows you to constrain which nodes
    # your pod is eligible to be scheduled based on labels on pods that are
    # already running on the node rather than based on labels on nodes.
    # There are currently two types of anti-affinity:
    #    "requiredDuringSchedulingIgnoredDuringExecution"
    #    "preferredDuringSchedulingIgnoredDuringExecution"
    # which denote “hard” vs. “soft” requirements, you can define your values
    # in "podAntiAffinityLabelSelector" and "podAntiAffinityTermLabelSelector"
    # correspondingly.
    # For example:
    # podAntiAffinityLabelSelector:
    # - key: security
    #   operator: In
    #   values: S1,S2
    #   topologyKey: "kubernetes.io/hostname"
    # This pod anti-affinity rule says that the pod requires not to be scheduled
    # onto a node if that node is already running a pod with label having key
    # “security” and value “S1”.
    podAntiAffinityLabelSelector: []
    podAntiAffinityTermLabelSelector: []

#
# sidecar-injector webhook configuration, refer to the
# charts/sidecarInjectorWebhook/values.yaml for detailed configuration
#
sidecarInjectorWebhook:
  enabled: true
  rewriteAppHTTPProbe: true

#
# galley configuration, refer to charts/galley/values.yaml
# for detailed configuration
#
galley:
  enabled: true

#
# mixer configuration
#
# @see charts/mixer/values.yaml, it takes precedence
mixer:
  enabled: true
  policy:
    # if policy is enabled the global.disablePolicyChecks has affect.
    enabled: true

  telemetry:
    enabled: true
    resources:
      requests:
        cpu: 500m
        memory: 1G
      limits:
        # It is best to do horizontal scaling of mixer using moderate cpu allocation.
        # We have experimentally found that these values work well.
        cpu: 500m
        memory: 1G
#
# pilot configuration
#
# @see charts/pilot/values.yaml
pilot:
  enabled: true
  traceSampling: 0
  resources:
    requests:
      cpu: 200m
      memory: 1048Mi

#
# security configuration
#
security:
  enabled: true

#
# nodeagent configuration
#
nodeagent:
  enabled: true
  env:
    CA_PROVIDER: "Citadel"
    CA_ADDR: "istio-citadel:8060"
    VALID_TOKEN: true

#
# addon grafana configuration
#
grafana:
  enabled: false

#
# addon prometheus configuration
#
prometheus:
  enabled: false

#
# addon servicegraph configuration
#
servicegraph:
  enabled: false

#
# addon jaeger tracing configuration
#
tracing:
  enabled: false

#
# addon kiali tracing configuration
#
kiali:
  enabled: false

#
# Istio CNI plugin enabled
#   This must be enabled to use the CNI plugin in Istio.  The CNI plugin is installed separately.
#   If true, the privileged initContainer istio-init is not needed to perform the traffic redirect
#   settings for the istio-proxy.
#
istio_cni:
  enabled: false

# addon Istio CoreDNS configuration
#
istiocoredns:
  enabled: false

# Common settings used among istio subcharts.
global:
  # Default hub for Istio images.
  # Releases are published to docker hub under 'istio' project.
  # Daily builds from prow are on gcr.io, and nightly builds from circle on docker.io/istionightly
  hub: docker.io/istio

  # Default tag for Istio images.
  tag: 1.1.8

  # Comma-separated minimum per-scope logging level of messages to output, in the form of <scope>:<level>,<scope>:<level>
  # The control plane has different scopes depending on component, but can configure default log level across all components
  # If empty, default scope and level will be used as configured in code
  logging:
    level: "default:info"

  # monitoring port used by mixer, pilot, galley
  monitoringPort: 15014

  k8sIngress:
    enabled: false
    # Gateway used for k8s Ingress resources. By default it is
    # using 'istio:ingressgateway' that will be installed by setting
    # 'gateways.enabled' and 'gateways.istio-ingressgateway.enabled'
    # flags to true.
    gatewayName: ingressgateway
    # enableHttps will add port 443 on the ingress.
    # It REQUIRES that the certificates are installed  in the
    # expected secrets - enabling this option without certificates
    # will result in LDS rejection and the ingress will not work.
    enableHttps: false

  proxy:
    image: proxyv2

    # cluster domain. Default value is "cluster.local".
    clusterDomain: "cluster.local"

    # Resources for the sidecar.
    resources:
      requests:
        cpu: 50m
        memory: 50Mi
      limits:
        cpu: 300m
        memory: 128Mi

    # Controls number of Proxy worker threads.
    # If set to 0 (default), then start worker thread for each CPU thread/core.
    concurrency: 2

    # Configures the access log for each sidecar.
    # Options:
    #   "" - disables access log
    #   "/dev/stdout" - enables access log
    accessLogFile: "/dev/stdout"

    # Configure how and what fields are displayed in sidecar access log. Setting to
    # empty string will result in default log format
    accessLogFormat: ""

    # Configure the access log for sidecar to JSON or TEXT.
    accessLogEncoding: JSON

    # Log level for proxy, applies to gateways and sidecars.  If left empty, "warning" is used.
    # Expected values are: trace|debug|info|warning|error|critical|off
    logLevel: ""

    # Configure the DNS refresh rate for Envoy cluster of type STRICT_DNS
    # 5 seconds is the default refresh rate used by Envoy
    dnsRefreshRate: 5s

    #If set to true, istio-proxy container will have privileged securityContext
    privileged: false

    # If set, newly injected sidecars will have core dumps enabled.
    enableCoreDump: false

    # Default port for Pilot agent health checks. A value of 0 will disable health checking.
    statusPort: 15020

    # The initial delay for readiness probes in seconds.
    readinessInitialDelaySeconds: 1

    # The period between readiness probes.
    readinessPeriodSeconds: 2

    # The number of successive failed probes before indicating readiness failure.
    readinessFailureThreshold: 30

    # istio egress capture whitelist
    # https://istio.io/docs/tasks/traffic-management/egress.html#calling-external-services-directly
    # example: includeIPRanges: "172.30.0.0/16,172.20.0.0/16"
    # would only capture egress traffic on those two IP Ranges, all other outbound traffic would
    # be allowed by the sidecar
    includeIPRanges: "*"
    excludeIPRanges: ""

    # pod internal interfaces
    kubevirtInterfaces: ""

    # istio ingress capture whitelist
    # examples:
    #     Redirect no inbound traffic to Envoy:    --includeInboundPorts=""
    #     Redirect all inbound traffic to Envoy:   --includeInboundPorts="*"
    #     Redirect only selected ports:            --includeInboundPorts="80,8080"
    includeInboundPorts: "*"
    excludeInboundPorts: ""

    # This controls the 'policy' in the sidecar injector.
    autoInject: enabled

    # Sets the destination Statsd in envoy (the value of the "--statsdUdpAddress" proxy argument
    # would be <host>:<port>).
    # Disabled by default.
    # The istio-statsd-prom-bridge is deprecated and should not be used moving forward.
    envoyStatsd:
      # If enabled is set to true, host and port must also be provided. Istio no longer provides a statsd collector.
      enabled: false
      host: # example: statsd-svc.istio-system
      port: # example: 9125

    # Sets the Envoy Metrics Service address, used to push Envoy metrics to an external collector
    # via the Metrics Service gRPC API. This contains detailed stats information emitted directly
    # by Envoy and should not be confused with the the Istio telemetry. The Envoy stats are also
    # available to scrape via the Envoy admin port at either /stats or /stats/prometheus.
    #
    # See https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/metrics/v2/metrics_service.proto
    # for details about Envoy's Metrics Service API.
    #
    # Disabled by default.
    envoyMetricsService:
      enabled: false
      host: # example: metrics-service.istio-system
      port: # example: 15000

    # Specify which tracer to use. One of: lightstep, zipkin, datadog
    tracer: "zipkin"

  proxy_init:
    # Base name for the proxy_init container, used to configure iptables.
    image: proxy_init

  # imagePullPolicy is applied to istio control plane components.
  # local tests require IfNotPresent, to avoid uploading to dockerhub.
  # TODO: Switch to Always as default, and override in the local tests.
  imagePullPolicy: IfNotPresent

  # controlPlaneMtls enabled. Will result in delays starting the pods while secrets are
  # propagated, not recommended for tests.
  controlPlaneSecurityEnabled: false

  # disablePolicyChecks disables mixer policy checks.
  # if mixer.policy.enabled==true then disablePolicyChecks has affect.
  # Will set the value with same name in istio config map - pilot needs to be restarted to take effect.
  disablePolicyChecks: true

  # policyCheckFailOpen allows traffic in cases when the mixer policy service cannot be reached.
  # Default is false which means the traffic is denied when the client is unable to connect to Mixer.
  policyCheckFailOpen: false

  # EnableTracing sets the value with same name in istio config map, requires pilot restart to take effect.
  enableTracing: true

  # Configuration for each of the supported tracers
  tracer:
    # Configuration for envoy to send trace data to LightStep.
    # Disabled by default.
    # address: the <host>:<port> of the satellite pool
    # accessToken: required for sending data to the pool
    # secure: specifies whether data should be sent with TLS
    # cacertPath: the path to the file containing the cacert to use when verifying TLS. If secure is true, this is
    #   required. If a value is specified then a secret called "lightstep.cacert" must be created in the destination
    #   namespace with the key matching the base of the provided cacertPath and the value being the cacert itself.
    #
    lightstep:
      address: ""                # example: lightstep-satellite:443
      accessToken: ""            # example: abcdefg1234567
      secure: true               # example: true|false
      cacertPath: ""             # example: /etc/lightstep/cacert.pem
    zipkin:
      # Host:Port for reporting trace data in zipkin format. If not specified, will default to
      # zipkin service (port 9411) in the same namespace as the other istio components.
      address: ""
    datadog:
      # Host:Port for submitting traces to the Datadog agent.
      address: "$(HOST_IP):8126"

  # Default mtls policy. If true, mtls between services will be enabled by default.
  mtls:
    # Default setting for service-to-service mtls. Can be set explicitly using
    # destination rules or service annotations.
    enabled: false

  # ImagePullSecrets for all ServiceAccount, list of secrets in the same namespace
  # to use for pulling any images in pods that reference this ServiceAccount.
  # For components that don't use ServiceAccounts (i.e. grafana, servicegraph, tracing)
  # ImagePullSecrets will be added to the corresponding Deployment(StatefulSet) objects.
  # Must be set for any clustser configured with private docker registry.
  imagePullSecrets:
  # - private-registry-key

  # Specify pod scheduling arch(amd64, ppc64le, s390x) and weight as follows:
  #   0 - Never scheduled
  #   1 - Least preferred
  #   2 - No preference
  #   3 - Most preferred
  arch:
    amd64: 2
    s390x: 2
    ppc64le: 2

  # Whether to restrict the applications namespace the controller manages;
  # If not set, controller watches all namespaces
  oneNamespace: false

  # Default node selector to be applied to all deployments so that all pods can be
  # constrained to run a particular nodes. Each component can overwrite these default
  # values by adding its node selector block in the relevant section below and setting
  # the desired values.
  defaultNodeSelector: {}

  # Whether to perform server-side validation of configuration.
  configValidation: true

  # Custom DNS config for the pod to resolve names of services in other
  # clusters. Use this to add additional search domains, and other settings.
  # see
  # https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#dns-config
  # This does not apply to gateway pods as they typically need a different
  # set of DNS settings than the normal application pods (e.g., in
  # multicluster scenarios).
  # NOTE: If using templates, follow the pattern in the commented example below.
  #podDNSSearchNamespaces:
  #- global
  #- "[[ valueOrDefault .DeploymentMeta.Namespace \"default\" ]].global"

  # If set to true, the pilot and citadel mtls will be exposed on the
  # ingress gateway
  meshExpansion:
    enabled: false
    # If set to true, the pilot and citadel mtls and the plain text pilot ports
    # will be exposed on an internal gateway
    useILB: false

  multiCluster:
    # Set to true to connect two kubernetes clusters via their respective
    # ingressgateway services when pods in each cluster cannot directly
    # talk to one another. All clusters should be using Istio mTLS and must
    # have a shared root CA for this model to work.
    enabled: false

  # A minimal set of requested resources to applied to all deployments so that
  # Horizontal Pod Autoscaler will be able to function (if set).
  # Each component can overwrite these default values by adding its own resources
  # block in the relevant section below and setting the desired resources values.
  defaultResources:
    requests:
      cpu: 10m
    #   memory: 128Mi
    # limits:
    #   cpu: 100m
    #   memory: 128Mi

  # enable pod distruption budget for the control plane, which is used to
  # ensure Istio control plane components are gradually upgraded or recovered.
  defaultPodDisruptionBudget:
    enabled: true
    # The values aren't mutable due to a current PodDisruptionBudget limitation
    # minAvailable: 1

  # Kubernetes >=v1.11.0 will create two PriorityClass, including system-cluster-critical and
  # system-node-critical, it is better to configure this in order to make sure your Istio pods
  # will not be killed because of low priority class.
  # Refer to https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
  # for more detail.
  priorityClassName: ""

  # Use the Mesh Control Protocol (MCP) for configuring Mixer and
  # Pilot. Requires galley (`--set galley.enabled=true`).
  useMCP: true

  # The trust domain corresponds to the trust root of a system
  # Refer to https://github.com/spiffe/spiffe/blob/master/standards/SPIFFE-ID.md#21-trust-domain
  # Indicate the domain used in SPIFFE identity URL
  # The default depends on the environment.
  #   kubernetes: cluster.local
  #   else:  default dns domain
  trustDomain: ""

  # Set the default behavior of the sidecar for handling outbound traffic from the application:
  # ALLOW_ANY - outbound traffic to unknown destinations will be allowed, in case there are no
  #   services or ServiceEntries for the destination port
  # REGISTRY_ONLY - restrict outbound traffic to services defined in the service registry as well
  #   as those defined through ServiceEntries
  # ALLOW_ANY is the default in 1.1.  This means each pod will be able to make outbound requests
  # to services outside of the mesh without any ServiceEntry.
  # REGISTRY_ONLY was the default in 1.0.  If this behavior is desired, set the value below to REGISTRY_ONLY.
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY

  # The namespace where globally shared configurations should be present.
  # DestinationRules that apply to the entire mesh (e.g., enabling mTLS),
  # default Sidecar configs, etc. should be added to this namespace.
  # configRootNamespace: istio-config

  # set the default set of namespaces to which services, service entries, virtual services, destination
  # rules should be exported to. Currently only one value can be provided in this list. This value
  # should be one of the following two options:
  # * implies these objects are visible to all namespaces, enabling any sidecar to talk to any other sidecar.
  # . implies these objects are visible to only to sidecars in the same namespace, or if imported as a Sidecar.egress.host
  #defaultConfigVisibilitySettings:
  #- '*'

  sds:
    # SDS enabled. IF set to true, mTLS certificates for the sidecars will be
    # distributed through the SecretDiscoveryService instead of using K8S secrets to mount the certificates.
    enabled: true
    udsPath: "unix:/var/run/sds/uds_path"
    useTrustworthyJwt: false
    useNormalJwt: true
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        cpu: 300m
        memory: 128Mi

  # Configure the mesh networks to be used by the Split Horizon EDS.
  #
  # The following example defines two networks with different endpoints association methods.
  # For `network1` all endpoints that their IP belongs to the provided CIDR range will be
  # mapped to network1. The gateway for this network example is specified by its public IP
  # address and port.
  # The second network, `network2`, in this example is defined differently with all endpoints
  # retrieved through the specified Multi-Cluster registry being mapped to network2. The
  # gateway is also defined differently with the name of the gateway service on the remote
  # cluster. The public IP for the gateway will be determined from that remote service (not
  # supported yet).
  #
  # meshNetworks:
  #   network1:
  #     endpoints:
  #     - fromCidr: "192.168.0.1/24"
  #     gateways:
  #     - address: 1.1.1.1
  #       port: 80
  #   network2:
  #     endpoints:
  #     - fromRegistry: reg1
  #     gateways:
  #     - registryServiceName: istio-ingressgateway
  #       port: 443
  #
  meshNetworks: {}

  # Specifies the global locality load balancing settings.
  # Locality-weighted load balancing allows administrators to control the distribution of traffic to
  # endpoints based on the localities of where the traffic originates and where it will terminate.
  # Please set either failover or distribute configuration but not both.
  #
  # localityLbSetting:
  #   distribute:
  #   - from: "us-central1/*"
  #     to:
  #       "us-central1/*": 80
  #       "us-central2/*": 20
  #
  # localityLbSetting:
  #   failover:
  #   - from: us-east
  #     to: eu-west
  #   - from: us-west
  #     to: us-east
  localityLbSetting: {}

  # Specifies whether helm test is enabled or not.
  # This field is set to false by default, so 'helm template ...'
  # will ignore the helm test yaml files when generating the template
  enableHelmTest: false

Node agent log shows that "failed to get root cert". Seems that node agent does not get root cert, perhaps the connection between node agent and Citadel timeouts. Could you provide more node agent logs and check if there are errors such as "CSR retry timeout " or "CSR failed "?

Sorry for the misleading logs, it seems that the "Failed to get root cert" logs only appeared once in one of our environment. For all of our other environments, we observed the same SSL VERIFY / CERT EXPIRED error but the node agent logs are normal (example shown below):
June 17th 2019, 02:05:59.124	2019-06-16T18:05:58.268516Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.38~my-application-748c7dcb77-6tgls.mynamespace~services.svc.cluster.local-35"
June 17th 2019, 02:05:58.128	2019-06-16T18:05:57.732654Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.71~my-application-two-7f857ffbd4-pgqtd.mynamespace~services.svc.cluster.local-23"
June 17th 2019, 02:05:58.127	2019-06-16T18:05:57.626791Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.13~my-application-three-5b87bc9565-899wh.mynamespace~services.svc.cluster.local-26"
June 17th 2019, 02:05:58.124	2019-06-16T18:05:57.503437Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.5~my-application-three-5b87bc9565-ldnrw.mynamespace~services.svc.cluster.local-33"
June 17th 2019, 02:05:57.135	2019-06-16T18:05:56.698009Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.71~my-application-two-7f857ffbd4-pgqtd.mynamespace~services.svc.cluster.local-25"
June 17th 2019, 02:05:57.132	2019-06-16T18:05:56.421798Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.38~my-application-748c7dcb77-6tgls.mynamespace~services.svc.cluster.local-35"
June 17th 2019, 02:05:57.128	2019-06-16T18:05:56.282202Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.27~my-application-748c7dcb77-j85g2.mynamespace~services.svc.cluster.local-19"
June 17th 2019, 02:05:57.125	2019-06-16T18:05:56.261387Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.5~my-application-three-5b87bc9565-ldnrw.mynamespace~services.svc.cluster.local-33"
June 17th 2019, 02:05:56.134	2019-06-16T18:05:55.996587Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.5~my-application-three-5b87bc9565-ldnrw.mynamespace~services.svc.cluster.local-32"
June 17th 2019, 02:05:56.133	2019-06-16T18:05:55.946790Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.13~my-application-three-5b87bc9565-899wh.mynamespace~services.svc.cluster.local-26"
June 17th 2019, 02:05:56.132	2019-06-16T18:05:55.903269Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.71~my-application-two-7f857ffbd4-pgqtd.mynamespace~services.svc.cluster.local-23"
June 17th 2019, 02:05:56.127	2019-06-16T18:05:55.773862Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.38~my-application-748c7dcb77-6tgls.mynamespace~services.svc.cluster.local-36"
June 17th 2019, 02:05:55.132	2019-06-16T18:05:54.654639Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.27~my-application-748c7dcb77-j85g2.mynamespace~services.svc.cluster.local-19"
June 17th 2019, 02:05:55.129	2019-06-16T18:05:54.326042Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.5~my-application-three-5b87bc9565-ldnrw.mynamespace~services.svc.cluster.local-32"
June 17th 2019, 02:05:55.126	2019-06-16T18:05:54.256648Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.38~my-application-748c7dcb77-6tgls.mynamespace~services.svc.cluster.local-36"
June 17th 2019, 01:55:55.125	2019-06-16T17:55:54.341694Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.27~my-application-748c7dcb77-j85g2.mynamespace~services.svc.cluster.local-21"
June 17th 2019, 01:55:55.124	2019-06-16T17:55:54.162717Z	info	SDS: push key/cert pair from node agent to proxy: "router~172.23.15.80~custom-ingressgateway-d47657b85-5w29p.istio-system~istio-system.svc.cluster.local-12"
June 17th 2019, 01:55:54.131	2019-06-16T17:55:53.957186Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.13~my-application-three-5b87bc9565-899wh.mynamespace~services.svc.cluster.local-28"
June 17th 2019, 01:55:54.129	2019-06-16T17:55:53.836438Z	info	SDS: push key/cert pair from node agent to proxy: "router~172.23.15.80~custom-ingressgateway-d47657b85-5w29p.istio-system~istio-system.svc.cluster.local-12"
June 17th 2019, 01:55:54.125	2019-06-16T17:55:53.802276Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.27~my-application-748c7dcb77-j85g2.mynamespace~services.svc.cluster.local-21"
June 17th 2019, 01:45:55.126	2019-06-16T17:45:54.575620Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.120~sleep-fdc95b5cc-k92l5.istio-system~istio-system.svc.cluster.local-17"
June 17th 2019, 01:45:55.125	2019-06-16T17:45:54.385893Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.120~sleep-fdc95b5cc-k92l5.istio-system~istio-system.svc.cluster.local-15"
June 17th 2019, 01:45:55.124	2019-06-16T17:45:54.340313Z	info	SDS: push key/cert pair from node agent to proxy: "router~172.23.15.80~custom-ingressgateway-d47657b85-5w29p.istio-system~istio-system.svc.cluster.local-13"
June 17th 2019, 01:45:54.180	2019-06-16T17:45:54.096419Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.120~sleep-fdc95b5cc-k92l5.istio-system~istio-system.svc.cluster.local-17"
June 17th 2019, 01:45:54.177	2019-06-16T17:45:53.846360Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.15.120~sleep-fdc95b5cc-k92l5.istio-system~istio-system.svc.cluster.local-15"

Occasionally, we also see some abnormality in the node-agent logs which I thought may be helpful:
June 13th 2019, 19:43:59.388	2019-06-13T11:43:59.352708Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.388	2019-06-13T11:43:59.352630Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.387	2019-06-13T11:43:59.352552Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-22" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:43:59.386	2019-06-13T11:43:59.352309Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.386	2019-06-13T11:43:59.352436Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-23" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:43:59.385	2019-06-13T11:43:59.352221Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-24" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:43:59.384	2019-06-13T11:43:59.352075Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.379	2019-06-13T11:43:59.351888Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-21" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.862	2019-06-13T11:30:18.940690Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-1" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.862	2019-06-13T11:30:18.940703Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.861	2019-06-13T11:30:18.940624Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.861	2019-06-13T11:30:18.940653Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-2" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.861	2019-06-13T11:30:18.940666Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.860	2019-06-13T11:30:18.940609Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-5" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.860	2019-06-13T11:30:18.940573Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.857	2019-06-13T11:30:18.940531Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-3" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:54.236	2019-06-13T11:29:53.312263Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-12"
June 13th 2019, 19:29:53.254	2019-06-13T11:29:52.808226Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-10"
June 13th 2019, 19:29:53.252	2019-06-13T11:29:52.553131Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-11"
June 13th 2019, 19:29:53.231	2019-06-13T11:29:52.548176Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-9"
June 13th 2019, 19:29:49.229	2019-06-13T11:29:48.073184Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:49.228	2019-06-13T11:29:48.073274Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:49.228	2019-06-13T11:29:48.073287Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:49.227	2019-06-13T11:29:48.073171Z	info	SDS: connection with "sidecar~172.23.15.33~my-application-7ddc445699-tpwb2.mynamespace~services.svc.cluster.local-9" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:49.227	2019-06-13T11:29:48.073264Z	info	SDS: connection with "sidecar~172.23.15.33~my-application-7ddc445699-tpwb2.mynamespace~services.svc.cluster.local-7" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:49.226	2019-06-13T11:29:48.072951Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:49.226	2019-06-13T11:29:48.073070Z	info	SDS: connection with "sidecar~172.23.15.33~my-application-7ddc445699-tpwb2.mynamespace~services.svc.cluster.local-6" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:49.225	2019-06-13T11:29:48.072738Z	info	SDS: connection with "sidecar~172.23.15.33~my-application-7ddc445699-tpwb2.mynamespace~services.svc.cluster.local-8" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:48.229	2019-06-13T11:29:47.412973Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:48.229	2019-06-13T11:29:47.412982Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:48.228	2019-06-13T11:29:47.412923Z	info	SDS: connection with "sidecar~172.23.14.82~my-application-7ddc445699-hxvnr.mynamespace~services.svc.cluster.local-5" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:48.228	2019-06-13T11:29:47.412960Z	info	SDS: connection with "sidecar~172.23.14.82~my-application-7ddc445699-hxvnr.mynamespace~services.svc.cluster.local-8" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:48.227	2019-06-13T11:29:47.412879Z	info	SDS: connection with "sidecar~172.23.14.82~my-application-7ddc445699-hxvnr.mynamespace~services.svc.cluster.local-6" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:48.227	2019-06-13T11:29:47.412896Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:48.226	2019-06-13T11:29:47.412805Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:48.225	2019-06-13T11:29:47.412722Z	info	SDS: connection with "sidecar~172.23.14.82~my-application-7ddc445699-hxvnr.mynamespace~services.svc.cluster.local-7" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:47.570	2019-06-13T11:29:47.042784Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.163~my-application-99988594c-kqphc.mynamespace~services.svc.cluster.local-12"
June 13th 2019, 19:29:47.569	2019-06-13T11:29:47.019292Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.163~my-application-99988594c-kqphc.mynamespace~services.svc.cluster.local-10"
June 13th 2019, 19:29:46.679	2019-06-13T11:29:45.853107Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.163~my-application-99988594c-kqphc.mynamespace~services.svc.cluster.local-11"
June 13th 2019, 19:29:46.675	2019-06-13T11:29:45.852465Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.163~my-application-99988594c-kqphc.mynamespace~services.svc.cluster.local-9"
June 13th 2019, 19:29:45.379	2019-06-13T11:29:45.319079Z	info	SDS: push key/cert pair from node agent to proxy: "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-23"
June 13th 2019, 19:29:45.378	2019-06-13T11:29:44.918636Z	info	SDS: push key/cert pair from node agent to proxy: "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-21"
June 13th 2019, 19:29:44.551	2019-06-13T11:29:44.314395Z	info	SDS: push root cert from node agent to proxy connection: "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-24"
June 13th 2019, 19:29:44.550	2019-06-13T11:29:44.314017Z	info	SDS: push root cert from node agent to proxy connection: "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-22"
June 13th 2019, 19:29:44.548	2019-06-13T11:29:43.721509Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:44.547	2019-06-13T11:29:43.721498Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-19" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:44.546	2019-06-13T11:29:43.721449Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:44.546	2019-06-13T11:29:43.721463Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:44.525	2019-06-13T11:29:43.721439Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-17" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:44.524	2019-06-13T11:29:43.721258Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:44.522	2019-06-13T11:29:43.721155Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-18" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:44.520	2019-06-13T11:29:43.721041Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-20" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:09.578	2019-06-13T11:29:09.128205Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:09.577	2019-06-13T11:29:09.128195Z	info	SDS: connection with "sidecar~172.23.14.234~my-application-7ddc445699-kf2vl.mynamespace~services.svc.cluster.local-7" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:09.577	2019-06-13T11:29:09.128197Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:09.576	2019-06-13T11:29:09.128167Z	info	SDS: connection with "sidecar~172.23.14.234~my-application-7ddc445699-kf2vl.mynamespace~services.svc.cluster.local-8" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:09.576	2019-06-13T11:29:09.128180Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:09.574	2019-06-13T11:29:09.128159Z	info	SDS: connection with "sidecar~172.23.14.234~my-application-7ddc445699-kf2vl.mynamespace~services.svc.cluster.local-5" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:09.573	2019-06-13T11:29:09.128123Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:29:09.571	2019-06-13T11:29:09.128076Z	info	SDS: connection with "sidecar~172.23.14.234~my-application-7ddc445699-kf2vl.mynamespace~services.svc.cluster.local-6" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:09.357	2019-06-13T11:29:08.609012Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.122~my-application-99988594c-h8xdq.mynamespace~services.svc.cluster.local-6"
June 13th 2019, 19:29:09.207	2019-06-13T11:29:08.240074Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.148~my-application-99988594c-dtlf8.mynamespace~services.svc.cluster.local-25"
June 13th 2019, 19:29:08.363	2019-06-13T11:29:08.209073Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.122~my-application-99988594c-h8xdq.mynamespace~services.svc.cluster.local-8"
June 13th 2019, 19:29:08.362	2019-06-13T11:29:07.554953Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.122~my-application-99988594c-h8xdq.mynamespace~services.svc.cluster.local-7"
June 13th 2019, 19:29:08.358	2019-06-13T11:29:07.554400Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.122~my-application-99988594c-h8xdq.mynamespace~services.svc.cluster.local-5"
June 13th 2019, 19:29:08.210	2019-06-13T11:29:08.127220Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.148~my-application-99988594c-dtlf8.mynamespace~services.svc.cluster.local-27"
June 13th 2019, 19:29:08.209	2019-06-13T11:29:07.675068Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.148~my-application-99988594c-dtlf8.mynamespace~services.svc.cluster.local-26

Thanks for the update. These logs look okay to me. @quanjielin could you help to verify the log?
The error logs in the proxy showed that sidecar proxy uses a expired server cert to do handshake with ingress gateway, which causes handshake failure.

Hi @JimmyCYJ, none of our clusters are experiencing the problem for now, but here's a snippet of the output for the /cert endpoint:
{
  "ca_cert": [
    {
      "path": "\u003cinline\u003e",
      "serial_number": "dc7fdb170faaa58afb8452f4e9fae279",
      "subject_alt_names": [],
      "days_until_expiration": "313",
      "valid_from": "2019-04-29T10:18:29Z",
      "expiration_time": "2020-04-28T10:18:29Z"
    }
  ],
  "cert_chain": [
    {
      "path": "\u003cinline\u003e",
      "serial_number": "d60032e6652589762b206fda055c9047",
      "subject_alt_names": [
      {
        "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application-name"
      }
      ],
      "days_until_expiration": "0",
      "valid_from": "2019-06-19T09:58:34Z",
      "expiration_time": "2019-06-20T09:58:34Z"
    }
  ]
}

As for the node agent logs, I do not have anything new to add other than the ones that I showed in the previous comment. We consistently see these in the node agent logs when we experience the error:
June 13th 2019, 19:43:59.388	2019-06-13T11:43:59.352708Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.388	2019-06-13T11:43:59.352630Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.387	2019-06-13T11:43:59.352552Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-22" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:43:59.386	2019-06-13T11:43:59.352309Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.386	2019-06-13T11:43:59.352436Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-23" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:43:59.385	2019-06-13T11:43:59.352221Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-24" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:43:59.384	2019-06-13T11:43:59.352075Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:43:59.379	2019-06-13T11:43:59.351888Z	info	SDS: connection with "router~172.23.15.113~istio-ingressgateway-7b7b8475c-vrmv4.istio-system~istio-system.svc.cluster.local-21" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.862	2019-06-13T11:30:18.940690Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-1" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.862	2019-06-13T11:30:18.940703Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.861	2019-06-13T11:30:18.940624Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.861	2019-06-13T11:30:18.940653Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-2" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.861	2019-06-13T11:30:18.940666Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.860	2019-06-13T11:30:18.940609Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-5" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:30:19.860	2019-06-13T11:30:18.940573Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
June 13th 2019, 19:30:19.857	2019-06-13T11:30:18.940531Z	info	SDS: connection with "sidecar~172.23.15.6~my-application-7ddc445699-t56wr.mynamespace~services.svc.cluster.local-3" terminated rpc error: code = Canceled desc = context canceled
June 13th 2019, 19:29:54.236	2019-06-13T11:29:53.312263Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-12"
June 13th 2019, 19:29:53.254	2019-06-13T11:29:52.808226Z	info	SDS: push key/cert pair from node agent to proxy: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-10"
June 13th 2019, 19:29:53.252	2019-06-13T11:29:52.553131Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-11"
June 13th 2019, 19:29:53.231	2019-06-13T11:29:52.548176Z	info	SDS: push root cert from node agent to proxy connection: "sidecar~172.23.14.32~my-application-99988594c-6jmh9.mynamespace~services.svc.cluster.local-9"

Thanks for your help!

These are info logs. It is because Citadel agent (we are replacing the name node agent) terminates connection during cert rotation, to force Envoy reconnect with its latest token. Given that the issue is gone and we don't have other logs or information for it, shall we close this issue for now and reopen it if this happen again?
@quanjielin

Hi @JimmyCYJ, I reopened it because the error will eventually occur again in a few days' time. I am already noticing some abnormalities in the /certs endpoint for one of the applications in my production environment:
{
 "certificates": [
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "d4cad65b21b2b4752ac193ddcee62c9a",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "18446744073709551613",
     "valid_from": "2019-06-17T02:27:34Z",
     "expiration_time": "2019-06-18T02:27:34Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "d4cad65b21b2b4752ac193ddcee62c9a",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "18446744073709551613",
     "valid_from": "2019-06-17T02:27:34Z",
     "expiration_time": "2019-06-18T02:27:34Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a8906d7d044560cdfac60645fb201054",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-19T23:53:30Z",
     "expiration_time": "2019-06-20T23:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "58e2e4663f778ee3bf3379df28bb6126",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T22:53:30Z",
     "expiration_time": "2019-06-21T22:53:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "5e01882fc8ef46479ab6415e10e0c84b",
     "subject_alt_names": [],
     "days_until_expiration": "329",
     "valid_from": "2019-05-16T05:32:57Z",
     "expiration_time": "2020-05-15T05:32:57Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "521dee11b3df53ac3d28da1f56349df0",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/my-namespace/sa/my-application"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2019-06-20T23:03:29Z",
     "expiration_time": "2019-06-21T23:03:29Z"
    }
   ]
  }
 ]
}

Some of the cert-chain certificates have already expired and are not renewed

I noticed that your deployment has istio-ingressgateway and custom-ingressgateway, could you briefly introduce the structure of deployment?
There are two certs with expiration date 2019-06-18T02:27:34Z, sometimes this could happen when the old cert is still in use by existing long live connections (e.g. HTTP2 and gRPC). They will be deleted when the connections are closed. I think this /cert is from a workload sidecar proxy. You could check the config dump of the sidecar, count the number of Listener and Cluster, and compare with the number of "cert_chain" to figure out if "cert_chain" lists more items than the number of Listener+Cluster.
Could you also use $kubectl get pods -n istio-system -o wide, to show us whether the node agent is restarted. There was a UDS mount fix (https://github.com/istio/istio/issues/14853) that would cause 503 if node agent is restarted. @quanjielin
Is your deployment running 1.1.8 release? Could you show us the helm template?

We deploy 2 separate load balancers for our cluster because we want to segregate traffic coming from the public and traffic coming from another private network.
I retrieved the config_dump from the sidecar and tried to count the number of Listeners and Clusters: From the latest output of /certs, I only have 36 counts of cert-chain entries, but I have 39 counts of listeners alone, and I'm unsure how to identify the number of clusters but I count 56 dynamic_active_clusters.
We manually restarted two of the node agents around the same time as when we restarted our services:

 NAME                                     READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE
custom-ingressgateway-d47657b85-g25gq    2/2     Running   0          10d   172.23.22.229   ip-172-23-22-179.ap-southeast-1.compute.internal   <none>
istio-citadel-7dbf78bf8f-sxpmp           1/1     Running   0          10d   172.23.22.134   ip-172-23-22-179.ap-southeast-1.compute.internal   <none>
istio-egressgateway-847ff88587-swjq9     1/1     Running   0          10d   172.23.22.242   ip-172-23-22-179.ap-southeast-1.compute.internal   <none>
istio-galley-7f874545bd-c6cb4            1/1     Running   0          10d   172.23.24.189   ip-172-23-24-174.ap-southeast-1.compute.internal   <none>
istio-ingressgateway-66bdfc7d8b-tsw4z    2/2     Running   0          8d    172.23.22.148   ip-172-23-22-179.ap-southeast-1.compute.internal   <none>
istio-nodeagent-54l6p                    1/1     Running   0          8d    172.23.23.17    ip-172-23-23-24.ap-southeast-1.compute.internal    <none>
istio-nodeagent-7gjkk                    1/1     Running   0          8d    172.23.22.89    ip-172-23-22-71.ap-southeast-1.compute.internal    <none>
istio-nodeagent-7h6n9                    1/1     Running   0          10d   172.23.22.240   ip-172-23-22-179.ap-southeast-1.compute.internal   <none>
istio-nodeagent-bg6pb                    1/1     Running   0          10d   172.23.24.180   ip-172-23-24-174.ap-southeast-1.compute.internal   <none>
istio-nodeagent-l42q8                    1/1     Running   0          10d   172.23.23.214   ip-172-23-23-163.ap-southeast-1.compute.internal   <none>
istio-pilot-b96cc4756-5zl2n              2/2     Running   0          10d   172.23.23.146   ip-172-23-23-163.ap-southeast-1.compute.internal   <none>
istio-policy-599d54d57d-pv6q4            2/2     Running   1          10d   172.23.22.212   ip-172-23-22-179.ap-southeast-1.compute.internal   <none>
istio-sidecar-injector-5c77d99d8-7c7st   1/1     Running   0          10d   172.23.22.197   ip-172-23-22-179.ap-southeast-1.compute.internal   <none>
istio-telemetry-bd7f7cd4c-flcgd          2/2     Running   1          10d   172.23.24.248   ip-172-23-24-174.ap-southeast-1.compute.internal   <none>

Yup, we are running 1.1.8 release.
Here you go:

---
# Source: istio/charts/galley/templates/poddisruptionbudget.yaml

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: istio-galley
  namespace: istio-system
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    release: istio
    istio: galley
spec:

  minAvailable: 1
  selector:
    matchLabels:
      app: galley
      release: istio
      istio: galley

---
# Source: istio/charts/gateways/templates/poddisruptionbudget.yaml

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: custom-ingressgateway
  namespace: istio-system
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: custom-ingressgateway
    istio: custom-ingressgateway
spec:

  minAvailable: 1
  selector:
    matchLabels:
      release: istio
      app: custom-ingressgateway
      istio: custom-ingressgateway
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: istio-egressgateway
  namespace: istio-system
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: istio-egressgateway
    istio: egressgateway
spec:

  minAvailable: 1
  selector:
    matchLabels:
      release: istio
      app: istio-egressgateway
      istio: egressgateway
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: istio-ingressgateway
    istio: ingressgateway
spec:

  minAvailable: 1
  selector:
    matchLabels:
      release: istio
      app: istio-ingressgateway
      istio: ingressgateway
---

---
# Source: istio/charts/mixer/templates/poddisruptionbudget.yaml

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: istio-policy
  namespace: istio-system
  labels:
    app: policy
    chart: mixer
    heritage: Tiller
    release: istio
    version: 1.1.8
    istio: mixer
    istio-mixer-type: policy
spec:

  minAvailable: 1
  selector:
    matchLabels:
      app: policy
      release: istio
      istio: mixer
      istio-mixer-type: policy
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: istio-telemetry
  namespace: istio-system
  labels:
    app: telemetry
    chart: mixer
    heritage: Tiller
    release: istio
    version: 1.1.8
    istio: mixer
    istio-mixer-type: telemetry
spec:

  minAvailable: 1
  selector:
    matchLabels:
      app: telemetry
      release: istio
      istio: mixer
      istio-mixer-type: telemetry
---

---
# Source: istio/charts/pilot/templates/poddisruptionbudget.yaml

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: istio-pilot
  namespace: istio-system
  labels:
    app: pilot
    chart: pilot
    heritage: Tiller
    release: istio
    istio: pilot
spec:

  minAvailable: 1
  selector:
    matchLabels:
      app: pilot
      release: istio
      istio: pilot

---
# Source: istio/charts/galley/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-galley-configuration
  namespace: istio-system
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    release: istio
    istio: galley
data:
  validatingwebhookconfiguration.yaml: |-
    apiVersion: admissionregistration.k8s.io/v1beta1
    kind: ValidatingWebhookConfiguration
    metadata:
      name: istio-galley
      namespace: istio-system
      labels:
        app: galley
        chart: galley
        heritage: Tiller
        release: istio
        istio: galley
    webhooks:
      - name: pilot.validation.istio.io
        clientConfig:
          service:
            name: istio-galley
            namespace: istio-system
            path: "/admitpilot"
          caBundle: ""
        rules:
          - operations:
            - CREATE
            - UPDATE
            apiGroups:
            - config.istio.io
            apiVersions:
            - v1alpha2
            resources:
            - httpapispecs
            - httpapispecbindings
            - quotaspecs
            - quotaspecbindings
          - operations:
            - CREATE
            - UPDATE
            apiGroups:
            - rbac.istio.io
            apiVersions:
            - "*"
            resources:
            - "*"
          - operations:
            - CREATE
            - UPDATE
            apiGroups:
            - authentication.istio.io
            apiVersions:
            - "*"
            resources:
            - "*"
          - operations:
            - CREATE
            - UPDATE
            apiGroups:
            - networking.istio.io
            apiVersions:
            - "*"
            resources:
            - destinationrules
            - envoyfilters
            - gateways
            - serviceentries
            - sidecars
            - virtualservices
        failurePolicy: Fail
      - name: mixer.validation.istio.io
        clientConfig:
          service:
            name: istio-galley
            namespace: istio-system
            path: "/admitmixer"
          caBundle: ""
        rules:
          - operations:
            - CREATE
            - UPDATE
            apiGroups:
            - config.istio.io
            apiVersions:
            - v1alpha2
            resources:
            - rules
            - attributemanifests
            - circonuses
            - deniers
            - fluentds
            - kubernetesenvs
            - listcheckers
            - memquotas
            - noops
            - opas
            - prometheuses
            - rbacs
            - solarwindses
            - stackdrivers
            - cloudwatches
            - dogstatsds
            - statsds
            - stdios
            - apikeys
            - authorizations
            - checknothings
            # - kuberneteses
            - listentries
            - logentries
            - metrics
            - quotas
            - reportnothings
            - tracespans
        failurePolicy: Fail
---
# Source: istio/charts/security/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-security-custom-resources
  namespace: istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
    istio: citadel
data:
  custom-resources.yaml: |-
    # Authentication policy to enable permissive mode for all services (that have sidecar) in the mesh.
    apiVersion: "authentication.istio.io/v1alpha1"
    kind: "MeshPolicy"
    metadata:
      name: "default"
      labels:
        app: security
        chart: security
        heritage: Tiller
        release: istio
    spec:
      peers:
      - mtls:
          mode: PERMISSIVE
  run.sh: |-
    #!/bin/sh

    set -x

    if [ "$#" -ne "1" ]; then
        echo "first argument should be path to custom resource yaml"
        exit 1
    fi

    pathToResourceYAML=${1}

    kubectl get validatingwebhookconfiguration istio-galley 2>/dev/null
    if [ "$?" -eq 0 ]; then
        echo "istio-galley validatingwebhookconfiguration found - waiting for istio-galley deployment to be ready"
        while true; do
            kubectl -n istio-system get deployment istio-galley 2>/dev/null
            if [ "$?" -eq 0 ]; then
                break
            fi
            sleep 1
        done
        kubectl -n istio-system rollout status deployment istio-galley
        if [ "$?" -ne 0 ]; then
            echo "istio-galley deployment rollout status check failed"
            exit 1
        fi
        echo "istio-galley deployment ready for configuration validation"
    fi
    sleep 5
    kubectl apply -f ${pathToResourceYAML}

---
# Source: istio/templates/configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
  namespace: istio-system
  labels:
    app: istio
    chart: istio
    heritage: Tiller
    release: istio
data:
  mesh: |-
    # Set the following variable to true to disable policy checks by the Mixer.
    # Note that metrics will still be reported to the Mixer.
    disablePolicyChecks: true

    # Set enableTracing to false to disable request tracing.
    enableTracing: true

    # Set accessLogFile to empty string to disable access log.
    accessLogFile: "/dev/stdout"

    # If accessLogEncoding is TEXT, value will be used directly as the log format
    # example: "[%START_TIME%] %REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\n"
    # If AccessLogEncoding is JSON, value will be parsed as map[string]string
    # example: '{"start_time": "%START_TIME%", "req_method": "%REQ(:METHOD)%"}'
    # Leave empty to use default log format
    accessLogFormat: ""

    # Set accessLogEncoding to JSON or TEXT to configure sidecar access log
    accessLogEncoding: 'JSON'
    mixerCheckServer: istio-policy.istio-system.svc.cluster.local:9091
    mixerReportServer: istio-telemetry.istio-system.svc.cluster.local:9091
    # policyCheckFailOpen allows traffic in cases when the mixer policy service cannot be reached.
    # Default is false which means the traffic is denied when the client is unable to connect to Mixer.
    policyCheckFailOpen: false
    # Let Pilot give ingresses the public IP of the Istio ingressgateway
    ingressService: istio-ingressgateway

    # Default connect timeout for dynamic clusters generated by Pilot and returned via XDS
    connectTimeout: 10s

    # DNS refresh rate for Envoy clusters of type STRICT_DNS
    dnsRefreshRate: 5s

    # Unix Domain Socket through which envoy communicates with NodeAgent SDS to get
    # key/cert for mTLS. Use secret-mount files instead of SDS if set to empty.
    sdsUdsPath: unix:/var/run/sds/uds_path

    # This flag is used by secret discovery service(SDS).
    # If set to true(prerequisite: https://kubernetes.io/docs/concepts/storage/volumes/#projected), Istio will inject volumes mount
    # for k8s service account JWT, so that K8s API server mounts k8s service account JWT to envoy container, which
    # will be used to generate key/cert eventually. This isn't supported for non-k8s case.
    enableSdsTokenMount: false

    # This flag is used by secret discovery service(SDS).
    # If set to true, envoy will fetch normal k8s service account JWT from '/var/run/secrets/kubernetes.io/serviceaccount/token'
    # (https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod)
    # and pass to sds server, which will be used to request key/cert eventually.
    # this flag is ignored if enableSdsTokenMount is set.
    # This isn't supported for non-k8s case.
    sdsUseK8sSaJwt: true

    # The trust domain corresponds to the trust root of a system.
    # Refer to https://github.com/spiffe/spiffe/blob/master/standards/SPIFFE-ID.md#21-trust-domain
    trustDomain:

    # Set the default behavior of the sidecar for handling outbound traffic from the application:
    # ALLOW_ANY - outbound traffic to unknown destinations will be allowed, in case there are no
    #   services or ServiceEntries for the destination port
    # REGISTRY_ONLY - restrict outbound traffic to services defined in the service registry as well
    #   as those defined through ServiceEntries
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY

    localityLbSetting:
      {}

    # The namespace to treat as the administrative root namespace for istio
    # configuration.
    rootNamespace: istio-system
    configSources:
    - address: istio-galley.istio-system.svc:9901

    defaultConfig:
      #
      # TCP connection timeout between Envoy & the application, and between Envoys.  Used for static clusters
      # defined in Envoy's configuration file
      connectTimeout: 10s
      #
      ### ADVANCED SETTINGS #############
      # Where should envoy's configuration be stored in the istio-proxy container
      configPath: "/etc/istio/proxy"
      binaryPath: "/usr/local/bin/envoy"
      # The pseudo service name used for Envoy.
      serviceCluster: istio-proxy
      # These settings that determine how long an old Envoy
      # process should be kept alive after an occasional reload.
      drainDuration: 45s
      parentShutdownDuration: 1m0s
      #
      # The mode used to redirect inbound connections to Envoy. This setting
      # has no effect on outbound traffic: iptables REDIRECT is always used for
      # outbound connections.
      # If "REDIRECT", use iptables REDIRECT to NAT and redirect to Envoy.
      # The "REDIRECT" mode loses source addresses during redirection.
      # If "TPROXY", use iptables TPROXY to redirect to Envoy.
      # The "TPROXY" mode preserves both the source and destination IP
      # addresses and ports, so that they can be used for advanced filtering
      # and manipulation.
      # The "TPROXY" mode also configures the sidecar to run with the
      # CAP_NET_ADMIN capability, which is required to use TPROXY.
      #interceptionMode: REDIRECT
      #
      # Port where Envoy listens (on local host) for admin commands
      # You can exec into the istio-proxy container in a pod and
      # curl the admin port (curl http://localhost:15000/) to obtain
      # diagnostic information from Envoy. See
      # https://lyft.github.io/envoy/docs/operations/admin.html
      # for more details
      proxyAdminPort: 15000
      #
      # Set concurrency to a specific number to control the number of Proxy worker threads.
      # If set to 0 (default), then start worker thread for each CPU thread/core.
      concurrency: 2
      #
      tracing:
        zipkin:
          # Address of the Zipkin collector
          address: zipkin.istio-system:9411
      #
      # Mutual TLS authentication between sidecars and istio control plane.
      controlPlaneAuthPolicy: NONE
      #
      # Address where istio Pilot service is running
      discoveryAddress: istio-pilot.istio-system:15010

  # Configuration file for the mesh networks to be used by the Split Horizon EDS.
  meshNetworks: |-
    networks: {}

---
# Source: istio/templates/sidecar-injector-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-sidecar-injector
  namespace: istio-system
  labels:
    app: istio
    chart: istio
    heritage: Tiller
    release: istio
    istio: sidecar-injector
data:
  config: |-
    policy: enabled
    template: |-
      rewriteAppHTTPProbe: true
      initContainers:
      [[ if ne (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) "NONE" ]]
      - name: istio-init
        image: "docker.io/istio/proxy_init:1.1.8"
        args:
        - "-p"
        - [[ .MeshConfig.ProxyListenPort ]]
        - "-u"
        - 1337
        - "-m"
        - [[ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode ]]
        - "-i"
        - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges`  "*"  ]]"
        - "-x"
        - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges`  ""  ]]"
        - "-b"
        - "[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts` (includeInboundPorts .Spec.Containers) ]]"
        - "-d"
        - "[[ excludeInboundPort (annotation .ObjectMeta `status.sidecar.istio.io/port`  15020 ) (annotation .ObjectMeta `traffic.sidecar.istio.io/excludeInboundPorts`  "" ) ]]"
        [[ if (isset .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces`) -]]
        - "-k"
        - "[[ index .ObjectMeta.Annotations `traffic.sidecar.istio.io/kubevirtInterfaces` ]]"
        [[ end -]]
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: 10m
            memory: 10Mi
          limits:
            cpu: 100m
            memory: 50Mi
        securityContext:
          runAsUser: 0
          runAsNonRoot: false
          capabilities:
            add:
            - NET_ADMIN
        restartPolicy: Always
      [[ end -]]
      containers:
      - name: istio-proxy
        image: [[ annotation .ObjectMeta `sidecar.istio.io/proxyImage`  "docker.io/istio/proxyv2:1.1.8"  ]]
        ports:
        - containerPort: 15090
          protocol: TCP
          name: http-envoy-prom
        args:
        - proxy
        - sidecar
        - --domain
        - $(POD_NAMESPACE).svc.cluster.local
        - --configPath
        - [[ .ProxyConfig.ConfigPath ]]
        - --binaryPath
        - [[ .ProxyConfig.BinaryPath ]]
        - --serviceCluster
        [[ if ne "" (index .ObjectMeta.Labels "app") -]]
        - [[ index .ObjectMeta.Labels "app" ]].$(POD_NAMESPACE)
        [[ else -]]
        - [[ valueOrDefault .DeploymentMeta.Name "istio-proxy" ]].[[ valueOrDefault .DeploymentMeta.Namespace "default" ]]
        [[ end -]]
        - --drainDuration
        - [[ formatDuration .ProxyConfig.DrainDuration ]]
        - --parentShutdownDuration
        - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]
        - --discoveryAddress
        - [[ annotation .ObjectMeta `sidecar.istio.io/discoveryAddress` .ProxyConfig.DiscoveryAddress ]]
        - --zipkinAddress
        - [[ .ProxyConfig.GetTracing.GetZipkin.GetAddress ]]
        - --connectTimeout
        - [[ formatDuration .ProxyConfig.ConnectTimeout ]]
        - --proxyAdminPort
        - [[ .ProxyConfig.ProxyAdminPort ]]
        [[ if gt .ProxyConfig.Concurrency 0 -]]
        - --concurrency
        - [[ .ProxyConfig.Concurrency ]]
        [[ end -]]
        - --controlPlaneAuthPolicy
        - [[ annotation .ObjectMeta `sidecar.istio.io/controlPlaneAuthPolicy` .ProxyConfig.ControlPlaneAuthPolicy ]]
      [[- if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port`  15020 ) "0") ]]
        - --statusPort
        - [[ annotation .ObjectMeta `status.sidecar.istio.io/port`  15020  ]]
        - --applicationPorts
        - "[[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/applicationPorts` (applicationPorts .Spec.Containers) ]]"
      [[- end ]]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

        - name: ISTIO_META_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ISTIO_META_CONFIG_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: ISTIO_META_INTERCEPTION_MODE
          value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
        [[ if .ObjectMeta.Annotations ]]
        - name: ISTIO_METAJSON_ANNOTATIONS
          value: |
                 [[ toJSON .ObjectMeta.Annotations ]]
        [[ end ]]
        [[ if .ObjectMeta.Labels ]]
        - name: ISTIO_METAJSON_LABELS
          value: |
                 [[ toJSON .ObjectMeta.Labels ]]
        [[ end ]]
        [[- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) ]]
        - name: ISTIO_BOOTSTRAP_OVERRIDE
          value: "/etc/istio/custom-bootstrap/custom_bootstrap.json"
        [[- end ]]
        imagePullPolicy: IfNotPresent
        [[ if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port`  15020 ) "0") ]]
        readinessProbe:
          httpGet:
            path: /healthz/ready
            port: [[ annotation .ObjectMeta `status.sidecar.istio.io/port`  15020  ]]
          initialDelaySeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds`  1  ]]
          periodSeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds`  2  ]]
          failureThreshold: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold`  30  ]]
        [[ end -]]securityContext:
          readOnlyRootFilesystem: true
          [[ if eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode) "TPROXY" -]]
          capabilities:
            add:
            - NET_ADMIN
          runAsGroup: 1337
          [[ else -]]

          runAsUser: 1337
          [[- end ]]
        resources:
          [[ if or (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -]]
          requests:
            [[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -]]
            cpu: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` ]]"
            [[ end ]]
            [[ if (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`) -]]
            memory: "[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory` ]]"
            [[ end ]]
        [[ else -]]
          limits:
            cpu: 300m
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 50Mi

        [[ end -]]
        volumeMounts:
        [[- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) ]]
        - mountPath: /etc/istio/custom-bootstrap
          name: custom-bootstrap-volume
        [[- end ]]
        - mountPath: /etc/istio/proxy
          name: istio-envoy
        - mountPath: /var/run/sds/uds_path
          name: sds-uds-path
          readOnly: true
          [[- if isset .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount` ]]
          [[ range $index, $value := fromJSON (index .ObjectMeta.Annotations `sidecar.istio.io/userVolumeMount`) ]]
        - name: "[[ $index ]]"
          [[ toYaml $value | indent 4 ]]
          [[ end ]]
          [[- end ]]
      volumes:
      [[- if (isset .ObjectMeta.Annotations `sidecar.istio.io/bootstrapOverride`) ]]
      - name: custom-bootstrap-volume
        configMap:
          name: [[ annotation .ObjectMeta `sidecar.istio.io/bootstrapOverride` `` ]]
      [[- end ]]
      - emptyDir:
          medium: Memory
        name: istio-envoy
      - name: sds-uds-path
        hostPath:
          path: /var/run/sds/uds_path
          type: Socket

---
# Source: istio/charts/galley/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-galley-service-account
  namespace: istio-system
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    release: istio

---
# Source: istio/charts/gateways/templates/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-ingressgateway-service-account
  namespace: istio-system
  labels:
    app: custom-ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-egressgateway-service-account
  namespace: istio-system
  labels:
    app: istio-egressgateway
    chart: gateways
    heritage: Tiller
    release: istio
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-ingressgateway-service-account
  namespace: istio-system
  labels:
    app: istio-ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
---

---
# Source: istio/charts/mixer/templates/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-mixer-service-account
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio

---
# Source: istio/charts/nodeagent/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-nodeagent-service-account
  namespace: istio-system
  labels:
    app: nodeagent
    chart: nodeagent
    heritage: Tiller
    release: istio
---
# Source: istio/charts/pilot/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-pilot-service-account
  namespace: istio-system
  labels:
    app: pilot
    chart: pilot
    heritage: Tiller
    release: istio

---
# Source: istio/charts/security/templates/cleanup-secrets.yaml
# The reason for creating a ServiceAccount and ClusterRole specifically for this
# post-delete hooked job is because the citadel ServiceAccount is being deleted
# before this hook is launched. On the other hand, running this hook before the
# deletion of the citadel (e.g. pre-delete) won't delete the secrets because they
# will be re-created immediately by the to-be-deleted citadel.
#
# It's also important that the ServiceAccount, ClusterRole and ClusterRoleBinding
# will be ready before running the hooked Job therefore the hook weights.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-cleanup-secrets-service-account
  namespace: istio-system
  annotations:
    "helm.sh/hook": post-delete
    "helm.sh/hook-delete-policy": hook-succeeded
    "helm.sh/hook-weight": "1"
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-cleanup-secrets-istio-system
  annotations:
    "helm.sh/hook": post-delete
    "helm.sh/hook-delete-policy": hook-succeeded
    "helm.sh/hook-weight": "1"
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-cleanup-secrets-istio-system
  annotations:
    "helm.sh/hook": post-delete
    "helm.sh/hook-delete-policy": hook-succeeded
    "helm.sh/hook-weight": "2"
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-cleanup-secrets-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-cleanup-secrets-service-account
    namespace: istio-system
---
apiVersion: batch/v1
kind: Job
metadata:
  name: istio-cleanup-secrets-1.1.8
  namespace: istio-system
  annotations:
    "helm.sh/hook": post-delete
    "helm.sh/hook-delete-policy": hook-succeeded
    "helm.sh/hook-weight": "3"
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
spec:
  template:
    metadata:
      name: istio-cleanup-secrets
      labels:
        app: security
        chart: security
        heritage: Tiller
        release: istio
    spec:
      serviceAccountName: istio-cleanup-secrets-service-account
      containers:
        - name: kubectl
          image: "docker.io/istio/kubectl:1.1.8"
          imagePullPolicy: IfNotPresent
          command:
          - /bin/bash
          - -c
          - >
              kubectl get secret --all-namespaces | grep "istio.io/key-and-cert" |  while read -r entry; do
                ns=$(echo $entry | awk '{print $1}');
                name=$(echo $entry | awk '{print $2}');
                kubectl delete secret $name -n $ns;
              done
      restartPolicy: OnFailure
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x

---
# Source: istio/charts/security/templates/create-custom-resources-job.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-security-post-install-account
  namespace: istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: istio-security-post-install-istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
rules:
- apiGroups: ["authentication.istio.io"] # needed to create default authn policy
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["networking.istio.io"] # needed to create security destination rules
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["admissionregistration.k8s.io"]
  resources: ["validatingwebhookconfigurations"]
  verbs: ["get"]
- apiGroups: ["extensions", "apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: istio-security-post-install-role-binding-istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-security-post-install-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-security-post-install-account
    namespace: istio-system
---
apiVersion: batch/v1
kind: Job
metadata:
  name: istio-security-post-install-1.1.8
  namespace: istio-system
  annotations:
    "helm.sh/hook": post-install
    "helm.sh/hook-delete-policy": hook-succeeded
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
spec:
  template:
    metadata:
      name: istio-security-post-install
      labels:
        app: security
        chart: security
        heritage: Tiller
        release: istio
    spec:
      serviceAccountName: istio-security-post-install-account
      containers:
        - name: kubectl
          image: "docker.io/istio/kubectl:1.1.8"
          imagePullPolicy: IfNotPresent
          command: [ "/bin/bash", "/tmp/security/run.sh", "/tmp/security/custom-resources.yaml" ]
          volumeMounts:
            - mountPath: "/tmp/security"
              name: tmp-configmap-security
      volumes:
        - name: tmp-configmap-security
          configMap:
            name: istio-security-custom-resources
      restartPolicy: OnFailure
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x

---
# Source: istio/charts/security/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-citadel-service-account
  namespace: istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio

---
# Source: istio/charts/sidecarInjectorWebhook/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-sidecar-injector-service-account
  namespace: istio-system
  labels:
    app: sidecarInjectorWebhook
    chart: sidecarInjectorWebhook
    heritage: Tiller
    release: istio
    istio: sidecar-injector

---
# Source: istio/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: istio-multi
  namespace: istio-system

---
# Source: istio/charts/galley/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-galley-istio-system
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    release: istio
rules:
- apiGroups: ["admissionregistration.k8s.io"]
  resources: ["validatingwebhookconfigurations"]
  verbs: ["*"]
- apiGroups: ["config.istio.io"] # istio mixer CRD watcher
  resources: ["*"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["networking.istio.io"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["authentication.istio.io"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["rbac.istio.io"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions","apps"]
  resources: ["deployments"]
  resourceNames: ["istio-galley"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods", "nodes", "services", "endpoints"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources: ["deployments/finalizers"]
  resourceNames: ["istio-galley"]
  verbs: ["update"]

---
# Source: istio/charts/gateways/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-ingressgateway-istio-system
  labels:
    app: custom-ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
rules:
- apiGroups: ["networking.istio.io"]
  resources: ["virtualservices", "destinationrules", "gateways"]
  verbs: ["get", "watch", "list", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-egressgateway-istio-system
  labels:
    app: egressgateway
    chart: gateways
    heritage: Tiller
    release: istio
rules:
- apiGroups: ["networking.istio.io"]
  resources: ["virtualservices", "destinationrules", "gateways"]
  verbs: ["get", "watch", "list", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-ingressgateway-istio-system
  labels:
    app: ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
rules:
- apiGroups: ["networking.istio.io"]
  resources: ["virtualservices", "destinationrules", "gateways"]
  verbs: ["get", "watch", "list", "update"]
---

---
# Source: istio/charts/mixer/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-mixer-istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
rules:
- apiGroups: ["config.istio.io"] # istio CRD watcher
  resources: ["*"]
  verbs: ["create", "get", "list", "watch", "patch"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["configmaps", "endpoints", "pods", "services", "namespaces", "secrets", "replicationcontrollers"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions", "apps"]
  resources: ["replicasets"]
  verbs: ["get", "list", "watch"]

---
# Source: istio/charts/nodeagent/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-nodeagent-istio-system
  labels:
    app: nodeagent
    chart: nodeagent
    heritage: Tiller
    release: istio
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]
---
# Source: istio/charts/pilot/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-pilot-istio-system
  labels:
    app: pilot
    chart: pilot
    heritage: Tiller
    release: istio
rules:
- apiGroups: ["config.istio.io"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["rbac.istio.io"]
  resources: ["*"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["networking.istio.io"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["authentication.istio.io"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["*"]
- apiGroups: ["extensions"]
  resources: ["ingresses", "ingresses/status"]
  verbs: ["*"]
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create", "get", "list", "watch", "update"]
- apiGroups: [""]
  resources: ["endpoints", "pods", "services", "namespaces", "nodes", "secrets"]
  verbs: ["get", "list", "watch"]

---
# Source: istio/charts/security/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-citadel-istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create", "get", "update"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create", "get", "watch", "list", "update", "delete"]
- apiGroups: [""]
  resources: ["serviceaccounts", "services"]
  verbs: ["get", "watch", "list"]
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]

---
# Source: istio/charts/sidecarInjectorWebhook/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: istio-sidecar-injector-istio-system
  labels:
    app: sidecarInjectorWebhook
    chart: sidecarInjectorWebhook
    heritage: Tiller
    release: istio
    istio: sidecar-injector
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["admissionregistration.k8s.io"]
  resources: ["mutatingwebhookconfigurations"]
  verbs: ["get", "list", "watch", "patch"]

---
# Source: istio/templates/clusterrole.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: istio-reader
rules:
  - apiGroups: ['']
    resources: ['nodes', 'pods', 'services', 'endpoints', "replicationcontrollers"]
    verbs: ['get', 'watch', 'list']
  - apiGroups: ["extensions", "apps"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]

---
# Source: istio/charts/galley/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-galley-admin-role-binding-istio-system
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-galley-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-galley-service-account
    namespace: istio-system

---
# Source: istio/charts/gateways/templates/clusterrolebindings.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-ingressgateway-istio-system
  labels:
    app: custom-ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-ingressgateway-istio-system
subjects:
- kind: ServiceAccount
  name: custom-ingressgateway-service-account
  namespace: istio-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-egressgateway-istio-system
  labels:
    app: egressgateway
    chart: gateways
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-egressgateway-istio-system
subjects:
- kind: ServiceAccount
  name: istio-egressgateway-service-account
  namespace: istio-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-ingressgateway-istio-system
  labels:
    app: ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-ingressgateway-istio-system
subjects:
- kind: ServiceAccount
  name: istio-ingressgateway-service-account
  namespace: istio-system
---

---
# Source: istio/charts/mixer/templates/clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-mixer-admin-role-binding-istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-mixer-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-mixer-service-account
    namespace: istio-system

---
# Source: istio/charts/nodeagent/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-nodeagent-istio-system
  labels:
    app: nodeagent
    chart: nodeagent
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-nodeagent-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-nodeagent-service-account
    namespace: istio-system
---
# Source: istio/charts/pilot/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-pilot-istio-system
  labels:
    app: pilot
    chart: pilot
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-pilot-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-pilot-service-account
    namespace: istio-system

---
# Source: istio/charts/security/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-citadel-istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-citadel-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-citadel-service-account
    namespace: istio-system

---
# Source: istio/charts/sidecarInjectorWebhook/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-sidecar-injector-admin-role-binding-istio-system
  labels:
    app: sidecarInjectorWebhook
    chart: sidecarInjectorWebhook
    heritage: Tiller
    release: istio
    istio: sidecar-injector
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-sidecar-injector-istio-system
subjects:
  - kind: ServiceAccount
    name: istio-sidecar-injector-service-account
    namespace: istio-system

---
# Source: istio/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: istio-multi
  labels:
    chart: istio-1.1.8
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: istio-reader
subjects:
- kind: ServiceAccount
  name: istio-multi
  namespace: istio-system

---
# Source: istio/charts/gateways/templates/role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: custom-ingressgateway-sds
  namespace: istio-system
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: istio-ingressgateway-sds
  namespace: istio-system
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
---

---
# Source: istio/charts/gateways/templates/rolebindings.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: custom-ingressgateway-sds
  namespace: istio-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: custom-ingressgateway-sds
subjects:
- kind: ServiceAccount
  name: custom-ingressgateway-service-account
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: istio-ingressgateway-sds
  namespace: istio-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: istio-ingressgateway-sds
subjects:
- kind: ServiceAccount
  name: istio-ingressgateway-service-account
---

---
# Source: istio/charts/galley/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: istio-galley
  namespace: istio-system
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    release: istio
    istio: galley
spec:
  ports:
  - port: 443
    name: https-validation
  - port: 15014
    name: http-monitoring
  - port: 9901
    name: grpc-mcp
  selector:
    istio: galley

---
# Source: istio/charts/gateways/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: custom-ingressgateway
  namespace: istio-system
  annotations:
    helm.sh/resource-policy: "keep"
    service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
    service.beta.kubernetes.io/load-balancer-source-ranges: "xxx.xxx.xxx/xx"
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: custom-ingressgateway
    istio: custom-ingressgateway
spec:
  type: LoadBalancer
  selector:
    release: istio
    app: custom-ingressgateway
    istio: custom-ingressgateway
  ports:
    -
      name: status-port
      port: 15020
      targetPort: 15020
    -
      name: http2
      nodePort: 32380
      port: 80
      targetPort: 80
    -
      name: https
      nodePort: 32390
      port: 443
---
apiVersion: v1
kind: Service
metadata:
  name: istio-egressgateway
  namespace: istio-system
  annotations:
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: istio-egressgateway
    istio: egressgateway
spec:
  type: ClusterIP
  selector:
    release: istio
    app: istio-egressgateway
    istio: egressgateway
  ports:
    -
      name: http2
      port: 80
    -
      name: https
      port: 443
    -
      name: tls
      port: 15443
      targetPort: 15443
---
apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  type: LoadBalancer
  selector:
    release: istio
    app: istio-ingressgateway
    istio: ingressgateway
  ports:
    -
      name: status-port
      port: 15020
      targetPort: 15020
    -
      name: http2
      nodePort: 31380
      port: 80
      targetPort: 80
    -
      name: https
      nodePort: 31390
      port: 443
---

---
# Source: istio/charts/mixer/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: istio-policy
  namespace: istio-system
  annotations:
   networking.istio.io/exportTo: "*"
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
    istio: mixer
spec:
  ports:
  - name: grpc-mixer
    port: 9091
  - name: grpc-mixer-mtls
    port: 15004
  - name: http-monitoring
    port: 15014
  selector:
    istio: mixer
    istio-mixer-type: policy
---
apiVersion: v1
kind: Service
metadata:
  name: istio-telemetry
  namespace: istio-system
  annotations:
   networking.istio.io/exportTo: "*"
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
    istio: mixer
spec:
  ports:
  - name: grpc-mixer
    port: 9091
  - name: grpc-mixer-mtls
    port: 15004
  - name: http-monitoring
    port: 15014
  - name: prometheus
    port: 42422
  selector:
    istio: mixer
    istio-mixer-type: telemetry
---

---
# Source: istio/charts/pilot/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: istio-pilot
  namespace: istio-system
  labels:
    app: pilot
    chart: pilot
    heritage: Tiller
    release: istio
    istio: pilot
spec:
  ports:
  - port: 15010
    name: grpc-xds # direct
  - port: 15011
    name: https-xds # mTLS
  - port: 8080
    name: http-legacy-discovery # direct
  - port: 15014
    name: http-monitoring
  selector:
    istio: pilot

---
# Source: istio/charts/security/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  # we use the normal name here (e.g. 'prometheus')
  # as grafana is configured to use this as a data source
  name: istio-citadel
  namespace: istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
    istio: citadel
spec:
  ports:
    - name: grpc-citadel
      port: 8060
      targetPort: 8060
      protocol: TCP
    - name: http-monitoring
      port: 15014
  selector:
    istio: citadel

---
# Source: istio/charts/sidecarInjectorWebhook/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: istio-sidecar-injector
  namespace: istio-system
  labels:
    app: sidecarInjectorWebhook
    chart: sidecarInjectorWebhook
    heritage: Tiller
    release: istio
    istio: sidecar-injector
spec:
  ports:
  - port: 443
  selector:
    istio: sidecar-injector

---
# Source: istio/charts/nodeagent/templates/daemonset.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: istio-nodeagent
  namespace: istio-system
  labels:
    app: nodeagent
    chart: nodeagent
    release: istio
    heritage: Tiller
    istio: nodeagent
spec:
  template:
    metadata:
      labels:
        app: nodeagent
        chart: nodeagent
        release: istio
        heritage: Tiller
        istio: nodeagent
    spec:
      serviceAccountName: istio-nodeagent-service-account
      containers:
      - name: nodeagent
        image: "docker.io/istio/node-agent-k8s:1.1.8"
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /var/run/sds
          name: sdsudspath
        env:
        - name: CA_ADDR
          value: "istio-citadel:8060"
        - name: CA_PROVIDER
          value: "Citadel"
        - name: Plugins
          value: ""
        - name: VALID_TOKEN
          value: "true"
        - name: "Trust_Domain"
          value: ""
      volumes:
      - name: sdsudspath
        hostPath:
          path: /var/run/sds
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x

---
# Source: istio/charts/galley/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-galley
  namespace: istio-system
  labels:
    app: galley
    chart: galley
    heritage: Tiller
    release: istio
    istio: galley
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: galley
        chart: galley
        heritage: Tiller
        release: istio
        istio: galley
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-galley-service-account
      containers:
        - name: galley
          image: "docker.io/istio/galley:1.1.8"
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: 443
          - containerPort: 15014
          - containerPort: 9901
          command:
          - /usr/local/bin/galley
          - server
          - --meshConfigFile=/etc/mesh-config/mesh
          - --livenessProbeInterval=1s
          - --livenessProbePath=/healthliveness
          - --readinessProbePath=/healthready
          - --readinessProbeInterval=1s
          - --deployment-namespace=istio-system
          - --insecure=true
          - --validation-webhook-config-file
          - /etc/config/validatingwebhookconfiguration.yaml
          - --monitoringPort=15014
          - --log_output_level=default:info
          volumeMounts:
          - name: certs
            mountPath: /etc/certs
            readOnly: true
          - name: config
            mountPath: /etc/config
            readOnly: true
          - name: mesh-config
            mountPath: /etc/mesh-config
            readOnly: true
          livenessProbe:
            exec:
              command:
                - /usr/local/bin/galley
                - probe
                - --probe-path=/healthliveness
                - --interval=10s
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            exec:
              command:
                - /usr/local/bin/galley
                - probe
                - --probe-path=/healthready
                - --interval=10s
            initialDelaySeconds: 5
            periodSeconds: 5
          resources:
            requests:
              cpu: 10m

      volumes:
      - name: certs
        secret:
          secretName: istio.istio-galley-service-account
      - name: config
        configMap:
          name: istio-galley-configuration
      - name: mesh-config
        configMap:
          name: istio
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x

---
# Source: istio/charts/gateways/templates/deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: custom-ingressgateway
  namespace: istio-system
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: custom-ingressgateway
    istio: custom-ingressgateway
spec:
  template:
    metadata:
      labels:
        chart: gateways
        heritage: Tiller
        release: istio
        app: custom-ingressgateway
        istio: custom-ingressgateway
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: custom-ingressgateway-service-account
      containers:
        - name: ingress-sds
          image: "docker.io/istio/node-agent-k8s:1.1.8"
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 300m
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 128Mi

          env:
          - name: "ENABLE_WORKLOAD_SDS"
            value: "false"
          - name: "ENABLE_INGRESS_GATEWAY_SDS"
            value: "true"
          - name: "INGRESS_GATEWAY_NAMESPACE"
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          volumeMounts:
          - name: ingressgatewaysdsudspath
            mountPath: /var/run/ingress_gateway
        - name: istio-proxy
          image: "docker.io/istio/proxyv2:1.1.8"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 15020
            - containerPort: 80
            - containerPort: 443
            - containerPort: 15090
              protocol: TCP
              name: http-envoy-prom
          args:
          - proxy
          - router
          - --domain
          - $(POD_NAMESPACE).svc.cluster.local
          - --log_output_level=default:info
          - --drainDuration
          - '45s' #drainDuration
          - --parentShutdownDuration
          - '1m0s' #parentShutdownDuration
          - --connectTimeout
          - '10s' #connectTimeout
          - --serviceCluster
          - custom-ingressgateway
          - --zipkinAddress
          - zipkin:9411
          - --proxyAdminPort
          - "15000"
          - --statusPort
          - "15020"
          - --controlPlaneAuthPolicy
          - NONE
          - --discoveryAddress
          - istio-pilot:15010
          readinessProbe:
            failureThreshold: 30
            httpGet:
              path: /healthz/ready
              port: 15020
              scheme: HTTP
            initialDelaySeconds: 1
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 2000m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi

          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: INSTANCE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: HOST_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.hostIP
          - name: ISTIO_META_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: ISTIO_META_CONFIG_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: ISTIO_META_USER_SDS
            value: "true"
          - name: ISTIO_META_ROUTER_MODE
            value: sni-dnat
          volumeMounts:
          - name: sdsudspath
            mountPath: /var/run/sds/uds_path
            readOnly: true
          - name: ingressgatewaysdsudspath
            mountPath: /var/run/ingress_gateway
          - name: istio-certs
            mountPath: /etc/certs
            readOnly: true
          - name: ingressgateway-certs
            mountPath: "/etc/istio/ingressgateway-certs"
            readOnly: true
          - name: ingressgateway-ca-certs
            mountPath: "/etc/istio/ingressgateway-ca-certs"
            readOnly: true
      volumes:
      - name: ingressgatewaysdsudspath
        emptyDir: {}
      - name: sdsudspath
        hostPath:
          path: /var/run/sds/uds_path
          type: Socket
      - name: istio-certs
        secret:
          secretName: istio.custom-ingressgateway-service-account
          optional: true
      - name: ingressgateway-certs
        secret:
          secretName: "istio-ingressgateway-certs"
          optional: true
      - name: ingressgateway-ca-certs
        secret:
          secretName: "istio-ingressgateway-ca-certs"
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-egressgateway
  namespace: istio-system
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: istio-egressgateway
    istio: egressgateway
spec:
  template:
    metadata:
      labels:
        chart: gateways
        heritage: Tiller
        release: istio
        app: istio-egressgateway
        istio: egressgateway
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-egressgateway-service-account
      containers:
        - name: istio-proxy
          image: "docker.io/istio/proxyv2:1.1.8"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
            - containerPort: 443
            - containerPort: 15443
            - containerPort: 15090
              protocol: TCP
              name: http-envoy-prom
          args:
          - proxy
          - router
          - --domain
          - $(POD_NAMESPACE).svc.cluster.local
          - --log_output_level=default:info
          - --drainDuration
          - '45s' #drainDuration
          - --parentShutdownDuration
          - '1m0s' #parentShutdownDuration
          - --connectTimeout
          - '10s' #connectTimeout
          - --serviceCluster
          - istio-egressgateway
          - --zipkinAddress
          - zipkin:9411
          - --proxyAdminPort
          - "15000"
          - --statusPort
          - "15020"
          - --controlPlaneAuthPolicy
          - NONE
          - --discoveryAddress
          - istio-pilot:15010
          readinessProbe:
            failureThreshold: 30
            httpGet:
              path: /healthz/ready
              port: 15020
              scheme: HTTP
            initialDelaySeconds: 1
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 2000m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi

          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: INSTANCE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: HOST_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.hostIP
          - name: ISTIO_META_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: ISTIO_META_CONFIG_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: ISTIO_META_ROUTER_MODE
            value: sni-dnat
          volumeMounts:
          - name: sdsudspath
            mountPath: /var/run/sds/uds_path
            readOnly: true
          - name: istio-certs
            mountPath: /etc/certs
            readOnly: true
          - name: egressgateway-certs
            mountPath: "/etc/istio/egressgateway-certs"
            readOnly: true
          - name: egressgateway-ca-certs
            mountPath: "/etc/istio/egressgateway-ca-certs"
            readOnly: true
      volumes:
      - name: sdsudspath
        hostPath:
          path: /var/run/sds/uds_path
          type: Socket
      - name: istio-certs
        secret:
          secretName: istio.istio-egressgateway-service-account
          optional: true
      - name: egressgateway-certs
        secret:
          secretName: "istio-egressgateway-certs"
          optional: true
      - name: egressgateway-ca-certs
        secret:
          secretName: "istio-egressgateway-ca-certs"
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  labels:
    chart: gateways
    heritage: Tiller
    release: istio
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  template:
    metadata:
      labels:
        chart: gateways
        heritage: Tiller
        release: istio
        app: istio-ingressgateway
        istio: ingressgateway
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-ingressgateway-service-account
      containers:
        - name: ingress-sds
          image: "docker.io/istio/node-agent-k8s:1.1.8"
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: 2000m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 128Mi

          env:
          - name: "ENABLE_WORKLOAD_SDS"
            value: "false"
          - name: "ENABLE_INGRESS_GATEWAY_SDS"
            value: "true"
          - name: "INGRESS_GATEWAY_NAMESPACE"
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          volumeMounts:
          - name: ingressgatewaysdsudspath
            mountPath: /var/run/ingress_gateway
        - name: istio-proxy
          image: "docker.io/istio/proxyv2:1.1.8"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 15020
            - containerPort: 80
            - containerPort: 443
            - containerPort: 15090
              protocol: TCP
              name: http-envoy-prom
          args:
          - proxy
          - router
          - --domain
          - $(POD_NAMESPACE).svc.cluster.local
          - --log_output_level=default:info
          - --drainDuration
          - '45s' #drainDuration
          - --parentShutdownDuration
          - '1m0s' #parentShutdownDuration
          - --connectTimeout
          - '10s' #connectTimeout
          - --serviceCluster
          - istio-ingressgateway
          - --zipkinAddress
          - zipkin:9411
          - --proxyAdminPort
          - "15000"
          - --statusPort
          - "15020"
          - --controlPlaneAuthPolicy
          - NONE
          - --discoveryAddress
          - istio-pilot:15010
          readinessProbe:
            failureThreshold: 30
            httpGet:
              path: /healthz/ready
              port: 15020
              scheme: HTTP
            initialDelaySeconds: 1
            periodSeconds: 2
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 2000m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 128Mi

          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: INSTANCE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: HOST_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.hostIP
          - name: ISTIO_META_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: ISTIO_META_CONFIG_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: ISTIO_META_USER_SDS
            value: "true"
          - name: ISTIO_META_ROUTER_MODE
            value: sni-dnat
          volumeMounts:
          - name: sdsudspath
            mountPath: /var/run/sds/uds_path
            readOnly: true
          - name: ingressgatewaysdsudspath
            mountPath: /var/run/ingress_gateway
          - name: istio-certs
            mountPath: /etc/certs
            readOnly: true
          - name: ingressgateway-certs
            mountPath: "/etc/istio/ingressgateway-certs"
            readOnly: true
          - name: ingressgateway-ca-certs
            mountPath: "/etc/istio/ingressgateway-ca-certs"
            readOnly: true
      volumes:
      - name: ingressgatewaysdsudspath
        emptyDir: {}
      - name: sdsudspath
        hostPath:
          path: /var/run/sds/uds_path
          type: Socket
      - name: istio-certs
        secret:
          secretName: istio.istio-ingressgateway-service-account
          optional: true
      - name: ingressgateway-certs
        secret:
          secretName: "istio-ingressgateway-certs"
          optional: true
      - name: ingressgateway-ca-certs
        secret:
          secretName: "istio-ingressgateway-ca-certs"
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x
---

---
# Source: istio/charts/mixer/templates/deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-policy
  namespace: istio-system
  labels:
    app: istio-mixer
    chart: mixer
    heritage: Tiller
    release: istio
    istio: mixer
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      istio: mixer
      istio-mixer-type: policy
  template:
    metadata:
      labels:
        app: policy
        chart: mixer
        heritage: Tiller
        release: istio
        istio: mixer
        istio-mixer-type: policy
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-mixer-service-account
      volumes:
      - name: istio-certs
        secret:
          secretName: istio.istio-mixer-service-account
          optional: true
      - name: uds-socket
        emptyDir: {}
      - name: policy-adapter-secret
        secret:
          secretName: policy-adapter-secret
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x
      containers:
      - name: mixer
        image: "docker.io/istio/mixer:1.1.8"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 15014
        - containerPort: 42422
        args:
          - --monitoringPort=15014
          - --address
          - unix:///sock/mixer.socket
          - --log_output_level=default:info
          - --configStoreURL=mcp://istio-galley.istio-system.svc:9901
          - --configDefaultNamespace=istio-system
          - --useAdapterCRDs=true
          - --trace_zipkin_url=http://zipkin:9411/api/v1/spans
        env:
        - name: GODEBUG
          value: "gctrace=1"
        - name: GOMAXPROCS
          value: "6"
        resources:
          requests:
            cpu: 10m

        volumeMounts:
        - name: istio-certs
          mountPath: /etc/certs
          readOnly: true
        - name: uds-socket
          mountPath: /sock
        livenessProbe:
          httpGet:
            path: /version
            port: 15014
          initialDelaySeconds: 5
          periodSeconds: 5
      - name: istio-proxy
        image: "docker.io/istio/proxyv2:1.1.8"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9091
        - containerPort: 15004
        - containerPort: 15090
          protocol: TCP
          name: http-envoy-prom
        args:
        - proxy
        - --domain
        - $(POD_NAMESPACE).svc.cluster.local
        - --serviceCluster
        - istio-policy
        - --templateFile
        - /etc/istio/proxy/envoy_policy.yaml.tmpl
        - --controlPlaneAuthPolicy
        - NONE
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        resources:
          limits:
            cpu: 300m
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 50Mi

        volumeMounts:
        - name: istio-certs
          mountPath: /etc/certs
          readOnly: true
        - name: uds-socket
          mountPath: /sock
        - name: policy-adapter-secret
          mountPath: /var/run/secrets/istio.io/policy/adapter
          readOnly: true

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-telemetry
  namespace: istio-system
  labels:
    app: istio-mixer
    chart: mixer
    heritage: Tiller
    release: istio
    istio: mixer
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      istio: mixer
      istio-mixer-type: telemetry
  template:
    metadata:
      labels:
        app: telemetry
        chart: mixer
        heritage: Tiller
        release: istio
        istio: mixer
        istio-mixer-type: telemetry
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-mixer-service-account
      volumes:
      - name: istio-certs
        secret:
          secretName: istio.istio-mixer-service-account
          optional: true
      - name: uds-socket
        emptyDir: {}
      - name: telemetry-adapter-secret
        secret:
          secretName: telemetry-adapter-secret
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x
      containers:
      - name: mixer
        image: "docker.io/istio/mixer:1.1.8"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 15014
        - containerPort: 42422
        args:
          - --monitoringPort=15014
          - --address
          - unix:///sock/mixer.socket
          - --log_output_level=default:info
          - --configStoreURL=mcp://istio-galley.istio-system.svc:9901
          - --configDefaultNamespace=istio-system
          - --useAdapterCRDs=true
          - --trace_zipkin_url=http://zipkin:9411/api/v1/spans
          - --averageLatencyThreshold
          - 100ms
          - --loadsheddingMode
          - enforce
        env:
        - name: GODEBUG
          value: "gctrace=1"
        - name: GOMAXPROCS
          value: "6"
        resources:
          limits:
            cpu: 500m
            memory: 1G
          requests:
            cpu: 500m
            memory: 1G

        volumeMounts:
        - name: istio-certs
          mountPath: /etc/certs
          readOnly: true
        - name: telemetry-adapter-secret
          mountPath: /var/run/secrets/istio.io/telemetry/adapter
          readOnly: true
        - name: uds-socket
          mountPath: /sock
        livenessProbe:
          httpGet:
            path: /version
            port: 15014
          initialDelaySeconds: 5
          periodSeconds: 5
      - name: istio-proxy
        image: "docker.io/istio/proxyv2:1.1.8"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9091
        - containerPort: 15004
        - containerPort: 15090
          protocol: TCP
          name: http-envoy-prom
        args:
        - proxy
        - --domain
        - $(POD_NAMESPACE).svc.cluster.local
        - --serviceCluster
        - istio-telemetry
        - --templateFile
        - /etc/istio/proxy/envoy_telemetry.yaml.tmpl
        - --controlPlaneAuthPolicy
        - NONE
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        resources:
          limits:
            cpu: 300m
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 50Mi

        volumeMounts:
        - name: istio-certs
          mountPath: /etc/certs
          readOnly: true
        - name: uds-socket
          mountPath: /sock

---

---
# Source: istio/charts/pilot/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-pilot
  namespace: istio-system
  # TODO: default template doesn't have this, which one is right ?
  labels:
    app: pilot
    chart: pilot
    heritage: Tiller
    release: istio
    istio: pilot
  annotations:
    checksum/config-volume: f8da08b6b8c170dde721efd680270b2901e750d4aa186ebb6c22bef5b78a43f9
spec:
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      istio: pilot
  template:
    metadata:
      labels:
        app: pilot
        chart: pilot
        heritage: Tiller
        release: istio
        istio: pilot
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-pilot-service-account
      containers:
        - name: discovery
          image: "docker.io/istio/pilot:1.1.8"
          imagePullPolicy: IfNotPresent
          args:
          - "discovery"
          - --monitoringAddr=:15014
          - --log_output_level=default:info
          - --domain
          - cluster.local
          - --secureGrpcAddr
          - ""
          - --keepaliveMaxServerConnectionAge
          - "30m"
          ports:
          - containerPort: 8080
          - containerPort: 15010
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 30
            timeoutSeconds: 5
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: GODEBUG
            value: "gctrace=1"
          - name: PILOT_PUSH_THROTTLE
            value: "100"
          - name: PILOT_DISABLE_XDS_MARSHALING_TO_ANY
            value: "1"
          resources:
            requests:
              cpu: 200m
              memory: 1048Mi

          volumeMounts:
          - name: config-volume
            mountPath: /etc/istio/config
          - name: istio-certs
            mountPath: /etc/certs
            readOnly: true
        - name: istio-proxy
          image: "docker.io/istio/proxyv2:1.1.8"
          imagePullPolicy: IfNotPresent
          ports:
          - containerPort: 15003
          - containerPort: 15005
          - containerPort: 15007
          - containerPort: 15011
          args:
          - proxy
          - --domain
          - $(POD_NAMESPACE).svc.cluster.local
          - --serviceCluster
          - istio-pilot
          - --templateFile
          - /etc/istio/proxy/envoy_pilot.yaml.tmpl
          - --controlPlaneAuthPolicy
          - NONE
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: INSTANCE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          resources:
            limits:
              cpu: 300m
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 50Mi

          volumeMounts:
          - name: istio-certs
            mountPath: /etc/certs
            readOnly: true
      volumes:
      - name: config-volume
        configMap:
          name: istio
      - name: istio-certs
        secret:
          secretName: istio.istio-pilot-service-account
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x

---
# Source: istio/charts/security/templates/deployment.yaml
# istio CA watching all namespaces
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-citadel
  namespace: istio-system
  labels:
    app: security
    chart: security
    heritage: Tiller
    release: istio
    istio: citadel
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: security
        chart: security
        heritage: Tiller
        release: istio
        istio: citadel
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-citadel-service-account
      containers:
        - name: citadel
          image: "docker.io/istio/citadel:1.1.8"
          imagePullPolicy: IfNotPresent
          args:
            - --append-dns-names=true
            - --grpc-port=8060
            - --grpc-hostname=citadel
            - --citadel-storage-namespace=istio-system
            - --custom-dns-names=istio-pilot-service-account.istio-system:istio-pilot.istio-system
            - --monitoring-port=15014
            - --self-signed-ca=true
          livenessProbe:
            httpGet:
              path: /version
              port: 15014
            initialDelaySeconds: 5
            periodSeconds: 5
          resources:
            requests:
              cpu: 10m

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x

---
# Source: istio/charts/sidecarInjectorWebhook/templates/deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: istio-sidecar-injector
  namespace: istio-system
  labels:
    app: sidecarInjectorWebhook
    chart: sidecarInjectorWebhook
    heritage: Tiller
    release: istio
    istio: sidecar-injector
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: sidecarInjectorWebhook
        chart: sidecarInjectorWebhook
        heritage: Tiller
        release: istio
        istio: sidecar-injector
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      serviceAccountName: istio-sidecar-injector-service-account
      containers:
        - name: sidecar-injector-webhook
          image: "docker.io/istio/sidecar_injector:1.1.8"
          imagePullPolicy: IfNotPresent
          args:
            - --caCertFile=/etc/istio/certs/root-cert.pem
            - --tlsCertFile=/etc/istio/certs/cert-chain.pem
            - --tlsKeyFile=/etc/istio/certs/key.pem
            - --injectConfig=/etc/istio/inject/config
            - --meshConfig=/etc/istio/config/mesh
            - --healthCheckInterval=2s
            - --healthCheckFile=/health
          volumeMounts:
          - name: config-volume
            mountPath: /etc/istio/config
            readOnly: true
          - name: certs
            mountPath: /etc/istio/certs
            readOnly: true
          - name: inject-config
            mountPath: /etc/istio/inject
            readOnly: true
          livenessProbe:
            exec:
              command:
                - /usr/local/bin/sidecar-injector
                - probe
                - --probe-path=/health
                - --interval=4s
            initialDelaySeconds: 4
            periodSeconds: 4
          readinessProbe:
            exec:
              command:
                - /usr/local/bin/sidecar-injector
                - probe
                - --probe-path=/health
                - --interval=4s
            initialDelaySeconds: 4
            periodSeconds: 4
          resources:
            requests:
              cpu: 10m

      volumes:
      - name: config-volume
        configMap:
          name: istio
      - name: certs
        secret:
          secretName: istio.istio-sidecar-injector-service-account
      - name: inject-config
        configMap:
          name: istio-sidecar-injector
          items:
          - key: config
            path: config
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x

---
# Source: istio/charts/gateways/templates/autoscale.yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: custom-ingressgateway
  namespace: istio-system
  labels:
    app: custom-ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: custom-ingressgateway
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: istio-egressgateway
  namespace: istio-system
  labels:
    app: egressgateway
    chart: gateways
    heritage: Tiller
    release: istio
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: istio-egressgateway
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: istio-ingressgateway
  namespace: istio-system
  labels:
    app: ingressgateway
    chart: gateways
    heritage: Tiller
    release: istio
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: istio-ingressgateway
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80
---

---
# Source: istio/charts/mixer/templates/autoscale.yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: istio-policy
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
    maxReplicas: 5
    minReplicas: 1
    scaleTargetRef:
      apiVersion: apps/v1beta1
      kind: Deployment
      name: istio-policy
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: istio-telemetry
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
    maxReplicas: 5
    minReplicas: 1
    scaleTargetRef:
      apiVersion: apps/v1beta1
      kind: Deployment
      name: istio-telemetry
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80
---

---
# Source: istio/charts/pilot/templates/autoscale.yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: istio-pilot
  namespace: istio-system
  labels:
    app: pilot
    chart: pilot
    heritage: Tiller
    release: istio
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: istio-pilot
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 80
---

---
# Source: istio/charts/sidecarInjectorWebhook/templates/mutatingwebhook.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: istio-sidecar-injector
  namespace: istio-system
  labels:
    app: sidecarInjectorWebhook
    chart: sidecarInjectorWebhook
    heritage: Tiller
    release: istio
webhooks:
  - name: sidecar-injector.istio.io
    clientConfig:
      service:
        name: istio-sidecar-injector
        namespace: istio-system
        path: "/inject"
      caBundle: ""
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    failurePolicy: Fail
    namespaceSelector:
      matchLabels:
        istio-injection: enabled

---
# Source: istio/charts/galley/templates/validatingwebhookconfiguration.yaml.tpl

---
# Source: istio/charts/gateways/templates/preconfigured.yaml

---
# Source: istio/charts/pilot/templates/meshexpansion.yaml

---
# Source: istio/charts/security/templates/enable-mesh-mtls.yaml

---
# Source: istio/charts/security/templates/enable-mesh-permissive.yaml

---
# Source: istio/charts/security/templates/meshexpansion.yaml

---
# Source: istio/charts/security/templates/tests/test-citadel-connection.yaml

---
# Source: istio/templates/endpoints.yaml

---
# Source: istio/templates/install-custom-resources.sh.tpl

---
# Source: istio/templates/service.yaml

---
# Source: istio/charts/mixer/templates/config.yaml

apiVersion: "config.istio.io/v1alpha2"
kind: attributemanifest
metadata:
  name: istioproxy
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  attributes:
    origin.ip:
      valueType: IP_ADDRESS
    origin.uid:
      valueType: STRING
    origin.user:
      valueType: STRING
    request.headers:
      valueType: STRING_MAP
    request.id:
      valueType: STRING
    request.host:
      valueType: STRING
    request.method:
      valueType: STRING
    request.path:
      valueType: STRING
    request.url_path:
      valueType: STRING
    request.query_params:
      valueType: STRING_MAP
    request.reason:
      valueType: STRING
    request.referer:
      valueType: STRING
    request.scheme:
      valueType: STRING
    request.total_size:
      valueType: INT64
    request.size:
      valueType: INT64
    request.time:
      valueType: TIMESTAMP
    request.useragent:
      valueType: STRING
    response.code:
      valueType: INT64
    response.duration:
      valueType: DURATION
    response.headers:
      valueType: STRING_MAP
    response.total_size:
      valueType: INT64
    response.size:
      valueType: INT64
    response.time:
      valueType: TIMESTAMP
    response.grpc_status:
      valueType: STRING
    response.grpc_message:
      valueType: STRING
    source.uid:
      valueType: STRING
    source.user: # DEPRECATED
      valueType: STRING
    source.principal:
      valueType: STRING
    destination.uid:
      valueType: STRING
    destination.principal:
      valueType: STRING
    destination.port:
      valueType: INT64
    connection.event:
      valueType: STRING
    connection.id:
      valueType: STRING
    connection.received.bytes:
      valueType: INT64
    connection.received.bytes_total:
      valueType: INT64
    connection.sent.bytes:
      valueType: INT64
    connection.sent.bytes_total:
      valueType: INT64
    connection.duration:
      valueType: DURATION
    connection.mtls:
      valueType: BOOL
    connection.requested_server_name:
      valueType: STRING
    context.protocol:
      valueType: STRING
    context.proxy_error_code:
      valueType: STRING
    context.timestamp:
      valueType: TIMESTAMP
    context.time:
      valueType: TIMESTAMP
    # Deprecated, kept for compatibility
    context.reporter.local:
      valueType: BOOL
    context.reporter.kind:
      valueType: STRING
    context.reporter.uid:
      valueType: STRING
    api.service:
      valueType: STRING
    api.version:
      valueType: STRING
    api.operation:
      valueType: STRING
    api.protocol:
      valueType: STRING
    request.auth.principal:
      valueType: STRING
    request.auth.audiences:
      valueType: STRING
    request.auth.presenter:
      valueType: STRING
    request.auth.claims:
      valueType: STRING_MAP
    request.auth.raw_claims:
      valueType: STRING
    request.api_key:
      valueType: STRING
    rbac.permissive.response_code:
      valueType: STRING
    rbac.permissive.effective_policy_id:
      valueType: STRING
    check.error_code:
      valueType: INT64
    check.error_message:
      valueType: STRING
    check.cache_hit:
      valueType: BOOL
    quota.cache_hit:
      valueType: BOOL

---
apiVersion: "config.istio.io/v1alpha2"
kind: attributemanifest
metadata:
  name: kubernetes
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  attributes:
    source.ip:
      valueType: IP_ADDRESS
    source.labels:
      valueType: STRING_MAP
    source.metadata:
      valueType: STRING_MAP
    source.name:
      valueType: STRING
    source.namespace:
      valueType: STRING
    source.owner:
      valueType: STRING
    source.serviceAccount:
      valueType: STRING
    source.services:
      valueType: STRING
    source.workload.uid:
      valueType: STRING
    source.workload.name:
      valueType: STRING
    source.workload.namespace:
      valueType: STRING
    destination.ip:
      valueType: IP_ADDRESS
    destination.labels:
      valueType: STRING_MAP
    destination.metadata:
      valueType: STRING_MAP
    destination.owner:
      valueType: STRING
    destination.name:
      valueType: STRING
    destination.container.name:
      valueType: STRING
    destination.namespace:
      valueType: STRING
    destination.service.uid:
      valueType: STRING
    destination.service.name:
      valueType: STRING
    destination.service.namespace:
      valueType: STRING
    destination.service.host:
      valueType: STRING
    destination.serviceAccount:
      valueType: STRING
    destination.workload.uid:
      valueType: STRING
    destination.workload.name:
      valueType: STRING
    destination.workload.namespace:
      valueType: STRING
---
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: requestcount
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: "1"
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    request_protocol: api.protocol | context.protocol | "unknown"
    response_code: response.code | 200
    response_flags: context.proxy_error_code | "-"
    permissive_response_code: rbac.permissive.response_code | "none"
    permissive_response_policyid: rbac.permissive.effective_policy_id | "none"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: requestduration
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: response.duration | "0ms"
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    request_protocol: api.protocol | context.protocol | "unknown"
    response_code: response.code | 200
    response_flags: context.proxy_error_code | "-"
    permissive_response_code: rbac.permissive.response_code | "none"
    permissive_response_policyid: rbac.permissive.effective_policy_id | "none"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: requestsize
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: request.size | 0
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    request_protocol: api.protocol | context.protocol | "unknown"
    response_code: response.code | 200
    response_flags: context.proxy_error_code | "-"
    permissive_response_code: rbac.permissive.response_code | "none"
    permissive_response_policyid: rbac.permissive.effective_policy_id | "none"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: responsesize
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: response.size | 0
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    request_protocol: api.protocol | context.protocol | "unknown"
    response_code: response.code | 200
    response_flags: context.proxy_error_code | "-"
    permissive_response_code: rbac.permissive.response_code | "none"
    permissive_response_policyid: rbac.permissive.effective_policy_id | "none"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: tcpbytesent
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: connection.sent.bytes | 0
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
    response_flags: context.proxy_error_code | "-"
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: tcpbytereceived
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: connection.received.bytes | 0
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.host | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
    response_flags: context.proxy_error_code | "-"
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: tcpconnectionsopened
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: "1"
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.name | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
    response_flags: context.proxy_error_code | "-"
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: tcpconnectionsclosed
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  value: "1"
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "source", "destination")
    source_workload: source.workload.name | "unknown"
    source_workload_namespace: source.workload.namespace | "unknown"
    source_principal: source.principal | "unknown"
    source_app: source.labels["app"] | "unknown"
    source_version: source.labels["version"] | "unknown"
    destination_workload: destination.workload.name | "unknown"
    destination_workload_namespace: destination.workload.namespace | "unknown"
    destination_principal: destination.principal | "unknown"
    destination_app: destination.labels["app"] | "unknown"
    destination_version: destination.labels["version"] | "unknown"
    destination_service: destination.service.name | "unknown"
    destination_service_name: destination.service.name | "unknown"
    destination_service_namespace: destination.service.namespace | "unknown"
    connection_security_policy: conditional((context.reporter.kind | "inbound") == "outbound", "unknown", conditional(connection.mtls | false, "mutual_tls", "none"))
    response_flags: context.proxy_error_code | "-"
  monitored_resource_type: '"UNSPECIFIED"'
---
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
  name: prometheus
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  compiledAdapter: prometheus
  params:
    metricsExpirationPolicy:
      metricsExpiryDuration: "10m"
    metrics:
    - name: requests_total
      instance_name: requestcount.metric.istio-system
      kind: COUNTER
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - request_protocol
      - response_code
      - response_flags
      - permissive_response_code
      - permissive_response_policyid
      - connection_security_policy
    - name: request_duration_seconds
      instance_name: requestduration.metric.istio-system
      kind: DISTRIBUTION
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - request_protocol
      - response_code
      - response_flags
      - permissive_response_code
      - permissive_response_policyid
      - connection_security_policy
      buckets:
        explicit_buckets:
          bounds: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
    - name: request_bytes
      instance_name: requestsize.metric.istio-system
      kind: DISTRIBUTION
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - request_protocol
      - response_code
      - response_flags
      - permissive_response_code
      - permissive_response_policyid
      - connection_security_policy
      buckets:
        exponentialBuckets:
          numFiniteBuckets: 8
          scale: 1
          growthFactor: 10
    - name: response_bytes
      instance_name: responsesize.metric.istio-system
      kind: DISTRIBUTION
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - request_protocol
      - response_code
      - response_flags
      - permissive_response_code
      - permissive_response_policyid
      - connection_security_policy
      buckets:
        exponentialBuckets:
          numFiniteBuckets: 8
          scale: 1
          growthFactor: 10
    - name: tcp_sent_bytes_total
      instance_name: tcpbytesent.metric.istio-system
      kind: COUNTER
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - connection_security_policy
      - response_flags
    - name: tcp_received_bytes_total
      instance_name: tcpbytereceived.metric.istio-system
      kind: COUNTER
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - connection_security_policy
      - response_flags
    - name: tcp_connections_opened_total
      instance_name: tcpconnectionsopened.metric.istio-system
      kind: COUNTER
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - connection_security_policy
      - response_flags
    - name: tcp_connections_closed_total
      instance_name: tcpconnectionsclosed.metric.istio-system
      kind: COUNTER
      label_names:
      - reporter
      - source_app
      - source_principal
      - source_workload
      - source_workload_namespace
      - source_version
      - destination_app
      - destination_principal
      - destination_workload
      - destination_workload_namespace
      - destination_version
      - destination_service
      - destination_service_name
      - destination_service_namespace
      - connection_security_policy
      - response_flags
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: promhttp
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  match: (context.protocol == "http" || context.protocol == "grpc") && (match((request.useragent | "-"), "kube-probe*") == false)
  actions:
  - handler: prometheus
    instances:
    - requestcount.metric
    - requestduration.metric
    - requestsize.metric
    - responsesize.metric
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: promtcp
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  match: context.protocol == "tcp"
  actions:
  - handler: prometheus
    instances:
    - tcpbytesent.metric
    - tcpbytereceived.metric
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: promtcpconnectionopen
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  match: context.protocol == "tcp" && ((connection.event | "na") == "open")
  actions:
  - handler: prometheus
    instances:
    - tcpconnectionsopened.metric
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: promtcpconnectionclosed
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  match: context.protocol == "tcp" && ((connection.event | "na") == "close")
  actions:
  - handler: prometheus
    instances:
    - tcpconnectionsclosed.metric
---
apiVersion: "config.istio.io/v1alpha2"
kind: handler
metadata:
  name: kubernetesenv
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  compiledAdapter: kubernetesenv
  params:
    # when running from mixer root, use the following config after adding a
    # symbolic link to a kubernetes config file via:
    #
    # $ ln -s ~/.kube/config mixer/adapter/kubernetes/kubeconfig
    #
    # kubeconfig_path: "mixer/adapter/kubernetes/kubeconfig"

---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: kubeattrgenrulerule
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  actions:
  - handler: kubernetesenv
    instances:
    - attributes.kubernetes
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: tcpkubeattrgenrulerule
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  match: context.protocol == "tcp"
  actions:
  - handler: kubernetesenv
    instances:
    - attributes.kubernetes
---
apiVersion: "config.istio.io/v1alpha2"
kind: kubernetes
metadata:
  name: attributes
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  # Pass the required attribute data to the adapter
  source_uid: source.uid | ""
  source_ip: source.ip | ip("0.0.0.0") # default to unspecified ip addr
  destination_uid: destination.uid | ""
  destination_port: destination.port | 0
  attribute_bindings:
    # Fill the new attributes from the adapter produced output.
    # $out refers to an instance of OutputTemplate message
    source.ip: $out.source_pod_ip | ip("0.0.0.0")
    source.uid: $out.source_pod_uid | "unknown"
    source.labels: $out.source_labels | emptyStringMap()
    source.name: $out.source_pod_name | "unknown"
    source.namespace: $out.source_namespace | "default"
    source.owner: $out.source_owner | "unknown"
    source.serviceAccount: $out.source_service_account_name | "unknown"
    source.workload.uid: $out.source_workload_uid | "unknown"
    source.workload.name: $out.source_workload_name | "unknown"
    source.workload.namespace: $out.source_workload_namespace | "unknown"
    destination.ip: $out.destination_pod_ip | ip("0.0.0.0")
    destination.uid: $out.destination_pod_uid | "unknown"
    destination.labels: $out.destination_labels | emptyStringMap()
    destination.name: $out.destination_pod_name | "unknown"
    destination.container.name: $out.destination_container_name | "unknown"
    destination.namespace: $out.destination_namespace | "default"
    destination.owner: $out.destination_owner | "unknown"
    destination.serviceAccount: $out.destination_service_account_name | "unknown"
    destination.workload.uid: $out.destination_workload_uid | "unknown"
    destination.workload.name: $out.destination_workload_name | "unknown"
    destination.workload.namespace: $out.destination_workload_namespace | "unknown"
---
# Configuration needed by Mixer.
# Mixer cluster is delivered via CDS
# Specify mixer cluster settings
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: istio-policy
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  host: istio-policy.istio-system.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 10000
        maxRequestsPerConnection: 10000
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: istio-telemetry
  namespace: istio-system
  labels:
    app: mixer
    chart: mixer
    heritage: Tiller
    release: istio
spec:
  host: istio-telemetry.istio-system.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 10000
        maxRequestsPerConnection: 10000
---

faced the same issue, seems there is an issue in ingressgateway - After I forcibly killed its pod, I found the issue disappeared

faced the same issue with istio 1.9, seems there is an issue in ingressgateway - After I forcibly killed its pod (restarted after termination), I found the issue disappeared

faced the same issue with istio 1.9, seems there is an issue in ingressgateway - After I forcibly killed its pod (restarted after termination), I found the issue disappeared

Thanks msolimans, it works! @msolimans

Same here, .. when I bounce the ingress gateway pod, I can curl a target  endpoint
`
vinayb@carbon ~/Downloads/tmp/nats-sub $ curl http://192.168.1.133:32499/
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIREDvinayb@carbon ~/Downloads/tmp/nats-sub $ curl -v http://192.168.1.133:32499/

Trying 192.168.1.133:32499...
Connected to 192.168.1.133 (192.168.1.133) port 32499 (#0)

GET / HTTP/1.1
Host: 192.168.1.133:32499
User-Agent: curl/7.73.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 201
< content-type: text/plain
< date: Mon, 24 May 2021 18:01:53 GMT
< server: istio-envoy
<
Connection #0 to host 192.168.1.133 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIREDvinayb@carbon ~/Downloads/tmp/nats-sub $ kubectl get po -A
NAMESPACE      NAME                                               READY   STATUS    RESTARTS   AGE
kube-system    metrics-server-86cbb8457f-6smlc                    1/1     Running   14         65d
keda           keda-operator-968bbb969-9g62x                      1/1     Running   14         65d
keda           keda-operator-metrics-apiserver-6db5fd7f95-8w5tp   1/1     Running   14         65d
kube-system    k8dash-87fd8b696-d696g                             1/1     Running   14         65d
kube-system    coredns-854c77959c-68fbx                           1/1     Running   14         65d
default        nats-pub-5b477bbf4c-v2bct                          2/2     Running   0          3d
default        nats-sub-56884f9f8-6422f                           2/2     Running   0          3d
istio-system   istiod-cfdddd5c6-9c8cd                             1/1     Running   0          2d23h
istio-system   svclb-istio-ingressgateway-f7qtz                   5/5     Running   0          2d23h
istio-system   istio-egressgateway-6d5b65b76-vbv7z                1/1     Running   0          2d23h
istio-system   istio-ingressgateway-9b8b5945d-w99xv               1/1     Running   0          2d23h
istio-system   jaeger-7f78b6fb65-h2lgd                            1/1     Running   0          2d23h
istio-system   kiali-85c8cdd5b5-m7zr9                             1/1     Running   0          2d23h
istio-system   grafana-f766d6c97-ddh5g                            1/1     Running   0          2d23h
istio-system   prometheus-69f7f4d689-wfq57                        2/2     Running   0          2d23h
kube-system    local-path-provisioner-5ff76fc89d-57vfc            1/1     Running   73         65d
default        php-apache-d4cf67d68-xvj95                         2/2     Running   0          6m8s
vinayb@carbon ~/Downloads/tmp/nats-sub $ kubectl delete po istio-ingressgateway-9b8b5945d-w99xv    -n istio-system
pod "istio-ingressgateway-9b8b5945d-w99xv" deleted
vinayb@carbon ~/Downloads/tmp/nats-sub $ kubectl get po -A
NAMESPACE      NAME                                               READY   STATUS    RESTARTS   AGE
kube-system    metrics-server-86cbb8457f-6smlc                    1/1     Running   14         65d
keda           keda-operator-968bbb969-9g62x                      1/1     Running   14         65d
keda           keda-operator-metrics-apiserver-6db5fd7f95-8w5tp   1/1     Running   14         65d
kube-system    k8dash-87fd8b696-d696g                             1/1     Running   14         65d
kube-system    coredns-854c77959c-68fbx                           1/1     Running   14         65d
default        nats-pub-5b477bbf4c-v2bct                          2/2     Running   0          3d
default        nats-sub-56884f9f8-6422f                           2/2     Running   0          3d
istio-system   istiod-cfdddd5c6-9c8cd                             1/1     Running   0          2d23h
istio-system   svclb-istio-ingressgateway-f7qtz                   5/5     Running   0          2d23h
istio-system   istio-egressgateway-6d5b65b76-vbv7z                1/1     Running   0          2d23h
istio-system   jaeger-7f78b6fb65-h2lgd                            1/1     Running   0          2d23h
istio-system   kiali-85c8cdd5b5-m7zr9                             1/1     Running   0          2d23h
istio-system   grafana-f766d6c97-ddh5g                            1/1     Running   0          2d23h
istio-system   prometheus-69f7f4d689-wfq57                        2/2     Running   0          2d23h
kube-system    local-path-provisioner-5ff76fc89d-57vfc            1/1     Running   73         65d
default        php-apache-d4cf67d68-xvj95                         2/2     Running   0          6m48s
istio-system   istio-ingressgateway-9b8b5945d-t4hms               1/1     Running   0          24s
vinayb@carbon ~/Downloads/tmp/nats-sub $ curl -v http://192.168.1.133:32499/
Trying 192.168.1.133:32499...
Connected to 192.168.1.133 (192.168.1.133) port 32499 (#0)

GET / HTTP/1.1
Host: 192.168.1.133:32499
User-Agent: curl/7.73.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Mon, 24 May 2021 18:04:28 GMT
< server: istio-envoy
< x-powered-by: PHP/5.6.14
< content-length: 3
< content-type: text/html; charset=UTF-8
< x-envoy-upstream-service-time: 134
<

`

In istio 1.9.3 also face same issue, is this issue will resolve in the futrue?

We're facing the same issue. Currently can also replicate in k8s in Docker Desktop . We recently rolled forward from Istio 1.8.1 to 1.10x and also now 1.11x and noticed the exact same issue with certs failing in the cluster causing 503s in Postman etc. It runs fine for a while (as little as a day) and then fails eventually. We're using the standard Istio Operator for deployment and my understanding is that SDS is enabled in the cluster and on the ingress gateway as standard nowadays.
k8 version: 1.21.3
istio version: 1.11.1
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istio-control-plane
spec:
profile: default
meshConfig:
accessLogFile: "/dev/stdout"
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
2021-09-01T10:33:04.283147Z     debug   envoy pool      [C9632] client disconnected, failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
All other logs look fine but this only seems to show up in the debug logs in the ingress gateway.
This is causing some serious developer constenation and concerns. Seems to be going on for a while now based on the above. Can we please get some advice and possible resolutions?!

Faced the same issue, a restart of the ingress pods and deployments fixed the issue. This issue requires more attention and investigation from the development side. This is certainly a reoccurring issue among users.

authors: CSimpiFoN DanoThom Janesee3 JimmyCYJ LuEason balamuru haiker2011 msolimans pitlv2109 quanjielin

repo: istio/istio