text:
Sidecar injection fails when using PSP with allowPrivilegeEscalation=false and istio-init container privleged=false

Describe the bug
Failed to deploy automatically injected sidecar container, with the istio-sidecar-injector configmap is configured  to use securityContext: privileged: false for proxy-init, when using a PodSecurityPolicy with privilegeEscalationAllowed: false and CAP_NET_ADMIN enabled, yet it works for manual sidecar injection using the same PSPs/ClusterRole/ClusterRoleBinding.

With automatic sidecar injection enabled
kubectl label namespace default istio-injection=enabled
kubectl get deployment httpbin

      message: 'pods "httpbin-7cf66f94c4-hzhhd" is forbidden: unable to validate against
        any pod security policy: [spec.initContainers[0].securityContext.capabilities.add:
        Invalid value: "NET_ADMIN": capability may not be added]'
      reason: FailedCreate

With manual sidecar injection and namespace label removed
kubectl label namespace default istio-injection-
istioctl kube-inject -f httpbin.yaml | kubectl create -f -

kubectl get pods
NAME                      READY     STATUS            RESTARTS   AGE
httpbin-986b585bf-z5xkb   0/2       PodInitializing   0          3s

sidecar-injector configmap
kubectl -n istio-system get cm istio-sidecar-injector -o yaml

apiVersion: v1
data:
  config: "policy: enabled\ntemplate: |-\n  initContainers:\n  - name: istio-init\n
    \   image: \"istio/proxy_init:1.0.3\"\n    args:\n    -
    \"-p\"\n    - [[ .MeshConfig.ProxyListenPort ]]\n    - \"-u\"\n    - 1337\n    -
    \"-m\"\n    - [[ annotation .ObjectMeta `sidecar.istio.io/interceptionMode` .ProxyConfig.InterceptionMode
    ]]\n    - \"-i\"\n    - \"[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeOutboundIPRanges`
    \ \"100.64.0.0/13\"  ]]\"\n    - \"-x\"\n    - \"[[ annotation .ObjectMeta `traffic.sidecar.istio.io/excludeOutboundIPRanges`
    \ \"\"  ]]\"\n    - \"-b\"\n    - \"[[ annotation .ObjectMeta `traffic.sidecar.istio.io/includeInboundPorts`
    (includeInboundPorts .Spec.Containers) ]]\"\n    - \"-d\"\n    - \"[[ excludeInboundPort
    (annotation .ObjectMeta `status.sidecar.istio.io/port`  0 ) (annotation .ObjectMeta
    `traffic.sidecar.istio.io/excludeInboundPorts`  \"\" ) ]]\"\n    imagePullPolicy:
    IfNotPresent\n    securityContext:\n      capabilities:\n        add:\n        -
    NET_ADMIN\n      privileged: false \n    restartPolicy: Always\n  containers:\n
    \ - name: istio-proxy\n    image: [[ annotation .ObjectMeta `sidecar.istio.io/proxyImage`
    \ \"istio/proxyv2:1.0.3\"  ]]\n\n    ports:\n    - containerPort:
    15090\n      protocol: TCP\n      name: http-envoy-prom\n\n    args:\n    - proxy\n
    \   - sidecar\n    - --configPath\n    - [[ .ProxyConfig.ConfigPath ]]\n    -
    --binaryPath\n    - [[ .ProxyConfig.BinaryPath ]]\n    - --serviceCluster\n    [[
    if ne \"\" (index .ObjectMeta.Labels \"app\") -]]\n    - [[ index .ObjectMeta.Labels
    \"app\" ]]\n    [[ else -]]\n    - \"istio-proxy\"\n    [[ end -]]\n    - --drainDuration\n
    \   - [[ formatDuration .ProxyConfig.DrainDuration ]]\n    - --parentShutdownDuration\n
    \   - [[ formatDuration .ProxyConfig.ParentShutdownDuration ]]\n    - --discoveryAddress\n
    \   - [[ .ProxyConfig.DiscoveryAddress ]]\n    - --discoveryRefreshDelay\n    -
    [[ formatDuration .ProxyConfig.DiscoveryRefreshDelay ]]\n    - --zipkinAddress\n
    \   - [[ .ProxyConfig.ZipkinAddress ]]\n    - --connectTimeout\n    - [[ formatDuration
    .ProxyConfig.ConnectTimeout ]]\n    - --proxyAdminPort\n    - [[ .ProxyConfig.ProxyAdminPort
    ]]\n    [[ if gt .ProxyConfig.Concurrency 0 -]]\n    - --concurrency\n    - [[
    .ProxyConfig.Concurrency ]]\n    [[ end -]]\n    - --controlPlaneAuthPolicy\n
    \   - [[ annotation .ObjectMeta `sidecar.istio.io/controlPlaneAuthPolicy` .ProxyConfig.ControlPlaneAuthPolicy
    ]]\n  [[- if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port`  0 ) \"0\")
    ]]\n    - --statusPort\n    - [[ annotation .ObjectMeta `status.sidecar.istio.io/port`
    \ 0  ]]\n    - --applicationPorts\n    - \"[[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/applicationPorts`
    (applicationPorts .Spec.Containers) ]]\"\n  [[- end ]]\n    env:\n    - name:
    POD_NAME\n      valueFrom:\n        fieldRef:\n          fieldPath: metadata.name\n
    \   - name: POD_NAMESPACE\n      valueFrom:\n        fieldRef:\n          fieldPath:
    metadata.namespace\n    - name: INSTANCE_IP\n      valueFrom:\n        fieldRef:\n
    \         fieldPath: status.podIP\n    - name: ISTIO_META_POD_NAME\n      valueFrom:\n
    \       fieldRef:\n          fieldPath: metadata.name\n    - name: ISTIO_META_INTERCEPTION_MODE\n
    \     value: [[ or (index .ObjectMeta.Annotations \"sidecar.istio.io/interceptionMode\")
    .ProxyConfig.InterceptionMode.String ]]\n    [[ if .ObjectMeta.Annotations ]]\n
    \   - name: ISTIO_METAJSON_ANNOTATIONS\n      value: |\n             [[ toJson
    .ObjectMeta.Annotations ]]\n    [[ end ]]\n    [[ range $k,$v := .ObjectMeta.Labels
    ]]\n    - name: ISTIO_META_[[ $k ]]\n      value: \"[[ $v ]]\"\n    [[ end ]]\n
    \   imagePullPolicy: IfNotPresent\n    [[ if (ne (annotation .ObjectMeta `status.sidecar.istio.io/port`
    \ 0 ) \"0\") ]]\n    readinessProbe:\n      httpGet:\n        path: /healthz/ready\n
    \       port: [[ annotation .ObjectMeta `status.sidecar.istio.io/port`  0  ]]\n
    \     initialDelaySeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/initialDelaySeconds`
    \ 1  ]]\n      periodSeconds: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/periodSeconds`
    \ 2  ]]\n      failureThreshold: [[ annotation .ObjectMeta `readiness.status.sidecar.istio.io/failureThreshold`
    \ 30  ]]\n    [[ end -]]securityContext:\n      \n      readOnlyRootFilesystem:
    true\n      [[ if eq (annotation .ObjectMeta `sidecar.istio.io/interceptionMode`
    .ProxyConfig.InterceptionMode) \"TPROXY\" -]]\n      capabilities:\n        add:\n
    \       - NET_ADMIN\n      runAsGroup: 1337\n      [[ else -]]\n      runAsUser:
    1337\n      [[ end -]]\n    restartPolicy: Always\n    resources:\n      [[ if
    (isset .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU`) -]]\n      requests:\n
    \       cpu: \"[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyCPU` ]]\"\n
    \       memory: \"[[ index .ObjectMeta.Annotations `sidecar.istio.io/proxyMemory`
    ]]\"\n    [[ else -]]\n      requests:\n        cpu: 10m\n      \n    [[ end -]]\n
    \   volumeMounts:\n    - mountPath: /etc/istio/proxy\n      name: istio-envoy\n
    \   - mountPath: /etc/certs/\n      name: istio-certs\n      readOnly: true\n
    \ volumes:\n  - emptyDir:\n      medium: Memory\n    name: istio-envoy\n  - name:
    istio-certs\n    secret:\n      optional: true\n      [[ if eq .Spec.ServiceAccountName
    \"\" -]]\n      secretName: istio.default\n      [[ else -]]\n      secretName:
    [[ printf \"istio.%s\" .Spec.ServiceAccountName ]]\n      [[ end -]]"
kind: ConfigMap

PSP, ClusterRole and ClusterRoleBinding shown in following sections
Expected behavior
With NONE of the inject container'ssecurityContext demanding privileged: true, a PSP with privilegeEscalationAllowed: false and CAP_NET_ADMIN enabled should satisty the requirements to successfully deploy a pod with the sidecar injected.
I have proved this by removing the namespace label istio-injecion (to disable automatic sidecar injection) and using manual sidecar injection using istioctl kube-inject -f httpbin.yaml to generate and apply the modified spec. In this case the deployment succeeds with the aforemention PSP.
Steps to reproduce the bug
With the istio-sidecar-injector configmap using the value privilege: false being used in the initContainer `securityContext, enable automatic injection in default namespace by labeling with istio-injection=enabled, as follows,
kubectl label namespace default istio-injection=enabled
Create a PSP, ClusterRole and ClusterRoleBinding as follows in the default namespace,
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: istio-psp
spec:
  allowPrivilegeEscalation: false
  allowedCapabilities:
  - NET_ADMIN
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - emptyDir
  - projected
  - secret
  - downwardAPI
  - persistentVolumeClaim
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-user
rules:
- apiGroups:
  - extensions
  resourceNames:
  - istio-psp
  resources:
  - podsecuritypolicies
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: psp-user-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp-user
subjects:
- apiGroup: ""
  kind: ServiceAccount
  name: default
  namespace: default

, and apply.
Now deploy sample/httpbin/httpbin.yaml to the default namespace.
`kubectl create -f sample/httpbin/httpbin.yaml
Version
kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-26T12:46:57Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

istioctl version
Version: 1.0.3
GitRevision: a44d4c8bcb427db16ca4a439adfbd8d9361b8ed3
User: root@0ead81bba27d
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean

Installation
Istio was installed using helm template with --set global.proxy.privileged=false among other settings.
Environment
Self managed kubernetes cluster running on AWS.
Cluster state
Can provide as needed.

@ostromart I believe you're the right person to point to for the sidecar injector, right? If not, could you redirect?

Yep, I can take a look at this.

From the first log I see:
message: 'pods "httpbin-7cf66f94c4-hzhhd" is forbidden: unable to validate against
any pod security policy: [spec.initContainers[0].securityContext.capabilities.add:
Invalid value: "NET_ADMIN": capability may not be added]'
So it seems that injection is failing in this case because you don't have NET_ADMIN, not because you need privileged. Further, the message suggests that NET_ADMIN is not a valid value - it does appear to be correct based on the pod security policies docs, but is not being accepted here.
My gut reaction is that it's just not in the correct format (string) and needs quotes around it i.e. 'NET_ADMIN'.
Give that a shot.

@ostromart I dont think I agree with that. I have verified that it works when I have a PSP with the following spec.
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
  name: aaa-relaxed-sudo
spec:
  allowPrivilegeEscalation: true
  allowedCapabilities:
  - NET_ADMIN
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - emptyDir
  - projected
  - secret
  - downwardAPI
  - persistentVolumeClaim

Its always the allowPrivilegeEscalation: true that makes it work and allowPrivilegeEscalation: false errors out with the same message stating
message: 'pods "httpbin-7cf66f94c4-hzhhd" is forbidden: unable to validate against
any pod security policy: [spec.initContainers[0].securityContext.capabilities.add:
Invalid value: "NET_ADMIN": capability may not be added]'

Since you have the repro, could you humor me and try it with the quotes?

@ostromart  Just tried editing the psp and add the quotes and it state,
kubectl edit psp istio-psp
Edit cancelled, no changes made.

Seems like it doesn't see any difference.
The weird part is, like I stated earlier, with the same PSP/clusterrole/clusterrolebinding and the istio-injection label removed from the namespace (disabling automatic injection, manual injection and deploy successfully deploys.
@ostromart Does the mutatingwebhookadmission controller (sidecar webhook) look for privileged: true and NET_ADMIN in the same checking block ? Maybe the error message is not accurate ? (just a hunch)

@ostromart Here's a snapshot of the logs, created right after kubectl create -f sleep.yaml from the istio-sidecar-injector pod in case you can find something here,
2019-02-08T20:49:54.624055Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=167b477f-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:54.625589Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:54.637807Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=167d6a57-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:54.639207Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:54.645159Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=167e8b39-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:54.646446Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:54.655066Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=16800d59-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:54.656830Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:54.702999Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=1687575d-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:54.704544Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:54.791137Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=1694c88d-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:54.792598Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:54.959477Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=16ae746f-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:54.960901Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:55.288149Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=16e0981c-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:55.289627Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:55.936281Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=17438758-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:55.937660Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:57.224611Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=18080fde-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:57.226326Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:49:59.793208Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=1990090a-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:49:59.794647Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

2019-02-08T20:50:04.927528Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=default Name= () UID=1c9f77d3-2be3-11e9-a26f-027c378b3a9e Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 9c5c868c-14ee-11e9-82aa-060f95718d52 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-08T20:50:04.929024Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"hcr.data.here.com/public_mirror/proxy_init:1.0.3","args":["-p","15001","-u","1337","-m","REDIRECT","-i","100.64.0.0/13","-x","","-b","","-d",""],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]},"privileged":false}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"hcr.data.here.com/public_mirror/proxyv2:1.0.3","args":["proxy","sidecar","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","sleep","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15005","--discoveryRefreshDelay","1s","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","MUTUAL_TLS"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_ANNOTATIONS","value":"{\"kubernetes.io/psp\":\"istio-psp\"}\n"},{"name":"ISTIO_META_app","value":"sleep"},{"name":"ISTIO_META_pod-template-hash","value":"5db5cd64c7"}],"resources":{"limits":{"cpu":"3","memory":"3Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","defaultMode":420,"optional":true}}},{"op":"add","path":"/metadata/annotations/sidecar.istio.io~1status","value":"{\"version\":\"f88596929713931612c6e225accbee8fe64857b76d8e5ff403ca7ffceffb2a5d\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}]

Not sure why it prints it 12 times ?!

I think in this case the prints are due to admission failure and retries. Normally on success there would only be one request per resource.
So I've never used PodSecurityPolicy, but my interpretation of the docs is that allowPrivilegeEscalation must be true in order to set NET_ADMIN in the container. The docs state: "This bool directly controls whether the no_new_privs flag gets set on the container process. " If you're trying to start a process in the proxy container with NET_ADMIN, it requires new privileges in the process which the flag is disallowing (the parent doesn't have NET_ADMIN). The policy admission controller recognizes this and gives the rather unintuitive error we are seeing.
The strange part to me is actually that manual injection works. I'd expect to see exactly the same error when the manually injected pod is created.

So I've never used PodSecurityPolicy, but my interpretation of the docs is that allowPrivilegeEscalation must be true in order to set NET_ADMIN in the container"

Can you point me to the docs ?

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privilege-escalation

The strange part to me is actually that manual injection works. I'd expect to see exactly the same error when the manually injected pod is created.

Yep that baffles me as well.
Do you know if the right serviceaccount gets used by the admission controller when deploying the modified spec ? I ask because the log shows, [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] and not serviceaccount: default

The log you're talking about is the PSP admission controller right?
In that case, it's something that runs out of kube-system so this would use that service account.
Do you actually see a successful admission response in the PSP controller logs for manual injection? I wonder if the controller even sees the request in this case because I'd expect the behavior to be the same as in auto-inject case.

Do you actually see a successful admission response in the PSP controller logs for manual injection?

What do you mean by PSP controller ? What pod is this ?

Oh, you're talking about an earlier log entry of a successful request from API server showing up on sidecar injector. In that case, what you're seeing is the request parameters (API server uses kube-system SA because it's part of kubernetes master).
If you log into the master node you should be able to see logs of admission requests: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/#looking-at-logs
Then, try creating a pod with auto injection and manual injection. In both cases you should see in the API server logs where the admission request is going out to the PSP controller.
If so, the question is why manual injection request works, so we'll need to look at the details of the request itself, which should show up in the PSP controller pod.
Are you using the same injection template as your istio release when doing manual injection, per instructions at:
https://istio.io/docs/setup/kubernetes/sidecar-injection/#manual-sidecar-injection
?

@ostromart I will try to get those logs later today.
Meanwhile I stumbled across this issue on the kubernetes page, https://github.com/kubernetes/kubernetes/issues/55435
Specifically,
    2\. Give the injecting controller SA access to elevated PSP permissions. The controller also needs permissions for anything the other containers use, so in practice it would be a fully privileged PSP. This also prevents the user (or any other controllers) from updating the pod, since the updated pod (which includes the injected sidecar) would be denied under the original PSP.

, which might explain why allowPrivilegeEscalation: true is required by the injecting controller.
Do you think its related ? My understand seems to link it to the problem, but I might be misunderstanding it.
WDYT ?

It's related but not quite what we're talking about here.  The way I read k8s docs, allowPrivilegeEscalation: true should be required, so to me we are getting the behavior I'd expect.
The gap is why manual injection works, and my current theory here is that you might be using the default injection template in (some older version of) istioctl rather than the one which is part of the release.

current theory here is that you might be using the default injection template in (some older version of) istioctl rather than the one which is part of the release.

Where does istioctl source it;s template from ? Is it baked into the binary ?

Yes, there's a version baked into the binary. It was probably intended to make life easier for users but it's not without its problems.

Facing similar issue here, works with istioctl but not with sidecar injector when we don't apply privileged PSP.

The gap is why manual injection works, and my current theory here is that you might be using the default injection template in (some older version of) istioctl rather than the one which is part of the release.

The istioctl version is 1.0.3 from the same upstream repo, used to deploy the sidecar-injector component.
Version: 1.0.3
GitRevision: a44d4c8bcb427db16ca4a439adfbd8d9361b8ed3
User: root@0ead81bba27d
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean

Could you check the master logs (link above) to see what the admission requests look like from auto and manual injection?

@ostromart I cant find /var/log/*.log for the k8s components on the master node (probably because of systemd), but I can acquire the logs from the individual docker containers. Is it the kube-controller-manager container or the api-server container that would provide the admission controller logs ?

I've never looked in there but it sounds like it would show up there. Try auto inject first because we know that definitely hits the PSP controller.

@ostromart
I just realized while doing an upgrade to istio-1.0.3 that I had a hack in place to use privileged from values.yaml. Istio-1.0.3 had proxy.privileged declared but not used in the sidecar-injector-configmap.yaml template.
I added
         {{ if .Values.global.proxy.privileged }}
          privileged: true
          {{ end -}}

in the security context for injected images.
The helm chart was deployed using --set proxy.privileged=false.
This might differ from the manually injected sidecar, due to the fact that it is not affected by changes to the deployed configmap.
Don't know if that help, but I just remembered so thought I would let you know.

@ostromart can you follow up on this issue?  Confirm this is P1.

1.0.3 by default has privileged: true in the inject template which is why this worked with manual injection. @nitishm if I understand correctly what you did, your hack would not affect manual injection because this doesn't use template parameters. It renders a built in manifest with some chosen parameters:
https://github.com/istio/istio/blob/98598f88f6ee9c1e6b3f03b652d8e0e3cd114fa2/istioctl/cmd/istioctl/kubeinject.go#L274
Closing for now since both auto and manual injection appear to be working as expected.
@nitishm if you believe otherwise, please reopen with the controller logs since this is the definitive source of truth for pod creation events.

I'm also encountering this issue, even when using istio-cni, which avoids needing proxy_init and thus the NET_ADMIN privilege, and with proxy.privileged: false in the Helm chart values. This is surprising to me because the securityContext doesn't seem to need privilege escalation:
    securityContext:
      procMount: Default
      readOnlyRootFilesystem: true
      runAsUser: 1337

I found issue https://github.com/kubernetes/kubernetes/issues/65716, which suggests that this could be solved if the sidecar injector added allowPrivilegeEscalation: false to istio-proxy's securityContext. I tested this out by adding that to the istio-sidecar-injector ConfigMap and restarting the sidecar injector, and it fixed this issue. Also, I added
          capabilities:
            drop:
            - ALL

to the security context, which allowed me to include requiredDropCapabilities: ["ALL"] in my PSP. Could we consider adding these to the security context in the helm chart?

Thanks for bringing this up @tbarrella, I've created https://github.com/istio/istio/issues/12231 to track it.

@tbarrella Could you post a diff somewhere showing the changes you made? Are they in sidecar-injector-configmap.yaml?

Yeah, I've since removed the change, but it should be around this line in 1.1.0

1.0.3 by default has privileged: true in the inject template which is why this worked with manual injection. @nitishm if I understand correctly what you did, your hack would not affect manual injection because this doesn't use template parameters. It renders a built in manifest with some chosen parameters:
https://github.com/istio/istio/blob/98598f88f6ee9c1e6b3f03b652d8e0e3cd114fa2/istioctl/cmd/istioctl/kubeinject.go#L274
Closing for now since both auto and manual injection appear to be working as expected.
@nitishm if you believe otherwise, please reopen with the controller logs since this is the definitive source of truth for pod creation events.

@nitishm @ostromart I ran into this too and figured it's due to some not-so intuitive way the PSP admission controller runs during mutating and validating phases. @mattmi88 has nicely explained it in this post-
https://github.com/istio/istio/issues/12231#issuecomment-479693483
And also consider the following snippet from the sidecar injector documentation here-https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/
Note that unlike manual injection, automatic injection occurs at the pod-level. You won’t see any change to the deployment itself. Instead you’ll want to check individual pods (via kubectl describe) to see the injected proxy.

So depending upon the PSPs you have configured for the given service account, manual sidecar injection works because the sidecars are added in the deployment wherein the mutating phase of the PSP controller is able to mutate the two injected sidecar containers as well in the pod spec. In case of automatic injection, these containers are not available during the mutating phase and are only seen during validation phase at which point the PSP that earlier matched and caused mutation gets discarded (as no mutation is allowed during validation). And that's why the automatic injection of sidecar fails with the same set of PSPs.

authors: harpratap hercynium munrodg neeleshkorade nitishm ostromart ozevren tbarrella

repo: istio/istio


text:
when use curl command to visit 'httpbin.org'  in pod, it shows 'Could not resolve host: httpbin.org'

This sounds like a DNS issue.
Can you run nslookup httpbin.org and nslookup google.com?
Can you curl httpbin.org from pods without a sidecar and it fails from pods in a sidecar?
Can you provide some more info?

Issue is abandoned. Please add follow up information if you are still seeing this

source: gharchive/issue

authors: howardjohn zhaofengcumt

license_type: permissive

license_source: github-api


text:
Add LDAP login option to Kiali Helm chart

(This is used to request new product features, please visit https://discuss.istio.io for questions on using Istio)
Describe the feature request
It would be good if the LDAP login option was supported by the Kiali Helm chart.
Describe alternatives you've considered
N/A
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[x] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[ ] Developer Infrastructure
Additional context
I'll create a pull request for this.

So it looks like creating a pull request on this project is slightly more involved that I had realised.
If the following block was added to install/kubernetes/helm/istio/charts/kiali/templates/configmap.yaml under the auth strategy and the Kiali LDAP documentation was added to the chart documentation. Kiali LDAP support should work.
{{- if eq .Values.dashboard.auth.strategy "ldap" }}
      ldap:
{{- with .Values.dashboard.auth.strategy.ldap }}
{{ toYaml . | indent 8 }}
{{- end }}
{{- end }}


text:
How to preserve client source IP (disable snat) at ingress gateway level (envoy) ?

Hello,
Problem description
How to configure Istio CNI and Envoy proxy at ingress level with TPROXY mode ?
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
I would like to keep the client source IP when its TCP packet leave the istio ingress GW pod
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
$ istioctl version --remote
client version: 1.4.3
citadel version: 1.4.3
citadel version: 1.4.3
citadel version: 1.4.3
galley version: 1.4.3
galley version: 1.4.3
galley version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
nodeagent version:
pilot version: 1.4.3
pilot version: 1.4.3
pilot version: 1.4.3
policy version: 1.4.3
policy version: 1.4.3
policy version: 1.4.3
sidecar-injector version: 1.4.3
sidecar-injector version: 1.4.3
sidecar-injector version: 1.4.3
telemetry version: 1.4.3
telemetry version: 1.4.3
telemetry version: 1.4.3
data plane version: 1.4.3 (29 proxies), 1.4.0 (2 proxies)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
istioctl
Istio is deployed with the following command (SDS & Istio CNI activated)
istioctl manifest generate --set profile=sds --set values.gateways.istio-egressgateway.enabled=false --set values.gateways.istio-ingressgateway.sds.enabled=true --set values.gateways.istio-ingressgateway.type=NodePort --set gateways.components.ingressGateway.k8s.hpaSpec.minReplicas=6 --set gateways.components.ingressGateway.k8s.hpaSpec.maxReplicas=8 --set values.global.mtls.enabled=false --set autoInjection.components.injector.k8s.replicaCount=3 --set configManagement.components.galley.k8s.replicaCount=3 --set trafficManagement.components.pilot.k8s.hpaSpec.minReplicas=3 --set policy.components.policy.k8s.hpaSpec.minReplicas=3 --set telemetry.components.telemetry.k8s.hpaSpec.minReplicas=3 -f istio-custom-resources.yaml > istio-sds-manifest-LoadBalancer8.yaml
istioctl manifest apply -f istio-sds-manifest-LoadBalancer8.yaml

Here the content of the istio-custom-resources.yaml  file
$ cat istio-custom-resources.yaml
apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
spec:
  cni:
    enabled: true
    components:
      cni:
        namespace: kube-system
  values:
    cni:
      excludeNamespaces:
       - default
       - istio-system
       - kube-node-lease
       - kube-public
       - kube-system
    mixer:
      telemetry:
        podAntiAffinityTermLabelSelector:
        - key: istio
          operator: In
          values: mixer
          topologyKey: failure-domain.beta.kubernetes.io/zone
          weight: 90
        - key: istio
          operator: In
          values: mixer
          topologyKey: kubernetes.io/hostname
          weight: 100
    pilot:
      podAntiAffinityTermLabelSelector:
      - key: istio
        operator: In
        values: pilot
        topologyKey: failure-domain.beta.kubernetes.io/zone
        weight: 90
      - key: istio
        operator: In
        values: pilot
        topologyKey: kubernetes.io/hostname
        weight: 100
    prometheus:
      podAntiAffinityTermLabelSelector:
      - key: app
        operator: In
        values: prometheus
        topologyKey: failure-domain.beta.kubernetes.io/zone
        weight: 90
      - key: app
        operator: In
        values: prometheus
        topologyKey: kubernetes.io/hostname
        weight: 100
    grafana:
      podAntiAffinityTermLabelSelector:
      - key: app
        operator: In
        values: grafana
        topologyKey: failure-domain.beta.kubernetes.io/zone
        weight: 90
      - key: app
        operator: In
        values: grafana
        topologyKey: kubernetes.io/hostname
        weight: 100
    gateways:
      istio-ingressgateway:
        ports:
        - name: status-port
          port: 15020
          targetPort: 15020
        - name: http2
          port: 80
          targetPort: 80
          nodePort: 30620
        - name: https
          port: 443
          nodePort: 31068
        - name: kiali
          port: 15029
          targetPort: 15029
        - name: prometheus
          port: 15030
          targetPort: 15030
        - name: grafana
          port: 15031
          targetPort: 15031
        - name: tracing
          port: 15032
          targetPort: 15032
        - name: tls
          port: 15443
          targetPort: 15443
  unvalidatedValues:
    cni:
      logLevel: info
    gateways:
      istio-ingressgateway:
        externalTrafficPolicy: Local
        podAntiAffinityTermLabelSelector:
        - key: app
          operator: In
          values: istio-ingressgateway
          topologyKey: failure-domain.beta.kubernetes.io/zone
          weight: 10
        - key: app
          operator: In
          values: istio-ingressgateway
          topologyKey: kubernetes.io/hostname
          weight: 20
        - key: app
          operator: In
          values: istio-ingressgateway
          topologyKey: accessed-from-internet
          weight: 100
  security:
    components:
      citadel:
        k8s:
          replicaCount: 3

Here is the infrastructure topology (very common).
Customer (IP 1.2.3.4) => LoadBalancer (HaProxy) => Kube Proxy (istio-ingressgateway) => Istio Ingress GW POD (envoy) => Customer POD (container + envoy sidecar)
All is working well into my k8s cluster and Istio Mesh. I'm using bookinfo application to test Istio.
Bookinfo application is reachable through HTTP and HTTPS.
My goal is to apply Istio AuthorizationPolicy based on source ipBlocks at the envoy proxy level of the customer POD. I want my customer to be able to filter the external trafic coming into their pods at the cusotmer POD level and not at the Ingress GW level.
To do that i need ingress GW Envoy proxy to preserve source IP (SNAT disable).
By default the souce IPs are natted by the envoy proxy of the Ingress GW using its own IP which is the IP of the ingress GW POD. So the packet coming into the envoy sidecar has the source IP of the Ingress GW POD and not the one of the real customer.
After many reads about the subject (tproxy mode) and certainly a misunderstanding from my side, i don't succeed in keeping the client source IP when the TCP packets goes outside the envoy proxy from the ingress gw pod.
I do not understand how to activate TPROXY mode on the Envoy proxy at the ingress GW  POD level.
Moreover, i think i need to modify Istio CNI DaemonSet to instruct it to use TPROXY mode to make iptables rules configured accordingly. How to do that with Istio CNI ?
Hope i've provided the needed information to get some help !
Thanks

I think that you can change externalTrafficPolicy: Cluster to externalTrafficPolicy: Local in istio-ingressgateway service

@luisra51  can not work in istio1.4.5 or istio1.5

@catman002 I try in 1.4.5 version and work

I think that you can change externalTrafficPolicy: Cluster to externalTrafficPolicy: Local in istio-ingressgateway service

Already done.
The externalTrafficPolicy parameter set to Local at the service level allow to keep source IP when the packet leave the ipvs proxy. So the IP is preserved when coming into the ingress GW POD. But not when leaving the Ingress GW Pod which is what i want.
Here is my "schema" annotated which illustrate where my problem is.
Customer (IP 1.2.3.4) => LoadBalancer (HaProxy) (source ip 1.2.3.4 is preserved thanks to haproxy "usesrc clientip") => Kube Proxy (istio-ingressgateway) (source ip 1.2.3.4 is preserved thanks to service externalTrafficPolicy: Local parameter) => Istio Ingress GW POD (envoy) (How to preserve the source IP here when the packet leave the Envoy Proxy ?) =>  Customer POD (container + envoy sidecar) (Source IP is lost at this level because source ip is natted by Istio Ingress GW POD)

@luisra51 can not work in istio1.4.5 or istio1.5

Are you telling me that it's impossible to preserve source IP after Istio ingress GW POD ?
How to do some ip based authorizationpolicies in that case ?
Thanks

no,  @jaxklag

With the change of externalTrafficPolicy: Local I resolved to keep the source ip.
The authorizationpolicies to restrict by ip did not work for a specific service. for the entire cluster yes

Hi @jaxklag,
Did you ever find a solution to this? I'm facing the exact same issue. Using GCP network load balancers, externalTrafficPolicy Local, DaemonSets, I see the ingress GW gets the real user's IP but just the pod IP makes it to the service pods. Any help appreciated! Thanks

Is there a solution for this issue?
I have the same issue with an azure AKS cluster. externalTrafficPolicy Local is working fine without istio. But with istio inplace the request will be always natted with the ip of the ingressgateway pod

Also curious on a resolution to this issue

Ran into this same issue, but with a bit of a different setup as i'm using apisix for the ingress gateway. Turns out i needed interceptionMode: TPROXY for this to work. I enabled this only for the ingress gateway pod(s) using the following annotation:
sidecar.istio.io/interceptionMode: TPROXY

see:

https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#ProxyConfig-InboundInterceptionMode
https://istio.io/latest/docs/reference/config/annotations/

I noticed that many of you have been inquiring about preserving client source IP in Istio. I recently wrote a blog post titled Maintaining Traffic Transparency: Preserving Client Source IP in Istio that addresses this topic in detail. I hope you find it helpful in solving your source IP preservation challenges. Feel free to check it out and let me know if you have any questions or feedback.

authors: FalconerTC brentmjohnson catman002 jaxklag jochenrichter luisra51 rootsongjc zackb


text:
istioctl install ignores log_output_level

If I set this to error I still get info logs

Fixed in https://github.com/istio/istio/pull/23315.

authors: howardjohn ostromart


text:
Donot output istiod tls key to file system

Currently, the self signed key/cert pairs are saved to /var/run/secrets/istio-dns. This is not secure and I think we can save the key/cert to k8s secret.
It seems CaCerts in ./etc/cacerts can also be avoided.

I don't think we need to write it anywhere, can't we just store it in memory? Its not shared between instances

Yeah, each istiod signs its own dns certs.

For cacerts we used in multicluster installation, it is mounted to all the istiod instances.

This is done

authors: howardjohn hzxuzhonghu


text:
Get error message in kubernetes IPv6 only cluster when executing command istioctl proxy-config route

Bug Description
I get error message when executing istioctl proxy-config route in my cluster which is kubernetes IPv6 only.
And the detail as below:
$ istioctl proxy-config route httpbin-74fb669cc6-bpm7b --name 80 -ojson
2021-09-14T06:19:58.748624Z	error	klog	an error occurred forwarding 45017 -> 15000: error forwarding port 15000 to pod df101dc6c65effd3b0904bf8d5a9b3cc124163cefac68106ce348e1247ebd647, uid : exit status 1: 2021/09/14 14:19:58 socat[27452] E connect(5, AF=2 127.0.0.1:15000, 16): Connection refused
Error: failed to execute command on httpbin-74fb669cc6-bpm7b.default sidecar: failure running port forward process: Get "http://127.0.0.1:45017/config_dump": EOF

Version
$ istioctl version
client version: 1.11.1
control plane version: ce6205d503e5c5e41af496ebbe01ece7dc6c3547
data plane version: 1.11.1 (4 proxies)
$ kubectl version --short
Client Version: v1.22.1
Server Version: v1.22.1

Additional Information
Found it when working on issue#
Affected product area

[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[X] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
[ ] Multi Cluster
[ ] Virtual Machine
[ ] Control Plane Revisions

Is this the right place to submit this?

[X] This is not a security vulnerability
[X] This is not a question about how to use Istio

@zhlsunshine Il just throw my findings here as per https://github.com/istio/istio/issues/34358.
Also I do notice that your port-forward is using socat. Its possible that your runtime doesn't have the fix I talk about.

Hi @pseudonator, I did some investigation again for this issue, as you know, Istio implement its own port-forwarder function, and it's very similar with kubectl, and it's okay for kubectl port-forward, but not works for Istio. So this makes me more confuse.

this error relates to kubernetes container runtime, so there are 2 cases:

Docker as the container runtime
According to Istioctl commands support in IPv6 only cluster user need to change the runtime from Docker to containerd
containerd as the container runtime
According to istioctl commands fail in an IPv6 only cluster #34358, user should make sure that the containerd version is higher than 1.5.0

There would be no error if follow alternative above.

authors: pseudonator zhlsunshine


text:
AuthorizationPolicy unexpectedly added to envoy outbound listener filter chain

Bug Description
We are using istio 1.14.0-beta.1 with outboundTrafficPolicy mode equals to "REGISTRY_ONLY" and therefore we create ServiceEntries to whitelist the external calls.
e.g.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: standard-port-external-services
  namespace: external
spec:
  hosts:
    - "example.net"
  exportTo:
    - "*"
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: https
      protocol: TLS
    - number: 80
      name: http
      protocol: HTTP
  resolution: NONE

And we also have an AuthorizationPolicy defined for the namespace where our pods are running:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: some-policy
spec:
  rules:
    - from:
        - source:
            namespaces: ["some-namespace"]

After upgrading from istio 1.12.0 to istio 1.14.0-beta1, all the whitelisted external calls were blocked by envoy rbac.
e.g.  (destination has been redacted)
debug	envoy rbac	checking connection: requestedServerName: *******, sourceIP: **********/51076, directRemoteIP: 1**********/:51076,remoteIP: **********/:51076, localAddress: **********/:443, ssl: none, dynamicMetadata:
debug	envoy rbac	enforced denied, matched policy none

Envoy config dump shows an unexpected network rbac EnvoyFilter for the listeners related to the whitelisted ServiceEntries.
{
     "name": "0.0.0.0_443",
     "active_state": {
      "version_info": "2022-05-26T09:15:32Z/2",
      "listener": {
       "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
       "name": "0.0.0.0_443",
       "address": {
        "socket_address": {
         "address": "0.0.0.0",
         "port_value": 443
        }
       },
       "filter_chains": [
        {
         "filter_chain_match": {
          "server_names": [
           "example.net"
          ]
         },
         "filters": [
          ...
          {
           "name": "envoy.filters.network.rbac",
           "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.network.rbac.v3.RBAC",
            "rules": {
             "policies": {
              "ns[some-namespace]-policy[some-policy]-rule[0]": {

                **** This is the unexpected AuthorizatioPolicy ****

This could be related to https://github.com/istio/istio/pull/38595. Does BuildTCP needs a check for class == networking.ListenerClassSidecarOutbound just like BuildHTTP?
Is there any other flag to control this behaviour?
Version
# istio
control plane version: 1.14.0-beta.1
data plane version: 1.14.0-beta.1 (10 proxies)

# kubernetes
Client Version: v1.22.0
Server Version: v1.21.9-eks-14c7a48

Additional Information
No response

I think this is probably the same as https://github.com/istio/istio/issues/39000 and a P0/release blocker

authors: dallasanta howardjohn


text:
[release-1.14] Fix sync issue in GCP platform

Manual cherrypick required.
#40700 failed to apply on top of branch "release-1.14":
Applying: Fix sync issue in GCP platform
Using index info to reconstruct a base tree...
M	pkg/bootstrap/platform/gcp.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/bootstrap/platform/gcp.go
CONFLICT (content): Merge conflict in pkg/bootstrap/platform/gcp.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Fix sync issue in GCP platform
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

https://github.com/istio/istio/pull/41960

authors: dhawton istio-testing


text:
Metric Pruning - Feature Promotion Tracking

Feature Stage

[ ] Feature released
[ ] Tracking feedback
[ ] Installation API (HELM) - TBD
[ ] Enabled By Default - TBD

Getting help
If you are having an issue with the feature and would like help or need to report a bug please open a separate issue and link it with this one for overarching tracking.
Calling for feedback
If you have enabled the following, your feedback would be greatly appreciated:
    METRIC_GRACEFUL_DELETION_INTERVAL: <duration>
    METRIC_ROTATION_INTERVAL: <duration>

Feedback template. Please copy this markdown into a comment and fill out if you would like to provide feedback. Please feel free to edit the comment if something changes and you would like to provide additional details.
Istio Version:
Previous Version (if upgrading):
Data plane variant (select all that apply):

  - [ ] Sidecar
  - [ ] Ambient

Cluster/Mesh Size:
Upgrade Method (select one):

  - [ ] Revisioned
  - [ ] In-place

Installation Method:

  - [ ] Helm
  - [ ] Istioctl
  - [ ] Other (please specify)

Feedback:

  - Issues prior to upgrade:
  - Issues during upgrade:
  - Issues post-upgrade:
  - Additional comments:

i donot think that ztunnel support it.

authors: ilrudie zirain


text:
[BOT PR] to update dependencies of istio

This PR will be merged automatically once checks are successful.
none

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: istio-testing
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: geeknoid
If they are not already assigned, you can assign the PR to them by writing /assign @geeknoid in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@istio-testing: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3e45e854a88d654532711e55ba27de7578fa0181
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3e45e854a88d654532711e55ba27de7578fa0181
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-k8s-tests.sh
3e45e854a88d654532711e55ba27de7578fa0181
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test istio-pilot-multicluster-e2e
/test istio-integ-k8s-tests

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
3e45e854a88d654532711e55ba27de7578fa0181
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
3e45e854a88d654532711e55ba27de7578fa0181
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-pilot-multicluster-e2e.sh
3e45e854a88d654532711e55ba27de7578fa0181
link
/test istio-pilot-multicluster-e2e

prow/istio-integ-k8s-tests.sh
3e45e854a88d654532711e55ba27de7578fa0181
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

source: gharchive/pull-request

authors: icygalz istio-testing