text:
Repeated request for authentication

Each time I launch PresenceLight it re-authenticate via the web browser.
I have tried uninstalling and reinstalling. I have installed 2.0.69.0 using the appxbundle.
Where are the credentials stored so I can clear them completely and see if I can get it to save?

hmmm odd, here is the token area. If you have issues let me know
C:\Users\username\AppData\Local\Packages\37828IsaacLevin.197278F15330A_zfcjmmeb3zeaa\LocalState\token.json

Oh I get that also - I thought that was expected on restart  (which is a pain but thought it was needed!)... I have the token file as noted but it does prompt on every restart.

I wiped that whole 37828IssacLevin directory and still get the same result.  Could our organization disallow storage of cached credentials?  I'll d/l the code later and trace the run.

All the code to do this is in SettingsService. I try to create the file where the exe is (program files) and if that fails I fall back to local storage. I can take a look and see what happens attaching to VS

2.0.75.0 - still prompts for auth on every start of the app.

I will need to wait to test as VS Attach to deployed WPF isn't working quite right with .NET 5

It might be the Azure AD permissions granted when the app is installed from the package / store.  I wonder if the cause is the conditional policy setup by our org for MFA.

@warthurton any update?

The same happens for me. However, I've installed PresenceLight via chocolatey, so I guess it's not related to the Microsoft Store.

Installed the same way. @thoemmi have you looked at the token file mentioned above?

@warthurton any update?

I'm having problems getting it to work when I compile myself.  I have been swamped lately so I haven't been able to spend as much time working on it as I wish.

@isaacrlevin Yes, there's a toke.json. However, it's a binary file, so I cannot read its content.

This is now fixed in Nightly. The token is moved to a location that PresenceLight has guaranteed write access to.

@thoemmi can you validate?

Unfortunately, 3.0.5 does not fix the issue. May I should give the Microsoft Store version a try?

3.0.13 is the one you want. Available by nightly install

I can confirm that 3.0.13 fixed the issue :+1:

Sorry for the delay. I also can confirm resolution of the error.

authors: isaacrlevin sborg-ms thoemmi warthurton

repo: isaacrlevin/PresenceLight


text:
Reduce memory usage by only creating one timer

This reduces memory usage and also speeds up the .set calls by a factor of 2.6.

Thanks, this is a very elegant improvement.  Published on 1.3.0, also went ahead and made cache.cancelTimer() a first-class public method.

source: gharchive/pull-request

authors: isaacs marcbachmann

repo: isaacs/ttlcache

license: ISC

license_type: permissive

license_source: github-api


text:
[화면개발] 확인 버튼 동작 이상함

2번째 페이지에서 확인버튼 누르면 취소됨

수정 완료

source: gharchive/issue

license: MIT


text:
Unable to set any other color other than white to denote a 0 value in a heatmap

I wish to have a heatmap with a 0 value corresponding to red and a 1 value corresponding to white in the Clustergrammer JS Module. However this seems unachievable by any parametric changes. The only option provided in Clustergrammer JS is changing the color of the positive and the negative end.
This Feature is especially needed for P-Value heat maps where a lower value denotes higher significance, hence a darker color corresponding to the 0 is preferred.

Thanks for the suggestion @cornhundred . It helped me indeed.
This issue is not pertaining to this repository. Hence I'm closing this issue here and would reopen it in the clustergrammer JS repository belonging to the MaayanLab.
Apologies.


text:
🛑 naturacert.isolucion.co is down

In d0a9903, naturacert.isolucion.co (https://naturacert.isolucion.co) was down:

HTTP code: 0
Response time: 0 ms

Resolved: naturacert.isolucion.co is back up in 54539a7 after 11 minutes.

repo: isolucionco/EstadoIsolucionHosting


text:
🛑 Cartama.isolucion.co is down

In 078d10d, Cartama.isolucion.co (https://Cartama.isolucion.co) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Cartama.isolucion.co is back up in e2ed26c after 1 hour, 16 minutes.


text:
🛑 ApiIsoDtaConectorBancoOccidente.isolucion.co is down

In 1c20c2a, ApiIsoDtaConectorBancoOccidente.isolucion.co (https://ApiIsoDtaConectorBancoOccidente.isolucion.co) was down:

HTTP code: 0
Response time: 0 ms

ok


text:
Right column index overlaps main body content.

The right column index on many pages overlaps the main body content. This is particular bad if the web browser isn't maximum on a large screen. The example below is half-screen width on 15" MBP (other half is terminal to run exercises) with a reasonably small font.
https://istio.io/docs/reference/contribute/style-guide.html.

Dup of #77.
Dave's got a fix for this coming up soon.

authors: ayj geeknoid


text:
Auto envoy injection to nginx-ingress-controller not working.

Describe the bug
As with issue #4840, I am unable to get istio to auto-inject the envoy proxy as a sidecar to my nginx-ingress-controller.  It does however inject the default-backend.
When I look in the logs for the sidecar-injector, I see:
2019-02-22T10:55:28.768549Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=ingress Name= (nginx-ingress-controller-544678595d-***** (actual name not yet known)) UID=5dc1655d-3690-11e9-bdf3-02c8c717b494 Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller c486e3e6-35ec-11e9-a1d3-02c8c717b494 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-22T10:55:28.768573Z	info	Skipping ingress/nginx-ingress-controller-544678595d-***** (actual name not yet known) due to policy check

for the default-backend I get:
2019-02-22T10:55:28.759514Z	info	AdmissionReview for Kind=/v1, Kind=Pod Namespace=ingress Name= (nginx-ingress-default-backend-56d99b86fb-***** (actual name not yet known)) UID=5dc0d3bb-3690-11e9-bdf3-02c8c717b494 Rfc6902PatchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller c486e3e6-35ec-11e9-a1d3-02c8c717b494 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
2019-02-22T10:55:28.761243Z	info	AdmissionResponse: patch=[{"op":"add","path":"/spec/initContainers","value":[{"name":"istio-init","image":"docker.io/istio/proxy_init:1.1.0-rc.0","args":["-p","15001","-u","1337","-m","REDIRECT","-i","*","-x","","-b","8080","-d","15020"],"resources":{"limits":{"cpu":"100m","memory":"50Mi"},"requests":{"cpu":"10m","memory":"10Mi"}},"imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_ADMIN"]}}}]},{"op":"add","path":"/spec/containers/-","value":{"name":"istio-proxy","image":"docker.io/istio/proxyv2:1.1.0-rc.0","args":["proxy","sidecar","--domain","$(POD_NAMESPACE).svc.cluster.local","--configPath","/etc/istio/proxy","--binaryPath","/usr/local/bin/envoy","--serviceCluster","nginx-ingress.$(POD_NAMESPACE)","--drainDuration","45s","--parentShutdownDuration","1m0s","--discoveryAddress","istio-pilot.istio-system:15010","--zipkinAddress","zipkin.istio-system:9411","--connectTimeout","10s","--proxyAdminPort","15000","--controlPlaneAuthPolicy","NONE","--statusPort","15020","--applicationPorts","8080","--concurrency","2"],"ports":[{"name":"http-envoy-prom","containerPort":15090,"protocol":"TCP"}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"INSTANCE_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}},{"name":"ISTIO_META_POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"ISTIO_META_CONFIG_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"ISTIO_META_INTERCEPTION_MODE","value":"REDIRECT"},{"name":"ISTIO_METAJSON_LABELS","value":"{\"app\":\"nginx-ingress\",\"component\":\"default-backend\",\"pod-template-hash\":\"56d99b86fb\",\"release\":\"nginx-ingress\"}\n"},{"name":"ISTIO_KUBE_APP_PROBERS","value":"{\"/app-health/nginx-ingress-default-backend/livez\":{\"path\":\"/healthz\",\"port\":8080,\"scheme\":\"HTTP\"}}"}],"resources":{"limits":{"cpu":"2","memory":"128Mi"},"requests":{"cpu":"100m","memory":"128Mi"}},"volumeMounts":[{"name":"istio-envoy","mountPath":"/etc/istio/proxy"},{"name":"istio-certs","readOnly":true,"mountPath":"/etc/certs/"},{"name":"default-token-rcs8d","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"httpGet":{"path":"/healthz/ready","port":15020},"initialDelaySeconds":1,"periodSeconds":2,"failureThreshold":30},"imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1337,"readOnlyRootFilesystem":true}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-envoy","emptyDir":{"medium":"Memory"}}},{"op":"add","path":"/spec/volumes/-","value":{"name":"istio-certs","secret":{"secretName":"istio.default","optional":true}}},{"op":"add","path":"/spec/securityContext","value":{}},{"op":"add","path":"/metadata/annotations","value":{"sidecar.istio.io/status":"{\"version\":\"08f55455e862c5191b88676351220e39413df4d226b5ac61aa20f18990316ebf\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}"}},{"op":"replace","path":"/spec/containers/0/livenessProbe/httpGet","value":{"path":"/app-health/nginx-ingress-default-backend/livez","port":15020,"scheme":"HTTP"}}]

There is obviously some policy at play here that I don't know about.  I just haven't been able to find it.
I've added
Expected behavior
I'd expect that after installing istio, and then the nginx-ingress-controller, that all the nginx-ingress-controller pods had the envoy proxy sidecar injected.
Steps to reproduce the bug

deploy cluster
install istio
install nginx-ingress-controller

Version
$ istioctl version
version.BuildInfo{Version:"1.1.0-rc.0", GitRevision:"3632361b439ae27962f0af472064d452fc63273b", User:"root", Host:"9efe8d69-34c2-11e9-b805-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-snapshot.6-26-g3632361"}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-13T23:15:13Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
$ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}

Installation
kubectl create ns istio-system

kubectl -n istio-system create secret generic kiali \
    --from-literal=username=admin \
    --from-literal=passphrase=supersecretpassword

helm upgrade --install --namespace istio-system \
    --wait \
    -f istio-1.1.0-rc.0/install/kubernetes/helm/istio-init/values.yaml \
    istio-init istio-1.1.0-rc.0/install/kubernetes/helm/istio-init

helm upgrade --install --namespace istio-system \
    --wait \
    -f istio-1.1.0-rc.0/install/kubernetes/helm/istio/values.yaml \
    istio istio-1.1.0-rc.0/install/kubernetes/helm/istio

values file for istio is:
gateways:
  enabled: true
  istio-ingressgateway:
    type: ClusterIP
    ports:
      ## You can add custom gateway ports
    - port: 80
      targetPort: 80
      name: http2
      # nodePort: 31380
    - port: 443
      name: https
      # nodePort: 31390
    # Example of a port to add. Remove if not needed
    - port: 31400
      name: tcp
      # nodePort: 31400
    ### PORTS FOR UI/metrics #####
    ## Disable if not needed
    - port: 15029
      targetPort: 15029
      name: https-kiali
    - port: 15030
      targetPort: 15030
      name: https-prometheus
    - port: 15031
      targetPort: 15031
      name: https-grafana
    - port: 15032
      targetPort: 15032
      name: https-tracing
      # This is the port where sni routing happens
    - port: 15443
      targetPort: 15443
      name: tls

sidecarInjectorWebhook:
  enabled: true
  rewriteAppHTTPProbe: true

tracing:
  enabled: true

Worth noting that I was having the same trouble with 1.0.5 before trying the 1.1.0 rc
Environment
AWS, CoreOS, Typhoon cluster
Cluster state
hand crafted istio dump attached.  does not include the k8s_resources.yaml manifest.  not entirely comfortable uploading that.
istio-dump.tar.gz

OK.  So I tried injecting into the deployment manually, and got this:
$ kubectl -n ingress get deployment nginx-ingress-controller -o yaml | istioctl kube-inject -f - | kubectl apply -f -
Skipping injection because "" has host networking enabled
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
deployment.extensions/nginx-ingress-controller configured

A little digging later, and it turns out you can't do injection on host-network installs of nginx.  Is there some way to work around that?  We set up an NLB to port forward ports 80/443, and a few others, to the cluster on those ports.  Changing to ClusterIP or Loadbalancer screws with the security groups.  In the case of nginx, where it's being used as the inbound traffic controller, surely rewriting the ip-tables isn't required (because it's edge)?
Either way.  How am I supposed to setup nginx so that it plays nicely with the NLB and Istio?

@ostromart is this something that you can help with?

It's also worth noting that CertManager is in the mix.

I've never attempted install with nginx - @costinm or @rshriram may be able to help.

I've made some progress.  I've set the ingress-gateway to just pass thru ports 80 and ports443, like so:
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-http-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-https-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 443
      name: https
      protocol: TCP
    hosts:
    - "*"
EOF

This works to the extent that istio routes traffic to nginx, so 👍 .  The problem I have now, is that the one service I have running; concourse; is only secured on the tcp ssh port 2222.  It isn't securing or even registering that it has port 8080 and ports 9391.  So whenever I go to access port 8080 from an istio-proxy container, it works; unexpectedly.  I was hoping that would fail, meaning that mtls was required.
istioctl authn tls-check concourse-web.dev-tools.svc.cluster.local

Warning! error execing into istio-pilot-7d954dccdd-vpdpw/istio-system discovery container: gc 1 @0.009s 6%: 0.007+0.68+0.65 ms clock, 0.057+0.064/0.92/0.84+5.2 ms cpu, 4->4->1 MB, 5 MB goal, 8 P
gc 2 @0.016s 9%: 0.006+0.65+0.75 ms clock, 0.051+0.22/0.80/1.4+6.0 ms cpu, 4->4->2 MB, 5 MB goal, 8 P

HOST:PORT                                          STATUS     SERVER     CLIENT     AUTHN POLICY          DESTINATION RULE
concourse-web.dev-tools.svc.cluster.local:2222     OK         mTLS       mTLS       default/dev-tools     default/dev-tools

I've clearly not configured something correctly.  Any ideas?

strangely, if I do:
istioctl authn tls-check
Warning! error execing into istio-pilot-7d954dccdd-vpdpw/istio-system discovery container: gc 1 @0.009s 5%: 0.007+0.59+0.48 ms clock, 0.056+0/0.66/1.1+3.8 ms cpu, 4->4->1 MB, 5 MB goal, 8 P
gc 2 @0.016s 8%: 0.006+0.71+0.77 ms clock, 0.051+0.26/0.91/1.1+6.2 ms cpu, 4->4->2 MB, 5 MB goal, 8 P

HOST:PORT                                                           STATUS       SERVER     CLIENT     AUTHN POLICY          DESTINATION RULE
<snip>
concourse-tsa.dev-tools.svc.cluster.local:2222                      OK           mTLS       mTLS       default/dev-tools     default/dev-tools
concourse-web.dev-tools.svc.cluster.local:2222                      OK           mTLS       mTLS       default/dev-tools     default/dev-tools
concourse-web.dev-tools.svc.cluster.local:8080                      OK           mTLS       mTLS       default/dev-tools     default/dev-tools
concourse-web.dev-tools.svc.cluster.local:9391                      OK           mTLS       mTLS       default/dev-tools     default/dev-tools
</snip>

I get all the ports.
Additionally, I have the following:
<snip>
istio-citadel.istio-system.svc.cluster.local:8060                   OK           mTLS       mTLS       default/              default/dev-tools
istio-citadel.istio-system.svc.cluster.local:15014                  OK           mTLS       mTLS       default/              default/dev-tools
istio-egressgateway.istio-system.svc.cluster.local:80               OK           mTLS       mTLS       default/              default/dev-tools
istio-egressgateway.istio-system.svc.cluster.local:443              OK           mTLS       mTLS       default/              default/dev-tools
istio-egressgateway.istio-system.svc.cluster.local:15443            OK           mTLS       mTLS       default/              default/dev-tools
istio-galley.istio-system.svc.cluster.local:443                     OK           mTLS       mTLS       default/              default/dev-tools
istio-galley.istio-system.svc.cluster.local:9901                    OK           mTLS       mTLS       default/              default/dev-tools
istio-galley.istio-system.svc.cluster.local:15014                   OK           mTLS       mTLS       default/              default/dev-tools
istio-ingressgateway.istio-system.svc.cluster.local:80              OK           mTLS       mTLS       default/              default/dev-tools
istio-ingressgateway.istio-system.svc.cluster.local:443             OK           mTLS       mTLS       default/              default/dev-tools
istio-ingressgateway.istio-system.svc.cluster.local:2222            OK           mTLS       mTLS       default/              default/dev-tools
istio-pilot.istio-system.svc.cluster.local:8080                     OK           mTLS       mTLS       default/              default/dev-tools
istio-pilot.istio-system.svc.cluster.local:15010                    OK           mTLS       mTLS       default/              default/dev-tools
istio-pilot.istio-system.svc.cluster.local:15011                    OK           mTLS       mTLS       default/              default/dev-tools
istio-pilot.istio-system.svc.cluster.local:15014                    OK           mTLS       mTLS       default/              default/dev-tools
istio-policy.istio-system.svc.cluster.local:9091                    CONFLICT     mTLS       HTTP       default/              istio-policy/istio-system
istio-policy.istio-system.svc.cluster.local:15004                   CONFLICT     mTLS       HTTP       default/              istio-policy/istio-system
istio-policy.istio-system.svc.cluster.local:15014                   CONFLICT     mTLS       HTTP       default/              istio-policy/istio-system
istio-sidecar-injector.istio-system.svc.cluster.local:443           OK           mTLS       mTLS       default/              default/dev-tools
istio-telemetry.istio-system.svc.cluster.local:9091                 CONFLICT     mTLS       HTTP       default/              istio-telemetry/istio-system
istio-telemetry.istio-system.svc.cluster.local:15004                CONFLICT     mTLS       HTTP       default/              istio-telemetry/istio-system
istio-telemetry.istio-system.svc.cluster.local:15014                CONFLICT     mTLS       HTTP       default/              istio-telemetry/istio-system
istio-telemetry.istio-system.svc.cluster.local:42422                CONFLICT     mTLS       HTTP       default/              istio-telemetry/istio-system
istiocoredns.istio-system.svc.cluster.local:53                      OK           mTLS       mTLS       default/              default/dev-tools
istiocoredns.istio-system.svc.cluster.local:53                      OK           mTLS       mTLS       default/              default/dev-tools
jaeger-agent.istio-system.svc.cluster.local:5775                    OK           mTLS       mTLS       default/              default/dev-tools
jaeger-agent.istio-system.svc.cluster.local:6831                    OK           mTLS       mTLS       default/              default/dev-tools
jaeger-agent.istio-system.svc.cluster.local:6832                    OK           mTLS       mTLS       default/              default/dev-tools
jaeger-collector.istio-system.svc.cluster.local:14267               OK           mTLS       mTLS       default/              default/dev-tools
jaeger-collector.istio-system.svc.cluster.local:14268               OK           mTLS       mTLS       default/              default/dev-tools
jaeger-query.istio-system.svc.cluster.local:16686                   OK           mTLS       mTLS       default/              default/dev-tools
kiali.istio-system.svc.cluster.local:20001                          OK           mTLS       mTLS       default/              default/dev-tools
kiam-agent.iam.svc.cluster.local:9620                               OK           mTLS       mTLS       default/              default/dev-tools
</snip>

Are those conflicts a misconfiguration?  They are that way after I turn on a system-wide mesh policy.

Solved!. Set the gateway correctly (the docs have it... https://preliminary.istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/) .

I've solved the ingress problem now.

@pms1969 Hi, can you please describe in more details how did you solve the problem?

@Demonian I basically followed the setup here: https://istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/
Set a gateway passthrough for ports 80 and 443.
# needed for ingress into a hostport daemonset for nginx.
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-http-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

# needed for ingress into a hostport daemonset for nginx.
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-https-gateway
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: PASSTHROUGH
    hosts:
    - "*"
EOF

authors: Demonian ostromart ozevren pms1969

repo: istio/istio


text:
Pilot configures proxy with wrong routes/listeners

Bug description
Pilot adds incorrect listeners/routes to envoy proxies, resulting in 503's.  This is the output of proxy-config routes on 2 identical pods created from the same deployment:
This is the proxy that received the correct config:
$ istioctl pc routes pod1.our-namespace
NOTE: This output only contains routes loaded via RDS.
NAME                                                                                VIRTUAL HOSTS
8080                                                                                4
9091                                                                                2
istio-telemetry.istio-system.svc.cluster.local:42422                                                    1
15004                                                                                2
15014                                                                                2
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1
inbound|8080|http|correct-service.our-namespace.svc.cluster.local     1

Here is the output from an incorrectly configured proxy on the other pod from the same deployment as above:
$ istioctl pc routes pod2.our-namespace
NOTE: This output only contains routes loaded via RDS.
NAME                                                           VIRTUAL HOSTS
8080                                                           4
istio-telemetry.istio-system.svc.cluster.local:42422           1
9091                                                           2
15004                                                          2
15014                                                          2
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1
inbound|8800|http|service-from-another-namespace.wrong-namespace.svc.cluster.local     1

Also notice the port number is not right either.  Restarting the second pod will result in a proxy with the same config as the first, which is correct.
Expected behavior
Pilot configures proxy with correct routes/listeners for the service.
Steps to reproduce the bug
It seems to only happen on a deployment when new pods are created and not every time.  Restarting the pods seems to yield correct listeners every time.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
$ istioctl version --remote
client version: 1.4.3
citadel version: 1.4.3
galley version: 1.4.3
galley version: 1.4.3
istio-ig-custom version:
istio-ig-custom version:
istio-ig-custom2 version:
istio-ig-custom2 version:
istio-ig-custom3 version:
istio-ig-custom3 version:
pilot version: 1.4.3
pilot version: 1.4.3
sidecar-injector version: 1.4.3
sidecar-injector version: 1.4.3
telemetry version: 1.4.3
telemetry version: 1.4.3
data plane version: 1.4.0 (1 proxies), 1.4.3 (229 proxies)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Helm 2 with Tiller
Environment where bug was observed (cloud vendor, OS, etc)
On premise, centos

We are seeing the same or simular issue randomly in our pod after upgrading from Istio 1.3.3 to 1.4.3
Steps to reproduce the bug
It seems to only happen on a deployment when new pods are created and not every time. Restarting the pods seems to yield correct listeners every time.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
$ istioctl version --remote

client version: 1.4.3
citadel version: 1.4.3
egressgateway version: 1.4.3
egressgateway version: 1.4.3
egressgateway version: 1.4.3
egressgateway version: 1.4.3
egressgateway version: 1.4.3
galley version: 1.4.3
ilbgateway version: 1.4.3
ilbgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
ingressgateway version: 1.4.3
nodeagent version:
[... nodeagent version ...]
nodeagent version:
pilot version: 1.4.3
pilot version: 1.4.3
policy version: 1.4.3
sidecar-injector version: 1.4.3
telemetry version: 1.4.3
telemetry version: 1.4.3
data plane version: 1.4.3 (838 proxies)

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.7-gke.23", GitCommit:"81c87c699557fed991e292cd328b2129c2f242a2", GitTreeState:"clean", BuildDate:"2019-11-07T19:23:23Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Helm 2 with Tiller
Environment where bug was observed (cloud vendor, OS, etc)

Google Cloud
GKE

$ gcloud container clusters list
NAME           LOCATION      MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION   NUM_NODES  STATUS
a-cluster      europe-west1  1.14.7-gke.23   x.x.x.x.x     n1-standard-4  1.14.7-gke.23  108        RUNNING

Additional information about this issue:
These are the routes in a normal pod (edited the names, I can provide the output wihtout edits via a private channel):
$ istioctl pc routes a-deployment-794f446794-zzwvm.correct-namespace
NOTE: This output only contains routes loaded via RDS.
NAME                                                                                VIRTUAL HOSTS
80                                                                                33
96                                                                                2
5556                                                                                2
8060                                                                                2
8080                                                                                10
8081                                                                                3
9090                                                                                3
9091                                                                                30
9901                                                                                2
10901                                                                                2
15004                                                                                3
15010                                                                                2
15014                                                                                7
15030                                                                                2
15031                                                                                2
jaeger-istio-collector.istio-system.svc.cluster.local:9411                                                 1
jaeger-istio-collector.istio-system.svc.cluster.local:14267                                                1
inbound|9091|http-prom-a-deployment-correct-namespace|a-deployment.correct-namespace.svc.cluster.local     1
jaeger-istio-collector.istio-system.svc.cluster.local:14268                                                1
istio-telemetry.istio-system.svc.cluster.local:42422                                                       1
inbound|9091|http-prom-a-deployment-correct-namespace|a-deployment.correct-namespace.svc.cluster.local     1
inbound|80|http-a-deployment-correct-namespace|a-deployment.correct-namespace.svc.cluster.local            1
inbound|80|http-a-deployment-correct-namespace|a-deployment.correct-namespace.svc.cluster.local            1
20001                                                                                2
                                                                                1

This are the routes from a faulty pod
istioctl pc routes a-deployment-794f446794-clvbg.correct-namespace
NOTE: This output only contains routes loaded via RDS.
NAME                                                                                VIRTUAL HOSTS
80                                                                                33
96                                                                                2
5556                                                                                2
8060                                                                                2
istio-telemetry.istio-system.svc.cluster.local:42422                                1
jaeger-istio-collector.istio-system.svc.cluster.local:14268                         1
inbound|80|http-other-deployment-wrong-namespace|other-deployment.wrong-namespace.svc.cluster.local     1
jaeger-istio-collector.istio-system.svc.cluster.local:9411                          1
jaeger-istio-collector.istio-system.svc.cluster.local:14267                         1
inbound|80|http-other-deployment-wrong-namespace|other-deployment.wrong-namespace.svc.cluster.local     1
8080                                                                                10
8081                                                                                3
9090                                                                                3
9091                                                                                30
9901                                                                                2
10901                                                                               2
15004                                                                               3
15010                                                                               2
15014                                                                               7
15030                                                                               2
15031                                                                               2
20001                                                                               2
                                                                                1

When executing an http request to a POD with the correct configuration we can see the headerx-envoy-decorator-operation set to the correct POD name:
curl -I http://a-deployment-794f446794-zzwvm.correct-namespace:8080
< server: istio-envoy
< x-envoy-decorator-operation: a-deployment.correct-namespace:80/*

If we execute the same request to a POD with a wrong configuration we see:
curl -I http://a-deployment-794f446794-clvbg.correct-namespace:8080
< server: istio-envoy
< x-envoy-decorator-operation: other-deployment.wrong-namespace:80/*

where you can notice the x-envoy-decorator-operation header indicates other-deployment.wrong-namespace:80 but we expected to see a-deployment-794f446794-clvbg.correct-namespace:80
Additionally, we have detected that the content response is correct, it is provided by a-deployment-794f446794-clvbg.correct-namespace but Policies are wrong (in our case, JWT Policy).
Kubernetes and Istio resources for a-deployment in correct-namespace:
VirtualService:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  creationTimestamp: "2019-08-07T06:15:32Z"
  generation: 3
  labels:
    app: a-deployment
    chart: base-v1.2.17
    fullapp: correct-namespace
    heritage: Tiller
    release: a-deployment-correct-namespace
    version: v1.2.17
  name: a-deployment
  namespace: correct-namespace
  resourceVersion: "281486396"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/correct-namespace/virtualservices/a-deployment
  uid: c3360aef-b8da-11e9-b4e4-42010ac50709
spec:
  gateways:
  - default/ingressgateway
  hosts:
  - correct-namespace.dev-01.my.domain.com
  - other.dev.my.domain.com
  http:
  - corsPolicy:
      allowCredentials: false
      allowHeaders:
      - origin
      - x-requested-with
      - accept
      - content-type
      - x-application-id
      - x-correlation-id
      - x-request-id
      - authorization
      - x-authorization
      - X-Customer-ID
      - X-SFID
      - X-MSISDN
      - x-front-client
      - scoring-token
      - x-target
      - cache-control
      allowOrigin:
      - '*'
      maxAge: 600s
    match:
    - uri:
        prefix: /v1/a-deployment/
    rewrite:
      uri: /
    route:
    - destination:
        host: a-deployment.correct-namespace.svc.cluster.local
        port:
          number: 80

Service:
piVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "9091"
    prometheus.io/probe: "true"
  creationTimestamp: "2019-08-07T06:15:32Z"
  labels:
    app: a-deployment
    chart: base-v1.2.17
    fullapp: correct-namespace
    heritage: Tiller
    release: a-deployment-correct-namespace
    version: v1.2.17
  name: a-deployment
  namespace: correct-namespace
  resourceVersion: "435659168"
  selfLink: /api/v1/namespaces/correct-namespace/services/a-deployment
  uid: c3314118-b8da-11e9-b4e4-42010ac50709
spec:
  clusterIP: 10.197.12.204
  ports:
  - name: http-a-deployment-correct-namespace
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: http-prom-a-deployment-correct-namespace
    port: 9091
    protocol: TCP
    targetPort: 9091
  selector:
    app: a-deployment
    release: a-deployment-correct-namespace
  sessionAffinity: None
  type: ClusterIP

Policy
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  creationTimestamp: "2019-10-21T14:11:59Z"
  generation: 1
  labels:
    app: a-deployment
    chart: base-v1.2.17
    fullapp: correct-namespace
    heritage: Tiller
    release: a-deployment-correct-namespace
    tier: backend
    version: v1.2.17
  name: a-deployment-authn
  namespace: correct-namespace
  resourceVersion: "281486392"
  selfLink: /apis/authentication.istio.io/v1alpha1/namespaces/correct-namespace/policies/a-deployment-authn
  uid: bf1eaa37-f40c-11e9-8d86-42010ac50709
spec:
  origins:
  - jwt:
      issuer: issuer1
      jwksUri: https://auth-provider-jwks.example.com/.well-known/jwks.json
      trigger_rules:
      - excluded_paths:
        - exact: /health
        - prefix: /docs
        - exact: /health
        - exact: /metrics
        included_paths:
        - prefix: /
  - jwt:
      issuer: issuer2
      jwksUri: https://auth-provider-jwks.example.com/.well-known/jwks.json
      trigger_rules:
      - excluded_paths:
        - exact: /health
        - prefix: /docs
        - exact: /health
        - exact: /metrics
        included_paths:
        - prefix: /
  principalBinding: USE_ORIGIN
  targets:
  - name: a-deployment

Kubernetes and Istio resources for other-deployment in wrong-namespace:
VirtualService
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  creationTimestamp: "2019-08-14T10:09:51Z"
  generation: 3
  labels:
    app: other-deployment
    chart: base-v1.2.17
    fullapp: wrong-namespace
    heritage: Tiller
    release: other-deployment-wrong-namespace
    version: v1.2.17
  name: other-deployment
  namespace: wrong-namespace
  resourceVersion: "294862447"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/wrong-namespace/virtualservices/other-deployment
  uid: a7b23296-be7b-11e9-bf65-42010ac5070b
spec:
  gateways:
  - default/ingressgateway
  hosts:
  - wrong-namespace.dev.my.domain.com
  - other-url.dev.my.domain.com
  http:
  - corsPolicy:
      allowCredentials: false
      allowHeaders:
      - origin
      - x-requested-with
      - accept
      - content-type
      - x-requestId
      - x-requestSessionId
      - authorization
      - cache-control
      allowOrigin:
      - '*'
      maxAge: 600s
    match:
    - uri:
        prefix: /other-deployment/
    rewrite:
      uri: /
    route:
    - destination:
        host: other-deployment.wrong-namespace.svc.cluster.local
        port:
          number: 80

Service
apiVersion: v1
kind: Service
metadata:
  annotations:
  creationTimestamp: "2019-08-14T10:09:51Z"
  labels:
    app: other-deployment
    chart: base-v1.2.17
    fullapp: wrong-namespace
    heritage: Tiller
    release: other-deployment-wrong-namespace
    version: v1.2.17
  name: other-deployment
  namespace: wrong-namespace
  resourceVersion: "435658730"
  selfLink: /api/v1/namespaces/wrong-namespace/services/other-deployment
  uid: a7afea06-be7b-11e9-bf65-42010ac5070b
spec:
  clusterIP: 10.197.10.227
  ports:
  - name: http-other-deployment-wrong-namespace
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: other-deployment
    release: other-deployment-wrong-namespace
  sessionAffinity: None
  type: ClusterIP

Policy
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  creationTimestamp: "2019-09-11T10:26:43Z"
  generation: 1
  labels:
    app: other-deployment
    chart: base-v1.2.17
    fullapp: wrong-namespace
    heritage: Tiller
    release: other-deployment-wrong-namespace
    tier: backend
    version: v1.2.17
  name: other-deployment-authn
  namespace: wrong-namespace
  resourceVersion: "294862440"
  selfLink: /apis/authentication.istio.io/v1alpha1/namespaces/wrong-namespace/policies/other-deployment-authn
  uid: a64ba2ce-d47e-11e9-bf65-42010ac5070b
spec:
  origins:
  - jwt:
      issuer: issuer3
      jwksUri: https://auth-provider-jwks.example.com/.well-known/jwks.json
      trigger_rules:
      - excluded_paths:
        - exact: /monitoring/health
        - prefix: /doc/
  principalBinding: USE_ORIGIN
  targets:
  - name: other-deployment

As for the config dump file, it is attached.
istio-config-dump-edited.json.gz
I hope this information is useful.
Thanks!

This may just be some recency bias, but this feels a lot like https://github.com/istio/istio/issues/20676. tl;dr is a pod IP gets re-used, causing pilot to generate config for the old pod instead of the new pod. So presumably, if this was the case, there was a pod with some ip 1.2.3.4 in service-from-another-namespace.wrong-namespace, then later your new pod in service correct-service.our-namespace comes up with pod ip 1.2.3.4 and gets the wrong config. I think this can only happen with pre-emptible nodes though, are you using those?
The tell-tale sign of this is if you see this pattern in the logs:
Handling event update for pod pod1-sha123 in namespace foo -> 1.2.3.4
Handling event update for pod pod1-sha123 in namespace foo -> 1.2.3.5 # same pod name, new IP
Handling event update for pod new-pod-sha345 in namespace foo -> 1.2.3.4 # re-use the old pod

@howardjohn In our case the nodes are long-living, they are on-prem and have been running for nearly a year.

Hi @howardjohn , we are indeed using preemtible nodes (and not preemtible).
I've search for "Handling event update for pod" in Stackdriver logs (whole cluster) and I couldn't find a single match. (Also tested substrings)

@pmoncadaisla if you are using pre-emptible nodes it seems likely the root cause. Thats weird it doesn't match, that log happens any time a pod is changed at all, do you not have info level logging enabled?

@howardjohn I can see this in the istio-proxy container of any POD: --proxyLogLevel="warning" and --log_output_level="default:info"
Pre-emptible notes seems to me the root cause, but, why is this happening after the upgrade from 1.3.3 to 1.4.3 and not before? We've seem this exact behaviour in 2 different Kubernetes clusters, upgraded from and to the same Istio version.

My understand was that this code path is entirely unchanged, so I am surprised it change during upgrade - very possible its a different root cause. The logs I am referring to are on pilot not the proxy.

You are right @howardjohn , I was looking for logs in a cluster were logs from pilot are excluded from ingestion at Stackdriver.
I've checked for the same one I reported before, and I can see what you described:
2020-02-05T12:10:59.408811Z info Handling event update for pod other-deployment-6d94d5f9db-8mt4d in namespace wrong-namespace -> 10.196.0.20
2020-02-05T12:12:50.781199Z info Handling event update for pod other-deploymenti-6d94d5f9db-8mt4d in namespace wrong-namespace  -> 10.196.0.9
2020-02-05T15:24:21.009595Z info Handling event update for pod a-deployment-794f446794-clvbg in namespace correct-namespace -> 10.196.0.20

In this case IP address 10.196.0.20 is being reused as you described.
I guess our options if we are affected by this issue are:

Shutdown preemtible nodes
Wait for 1.5 and expect #20676 to be included

@pmoncadaisla this will be backported to 1.4 as well. If you want, you can run the dev builds that include this commit: https://github.com/istio/istio/wiki/Dev Builds. You can see all commits at https://github.com/istio/istio/commits/release-1.4. Note that we don't validate these much, so it would be at your own risk

Hi @howardjohn , we have removed our preemptible nodes until the fix is included in a release.
We will be monitoring and we will keep you informed if we see the same behaviour after removing preemptible nodes.
If we don't have problems, then our issue is not related to @jasonwoodman-ascend 's

Its possible @jasonwoodman-ascend is the same issue without pre-emptible nodes. The symptom here is a running pod has its IP address changed. I've only seen it on node restarts, but perhaps it could happen in other cases?

@howardjohn  @pmoncadaisla I think my issue can be explained by the same root cause.  In our case, though the nodes are long living, we do restart them for patching.  I am guessing our issue started in our last patching cycle, the timing seems about right.
At this point I am waiting for the fix to be released and will re-evaluate then as I think it will most likely fix it.

I think, we have same or very similar issue, so that's why I'm not filing a new bug. In our case as well istio pickups listeners from completely different namespace (kube-system) instead of namespace where service is deployed. However, it's intermittent issue, we have to run delete-create loop for deployment/service to reproduce it. The only thing that helped as is to manually define Sidecar object and repeat all the ports defined in Service object.

This should be fixed in 1.4+ now. If you still see this issue on these versions, please reopen/open another issue. Thanks!

authors: howardjohn infa-rbliznet jasonwoodman-ascend pmoncadaisla


text:
Istio CNI plugin install with SELinux enabled not documented

Describe the feature request
Istio CNI plugin install and repair when SELinux is enabled and docker daemon is running with --selinux-enabled flag aren't documented.
install-cni container fails when it runs with SELinux enabled but SELinux requirements aren't documented to get Istio CNI plugin running without issues.
Describe alternatives you've considered
Add to Istio CNI plugin install documentation SELinux requirements to get Istio CNI plugin running without issues.
Additional context

Closing as a duplicate of https://github.com/istio/istio/issues/23605. Still valid though. Thanks!


text:
when VirtualService Status Error，istiod can not start

istio version: 1.18.3
kubernetes:1.18.10
Bug description
when VirtualService config error, istiod can not started
istiod error log:
{"level":"error","time":"2021-02-23T06:40:18.954868Z","scope":"klog","msg":"k8s.io/client-go@v0.19.3/tools/cache/reflector.go:156: Failed to watch *v1alpha3.VirtualService: failed to list *v1alpha3.VirtualService: v1alpha3.VirtualServiceList.Items: []v1alpha3.VirtualService: v1alpha3.VirtualService.v1alpha3.VirtualService.Status: unmarshalerDecoder: unknown value \"Error\" for enum istio.analysis.v1alpha1.AnalysisMessageBase_Level, error found in #10 byte of ...|l+v1\\\"\"}]}},{\"apiVer|..., bigger context ...|\"map-nginx.project-1925.svc.cluster.local+v1\\\"\"}]}},{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"ki|...[]"}

error VirtualService config
spec:
exportTo:
- .
gateways:
- gateway-122-71
hosts:
- nginx-map.gw-wso2t-sy-in.earth.xcloud.lenovo.com
http:
- match:
- uri:
prefix: /
rewrite:
uri: /
route:
- destination:
host: map-nginx.project-1925.svc.cluster.local
port:
number: 80
subset: v1
weight: 100
status:
validationMessages:
- code: IST0101
documentation_url: https://istio.io/docs/reference/config/analysis/ist0101/?ref=status-controller
level: Error
message: 'Referenced host+subset in destinationrule not found: "map-nginx.project-1925.svc.cluster.local+v1"'

[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[X] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
Expected behavior

@zhengqisong can u provide a complete well indented vs to help us understand what happened?

+1

yikes. We need to fix this. One issue - I don't know how :slightly_smiling_face:

Just ran into this exact problem when upgrading istio - had to delete the offending resources for the upgrade to work correctly

@jasonwzm do you know if its possible to discard bad configs instead of block?

I removed the strict deserialization from CLI things and expect it will be easy to fix for istiod.
@howardjohn To test any fixes, I need to be able to reproduce.  Two years ago it was easy to put bogus stuff in status using commands like kubectl patch vs bookinfo --type=json --patch='[{"op": "replace", "path": "/status", "value": {"startTime": "2019-06-05T17:52:46Z", "state": "ACTIVE"}}]'.  Kubernetes no longer allows it.  I made two guesses as to how to enable adding bogus stuff to /status and failed:
// Doesn't work, schema required
kubectl patch crd virtualservices.networking.istio.io --type='json' -p='[{"op": "remove", "path": "/spec/versions/0/schema"}]'

// Doesn't work "no change"
kubectl patch crd virtualservices.networking.istio.io --type='json' -p='[{"op": "remove", "path": "/metadata/managedFields"}]'

@therealmitchconnors
Any idea what protects /status?  I know it is something in the API server, because if I do --dry-run=client I can set the status, but if I do --dry-run=server the status does not get patched.

You might need a raw api call. when you do patch it probably sends a PUT
call to /crds/virtualservices/foo or similar - we need it to go to
/crds/virtualservices/foo/status. You might be able to add -v9 to the
kubectl command to get the API its calling, add /status, then call it with
kubectl patch --raw?
On Thu, Mar 18, 2021 at 7:09 AM Ed Snible @.***> wrote:

I removed the strict deserialization from CLI things and expect it will be
easy to fix for istiod.
@howardjohn https://github.com/howardjohn To test any fixes, I need to
be able to reproduce. Two years ago it was easy to put bogus stuff in
status using commands like kubectl patch vs bookinfo --type=json
--patch='[{"op": "replace", "path": "/status", "value": {"startTime":
"2019-06-05T17:52:46Z", "state": "ACTIVE"}}]'. Kubernetes no longer
allows it. I made two guesses as to how to enable adding bogus stuff to
/status and failed:
// Doesn't work, schema required
kubectl patch crd virtualservices.networking.istio.io --type='json' -p='[{"op": "remove", "path": "/spec/versions/0/schema"}]'
// Doesn't work "no change"
kubectl patch crd virtualservices.networking.istio.io --type='json' -p='[{"op": "remove", "path": "/metadata/managedFields"}]'
@therealmitchconnors https://github.com/therealmitchconnors
Any idea what protects /status? I know it is something in the API server,
because if I do --dry-run=client I can set the status, but if I do
--dry-run=server the status does not get patched.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/31014#issuecomment-801959404, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXN2G2H4NTGOUM4MSU3TEICTNANCNFSM4YB6FBNQ
.

There is no patch --raw in kubectl (I checked 1.19 and 1.20).
I tried
MYAPISERVER=c7.us-south.containers.cloud.ibm.com:28514
echo '{"status": "banana"}' | kubectl create --raw https://${MYAPISERVER}/apis/networking.istio.io/v1beta1/namespaces/default/virtualservices/httpbin/status -f -

... but it fails with 'create is not supported on resources of kind "virtualservices.networking.istio.io"'

https://github.com/kubernetes/kubectl/issues/564
On Thu, Mar 18, 2021 at 10:21 AM Ed Snible @.***> wrote:

There is no patch --raw in kubectl (I checked 1.19 and 1.20).
I tried
MYAPISERVER:28514=c7.us-south.containers.cloud.ibm.com
echo '{"status": "banana"}' | kubectl create --raw https://${MYAPISERVER}/apis/networking.istio.io/v1beta1/namespaces/default/virtualservices/httpbin/status -f -
... but it fails with 'create is not supported on resources of kind "
virtualservices.networking.istio.io"'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/31014#issuecomment-802141282, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXNETDAT275QKNDCPB3TEIZB7ANCNFSM4YB6FBNQ
.

It seems to be an issue when analysis is enabled during startup and the unmarshaller encountered an issue. I vaguely remember that we changed the format of message level we put in status in CRDs?

To manually put garbage into status, first install Istio with WITHOUT values.global.istiod.enableAnalysis=true and WITHOUT values.pilot.env.PILOT_ENABLE_STATUS=true.  Then
kubectl proxy &
curl -k -s -X PATCH -H "Accept: application/json, */*" \
-H "Content-Type: application/merge-patch+json" \
127.0.0.1:8001/apis/networking.istio.io/v1beta1/namespaces/default/virtualservices/httpbin/status \
--data '{"status":{"monkey":"banana"}}'

At this point, for master, I have no problem restarting or re-installing Istio with or without analysis.  It is possible this was fixed in an earlier version.  Checking...

I have a backtrace.  istio.io/istio is not in the path.  istio.io/api is generated code.  Not sure what to do.
A user with authority to do kubectl proxy and modify any VirtualService can use the technique in https://github.com/istio/istio/issues/31014#issuecomment-802810868 to cause trouble for the control plane.
istio.io/api/meta/v1alpha1.(*IstioStatus).UnmarshalJSON(0xc000df93c8, 0xc002c300f0, 0x45, 0x50, 0xc000df93c8, 0x6a48b38)
	/Users/snible/go/pkg/mod/istio.io/api@v0.0.0-20210318104759-fbefbc937cef/meta/v1alpha1/status_json.gen.go:35 +0x18d
github.com/json-iterator/go.(*unmarshalerDecoder).Decode(0xc001f6dc40, 0xc002abf1b0, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_marshaler.go:200 +0xdb
github.com/json-iterator/go.(*referenceDecoder).Decode(0xc001f6dc50, 0xc000df93c8, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_optional.go:128 +0x68
github.com/json-iterator/go.(*structFieldDecoder).Decode(0xc001f958e0, 0xc000df9200, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_struct_decoder.go:1054 +0x78
github.com/json-iterator/go.(*fiveFieldsStructDecoder).Decode(0xc000b38e40, 0xc000df9200, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_struct_decoder.go:739 +0x311
github.com/json-iterator/go.(*sliceDecoder).doDecode(0xc00106bd70, 0xc0002547c8, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_slice.go:86 +0xdd
github.com/json-iterator/go.(*sliceDecoder).Decode(0xc00106bd70, 0xc0002547c8, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_slice.go:60 +0x45
github.com/json-iterator/go.(*structFieldDecoder).Decode(0xc001f95c20, 0xc000254770, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_struct_decoder.go:1054 +0x78
github.com/json-iterator/go.(*fourFieldsStructDecoder).Decode(0xc0010587d0, 0xc000254770, 0xc000f03050)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_struct_decoder.go:697 +0xbf
github.com/json-iterator/go.(*Iterator).ReadVal(0xc000f03050, 0x3ae97c0, 0xc000254770)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect.go:79 +0xc2
github.com/json-iterator/go.(*frozenConfig).Unmarshal(0xc000374c80, 0xc002b57000, 0xc49, 0x1000, 0x3ae97c0, 0xc000254770, 0x0, 0x0)
	/Users/snible/go/pkg/mod/github.com/json-iterator/go@v1.1.10/config.go:348 +0xb7
k8s.io/apimachinery/pkg/runtime/serializer/json.(*Serializer).Decode(0xc000226550, 0xc002b57000, 0xc49, 0x1000, 0x0, 0x4033d98, 0xc000254770, 0xc00d5a63302ad750, 0xc059add2, 0x5cdda00, ...)
	/Users/snible/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/runtime/serializer/json/json.go:264 +0x5be
k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode(0x3ffcc00, 0xc000226550, 0xc002b57000, 0xc49, 0x1000, 0x0, 0x4033d98, 0xc000254770, 0xc002c24160, 0xc000bc5770, ...)
	/Users/snible/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/runtime/helper.go:252 +0x97
k8s.io/client-go/rest.Result.Into(0xc002b57000, 0xc49, 0x1000, 0x0, 0x0, 0x0, 0xc001337c90, 0x10, 0x0, 0x0, ...)
	/Users/snible/go/pkg/mod/k8s.io/client-go@v0.20.4/rest/request.go:1273 +0xb4
istio.io/client-go/pkg/clientset/versioned/typed/networking/v1alpha3.(*virtualServices).List(0xc002c181c0, 0x407ce28, 0xc000076098, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/snible/go/pkg/mod/istio.io/client-go@v0.0.0-20210218000043-b598dd019200/pkg/clientset/versioned/typed/networking/v1alpha3/virtualservice.gen.go:93 +0x29d
istio.io/client-go/pkg/informers/externalversions/networking/v1alpha3.NewFilteredVirtualServiceInformer.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3c28a4f, ...)
	/Users/snible/go/pkg/mod/istio.io/client-go@v0.0.0-20210218000043-b598dd019200/pkg/informers/externalversions/networking/v1alpha3/virtualservice.gen.go:63 +0x1bc
k8s.io/client-go/tools/cache.(*ListWatch).List(0xc000e16888, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/snible/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/listwatch.go:106 +0x75
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1.2(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x3c28a4f, ...)
	/Users/snible/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:283 +0x75
k8s.io/client-go/tools/pager.SimplePageFunc.func1(0x407ce28, 0xc000076090, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/snible/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/pager/pager.go:40 +0x75
k8s.io/client-go/tools/pager.(*ListPager).List(0xc002c39e60, 0x407ce28, 0xc000076090, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/snible/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/pager/pager.go:91 +0x175
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1.1(0xc002b7f4a0, 0xc000dfe1c0, 0xc002260870, 0xc002c06860, 0xc002c1a009, 0xc002c06870, 0xc001b83bc0)
	/Users/snible/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:309 +0x1e5
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1
	/Users/snible/go/pkg/mod/k8s.io/client-go@v0.20.4/tools/cache/reflector.go:271 +0x298

I am test in istio 1.6.8,1.6.13 1.7.4 ,1.7.6,1.8.3
public-system(the namespace publish service to outward)  has  gateway, virtualservice(exportTo: .)
project-1925(the namespace publish service to namespace`s service) has service, deploy, destinationrule, virtualservice(exportTo: .)
In <=1.7.4 public-system`s   virtualservice has under error, but istod can restart
validationMessages:
- code: IST0101
documentation_url: https://istio.io/docs/reference/config/analysis/ist0101/?ref=status-controller
level: Error
message: 'Referenced host+subset in destinationrule not found: "map-nginx.project-1925.svc.cluster.local+v1"'

in 1.7.6 public-system`s   virtualservice no error
upgrade from 1.7.4 to 1.8.3, istiod(1.8.3) canot not start
upgrade from 1.7.4 to 1.7.6, the public-system`s   virtualservice  error remain, but istiod(1.7.6) can started, and service can be connected.
upgrade from 1.7.4 to 1.7.6, then upgrade from 1.7.6 to 1.8.3, the public-system`s   virtualservice  error remain, but istiod(1.8.3) can not started,
{"level":"error","time":"2021-03-22T10:39:08.252656Z","scope":"klog","msg":"k8s.io/client-go@v0.19.3/tools/cache/reflector.go:156: Failed to watch *v1alpha3.VirtualService: failed to list
*v1alpha3.VirtualService: v1alpha3.VirtualServiceList.Items: []v1alpha3.VirtualService: v1alpha3.VirtualService.v1alpha3.VirtualService.Status: unmarshalerDecoder: unknown value \"Error\"
for enum istio.analysis.v1alpha1.AnalysisMessageBase_Level, error found in #10 byte of ...|l+v1\\\"\"}]}},{\"apiVer|..., bigger context ...|esh-b.project-1896.svc.frankfurtinp.local+v1\\\"
\"}]}},{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"ki|...[]"}

install 1.7.6, then upgrade  upgrade from 1.7.6 to 1.8.3 the public-system`s   virtualservice  no error, istiod(1.8.3) can started
I think This issues  has two bug:

vs check has a bug, destination no support other namespaces destinationrule .

vs in public-system
http:
  - match:
    - uri:
        prefix: /
    rewrite:
      uri: /
    route:
    - destination:
        host: map-nginx.project-1896.svc.cluster.local
        port:
          number: 80
        subset: v1
      weight: 10

istiod(1.8.3) has a bug for parse VirtualService that has Error(level: Error)

status:
    validationMessages:
    - code: IST0101
      documentation_url: https://istio.io/docs/reference/config/analysis/ist0101/?ref=status-controller
      level: Error      #this line  parse Error
      message: 'Referenced host+subset in destinationrule not found: "map-nginx.project-1896.svc.cluster.local+v1"

@zhengqisong Sorry for the late reply. Thank you for providing many details on reproducing the issue.
This is caused by https://github.com/istio/istio/pull/25960 where we were updating the validation output message to align with our status API. This changed the API output from 1.7 to 1.8 and therefore you see the unmarshalling error during istiod startup.
We are working on a refactoring of analysis and hoping to move it (including the API) out of alpha before 1.11. Before that, the upgrade experience may not be ideal when you have analysis enabled.
Does the current behavior cause additional trouble using Istio?

Jason, I think the problem extends beyond that? There is a potential DOS
vector by generating invalid configuration in etcd/api-server. istiod
should be resilient to bad configs
On Mon, Mar 29, 2021 at 10:25 AM Jason Wang @.***>
wrote:

@zhengqisong https://github.com/zhengqisong Sorry for the late reply.
Thank you for providing many details on reproducing the issue.
This is caused by #25960 https://github.com/istio/istio/pull/25960
where we were updating the validation output message to align with our
status API. This changed the API output from 1.7 to 1.8 and therefore you
see the unmarshalling error during istiod startup.
We are working on a refactoring of analysis and hoping to move it
(including the API) out of alpha before 1.11. Before that, the upgrade
experience may not be ideal when you have analysis enabled.
Does the current behavior cause additional trouble using Istio?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/31014#issuecomment-809562694, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXL32XQZO3XGSY2KF3TTGCZWZANCNFSM4YB6FBNQ
.

@howardjohn Yes, that is the bigger concern here. This applies to any istio config that can be patched by users in etcd, though patching etcd is a highly privileged action. We should look into a more general way of handling that.
In this particular case, it is our code that output differently than expected in an enum field.

I modify etcd data success upgrade istiod

first get virtualservice-63-66  json in etcd

ETCDCTL_API=3 ./etcdctl --endpoints="https://127.0.0.1:2379"  put /registry/networking.istio.io/virtualservices/mesh-system/virtualservice-63-66

scal deploy istiod --replicas=0

update virtualservice-63-66 drop status info in etcd

ETCDCTL_API=3 ./etcdctl --endpoints="https://127.0.0.1:2379" put /registry/networking.istio.io/virtualservices/mesh-system/virtualservice-152-75  '{"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"creationTimestamp":"2021-04-06T03:28:39Z","generation":1,"labels":{"application":"applicatiion-4089","project":"project-1927"},"managedFields":[{"apiVersion":"networking.istio.io/v1alpha3","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:application":{},"f:project":{}}},"f:spec":{".":{},"f:exportTo":{},"f:gateways":{},"f:hosts":{},"f:http":{}}},"manager":"okhttp","operation":"Update","time":"2021-04-06T03:28:39Z"},{"apiVersion":"networking.istio.io/v1alpha3","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:validationMessages":{}}},"manager":"pilot-discovery","operation":"Update","time":"2021-04-06T03:28:39Z"}],"name":"virtualservice-152-75","namespace":"mesh-system","uid":"899e3b64-fe55-4b4f-825b-d5e058842281"},"spec":{"exportTo":["."],"gateways":["gateway-152-75"],"hosts":["sdf4gdsfg.demo.com"],"http":[{"match":[{"uri":{"prefix":"/"}}],"rewrite":{"uri":"/"},"route":[{"destination":{"host":"mesh-01-nginx.project-1927.svc.cluster.local","port":{"number":80},"subset":"v1"},"weight":100}]}]}}'

scal deploy istiod --replicas=2
istiod only started new version

Hi everyone! First at all, thanks for your work and contribution!
I ran with the same problem upgrading Istio control plane, from v1.7.4 to v1.8.5 and from v1.6.14 to v1.8.5.
We have a lot of Istio's resources so we cannot patch them all to remove the status. There is any workaround to fix this?
Also the problem is not only in VirtualService CRD, seems the Sidecar is affected too:
{"level":"error","time":"2021-05-04T16:03:13.205337Z","scope":"klog","msg":"k8s.io/client-go@v0.19.3/tools/cache/reflector.go:156: Failed to watch *v1alpha3.Sidecar: failed to list *v1alpha3.Sidecar: v1alpha3.SidecarList.Items: []v1alpha3.Sidecar: v1alpha3.Sidecar.v1alpha3.Sidecar.Status: unmarshalerDecoder: unknown value \"Error\" for enum istio.analysis.v1alpha1.AnalysisMessageBase_Level, error found in #10 byte of ...|avior.\"}]}},{\"apiVer|..., bigger context ...|elector, which can lead to undefined behavior.\"}]}},{\"apiVersion\":\"networking.istio.io/v1alpha3\",\"ki|...[]"}
Thanks in advance, regards!

authors: esnible howardjohn hzxuzhonghu jasonwzm psanzm samirmajen xichengliudui zhengqisong


text:
service with externalname overrides istio-proxy 443 rules

Bug Description
I have some services without selector pointing to endpoints for external services to the cluster in order to monitor them.
At the moment I create the following object:
apiVersion: v1
kind: Service
metadata:
  name: nexus-metrics-monitoring
  namespace: external-monitoring
  labels:
    app: nexus-metrics-monitoring
spec:
  externalName: nexus.internal
  ports:
  - name: nexus
    port: 4443
    protocol: TCP
    targetPort: 443
  - name: node
    port: 9100
    protocol: TCP
    targetPort: 9100
  - name: containers
    port: 8080
    protocol: TCP
    targetPort: 8080
  type: ExternalName

The TLS connections originated from inside the cluster in a pod with the istio-proxy sidecar starts to fail returning me 404 errors.
This is the istioctl proxy-config listeners toolkit-68754d4d6b-d7hm5 | grep 443  output:
0.0.0.0        443   Trans: raw_buffer; App: HTTP                                             Route: 443
0.0.0.0        443   ALL                                                                      Cluster: outbound|443||nexus-metrics-monitoring.external-monitoring.svc.cluster.local
10.100.107.185 443   ALL                                                                      Cluster: outbound|443||istiod.istio-system.svc.cluster.local
10.107.150.172 443   ALL                                                                      Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
10.107.212.31  443   ALL                                                                      Cluster: outbound|443||keycloak.keycloak.svc.cluster.local
10.108.30.165  443   ALL                                                                      Cluster: outbound|443||istio-egressgateway.istio-system.svc.cluster.local
10.96.0.1      443   ALL                                                                      Cluster: outbound|443||kubernetes.default.svc.cluster.local
10.97.216.209  443   ALL                                                                      Cluster: outbound|443||step-certificates.step-certificates.svc.cluster.local
10.97.92.163   443   ALL                                                                      Cluster: outbound|443||monitoring-operator.monitoring.svc.cluster.local

And this is the output of the same command when the service is deleted and all works fine:
0.0.0.0        443   ALL                                                                      PassthroughCluster
10.100.107.185 443   ALL                                                                      Cluster: outbound|443||istiod.istio-system.svc.cluster.local
10.107.150.172 443   ALL                                                                      Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
10.107.212.31  443   ALL                                                                      Cluster: outbound|443||keycloak.keycloak.svc.cluster.local
10.108.30.165  443   ALL                                                                      Cluster: outbound|443||istio-egressgateway.istio-system.svc.cluster.local
10.96.0.1      443   ALL                                                                      Cluster: outbound|443||kubernetes.default.svc.cluster.local
10.97.216.209  443   ALL                                                                      Cluster: outbound|443||step-certificates.step-certificates.svc.cluster.local
10.97.92.163   443   ALL                                                                      Cluster: outbound|443||monitoring-operator.monitoring.svc.cluster.local

Version
istioctl version --remote
client version: 1.10.0
control plane version: 1.10.0
data plane version: 1.10.0 (47 proxies)

kubectl version --short
Client Version: v1.22.1
Server Version: v1.22.1

helm version --short
v3.4.1+gc4e7485

Additional Information
No response

See https://istio.io/latest/docs/ops/configuration/traffic-management/traffic-routing/#externalname-services you can name your 443 port with https prefix (https-nexus) to use SNI based routing.

@bianpengyuan it worked thank you.

One more thing, shouldn't this be awared somehow?
I mean, when you run "istioctl analyze --all-namespaces" this errors are being shown as an INFO [IST0118], but exactly this case an https port changed completely the behavior of all the connections from inside.

Yeah good point. It is an INFO because we have protocol sniffing and it is not necessary to name port. But in the case of externalname service, unnamed port can be disruptive. We probably should add a bit more warn message when service is externalname type.

@bianpengyuan I'd like to work on this issue. Can you assign it to me please?

@eliavem done. thanks for taking this!

Hi @bianpengyuan, I can see there are two possibilities for making users aware about this issue.

We can extend the diag.MessageType for message "PortNameIsNotUnderNamingConvention" - Info [IST0118]. We can add something like "Warning: unnamed port can affect [block] routing to external services for ExternalName service type"

If we want to display this warning only when we have an ExternalName service type, then we could create a separate diag.MessageType of type WARNING. In PortNameAnalyzer, we can add a check for ExternalName service type and display this message instead of [IST0118].

I would like to understand if your suggestion above - "We probably should add a bit more warn message when service is externalname type" can be covered with point #1

Hi @eliavem thank you for picking up this WI.  I'd vote for only display this warning (perhaps even an error?) for service that has ExternalName.  Because in normal services (not having ExternalName), it should be fine without causing users any issue with auto protocol detection.

I agree we should create a separated warning message for service of externalName.

@bianpengyuan I close this as you already fixed and merged it.
Thank you.

authors: bianpengyuan eliavem linsun ricosega


text:
Tracing: Istio HTTP->gRPC span doesn't appear in trace.

Inspired by investigations done in https://github.com/istio/istio/issues/11696
Environment:
IstioGateway->Service-A(http)->Service-B(gRPC)
Ports in services are named properly.
Service A Takes headers from incoming HTTP request and adds them to the outdoing gRPC requests to Service B:
		XRequestID:      requestID,
		XB3TraceID:      getHeader(xB3TraceID),
		XB3SpanID:       getHeader(xB3SpanID),
		XB3ParentSpanID: getHeader(xB3ParentSpanID),
		XB3Sampled:      getHeader(xB3Sampled),
		XB3Flag:         getHeader(xB3Flags),
		B3:              getHeader(b3),

In Jaeger, I only see the trace from IstioGateway to Service A but no span for ServiceA to ServiceB.
Proxy log format is:
accessLogFormat: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-B3-SAMPLED)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(X-B3-TRACEID)%\" \"%REQ(X-B3-SPANID)%\" \"%REQ(X-B3-PARENTSPANID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"

The logs from the proxies:
InressGateway:
[2021-12-07T21:26:50.090Z] "POST /init HTTP/1.1" 200 - via_upstream - "-" 828 755 47 47 "78.133.70.82, 34.102.217.248,35.191.10.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" "1" "303cef67-e3cb-9ff9-80aa-614c19d502b3" "b8b2896792ea92498a84596ac59bb845" "8a84596ac59bb845" "-" "gs1-gs-x.domain.tld" "10.100.32.24:8005" outbound|80||gs1-api-game-server.d01.svc.cluster.local 10.100.37.6:46938 10.100.37.6:80 35.191.10.246:49843 - play-for-real

ServiceA Incoming
[2021-12-07T21:26:50.091Z] "POST /init HTTP/1.1" 200 - via_upstream - "-" 828 755 46 45 "78.133.70.82, 34.102.217.248,35.191.10.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" "1" "303cef67-e3cb-9ff9-80aa-614c19d502b3" "b8b2896792ea92498a84596ac59bb845" "7845ffbbf6266607" "8a84596ac59bb845" "gs1-gs-x.domain.tld" "10.100.32.24:8005" inbound|8005|| 127.0.0.6:52327 10.100.32.24:8005 35.191.10.246:0 outbound_.80_._.gs1-api-game-server.d01.svc.cluster.local default

ServiceA Outgoing
[2021-12-07T21:26:50.095Z] "POST /core_service_v1.CoreService/GetBalance HTTP/2" 200 - via_upstream - "-" 55 17 18 18 "-" "grpc-go/1.36.1" "1" "303cef67-e3cb-9ff9-80aa-614c19d502b3" "b8b2896792ea92498a84596ac59bb845" "7845ffbbf6266607" "8a84596ac59bb845" "gs1-api-core:50101" "10.100.37.2:8007" outbound|50101||gs1-api-core.d01.svc.cluster.local 10.100.32.24:49578 10.100.66.69:50101 10.100.32.24:45616 - default

ServiceB Incoming
[2021-12-07T21:26:50.096Z] "POST /core_service_v1.CoreService/GetBalance HTTP/2" 200 - via_upstream - "-" 55 17 15 14 "-" "grpc-go/1.36.1" "1" "303cef67-e3cb-9ff9-80aa-614c19d502b3" "b8b2896792ea92498a84596ac59bb845" "7845ffbbf6266607" "8a84596ac59bb845" "gs1-api-core:50101" "10.100.37.2:8007" inbound|8007|| 127.0.0.6:53567 10.100.37.2:8007 10.100.32.24:49578 outbound_.50101_._.gs1-api-core.d01.svc.cluster.local default

Based on that proxy logs it seems that spainId and parrentSpanId are not replaced by the envoy and propagated "as is" down to the ServiceB.
As an alternative experiment I called Service B via grpcCurl from the pod of ServiceA:
./grpcurl -rpc-header 'x-b3-sampled: 1' -rpc-header 'x-b3-traceid: ebf7561051ac6433952010cd9cac40d3' -rpc-header 'x-b3-parentspanid: 297e3236182ca725' -rpc-header 'x-b3-spanid: 952010cd9cacdddd' -rpc-header 'x-request-id: a48ace99-0d2f-9fa9-9aad-83d5fe03c579' -plaintext -proto core_service_v1.proto gs1-api-core:50101 core_service_v1.CoreService/GetBalance
ServiceA outgoing:
[2021-12-07T21:37:57.691Z] "POST /core_service_v1.CoreService/GetBalance HTTP/2" 200 - via_upstream - "-" 63 17 29 29 "-" "grpcurl/v1.8.5 grpc-go/1.37.0" "1" "a48ace99-0d2f-9fa9-9aad-83d5fe03c579" "ebf7561051ac6433952010cd9cac40d3" "7fc4a980a41d3edf" "952010cd9cacdddd" "gs1-api-core:50101" "10.100.37.2:8007" outbound|50101||gs1-api-core.d01.svc.cluster.local 10.100.32.24:54452 10.100.66.69:50101 10.100.32.24:38614 - default

ServiceB Incoming:
[2021-12-07T21:37:57.692Z] "POST /core_service_v1.CoreService/GetBalance HTTP/2" 200 - via_upstream - "-" 63 17 28 27 "-" "grpcurl/v1.8.5 grpc-go/1.37.0" "1" "a48ace99-0d2f-9fa9-9aad-83d5fe03c579" "ebf7561051ac6433952010cd9cac40d3" "cbc05a179e71686a" "7fc4a980a41d3edf" "gs1-api-core:50101" "10.100.37.2:8007" inbound|8007|| 127.0.0.6:42243 10.100.37.2:8007 10.100.32.24:54452 outbound_.50101_._.gs1-api-core.d01.svc.cluster.local default

As you can see, in this case, spans were replaced properly.
Debug log with grpc metadata from the application ServiceB when called by ServiceA:
2021/12/07 21:57:25 gRPC server metadata x-b3-flags:[];x-forwarded-proto:[http];:authority:[gs1-api-core:50101];user-agent:[grpc-go/1.36.1];x-b3-traceid:[3ffc9fb9cb04232b3a0ec68e6cc286ff];x-b3-spanid:[ffa5bc3f75abc782];x-b3-parentspanid:[3a0ec68e6cc286ff];x-forwarded-client-cert:[By=spiffe://cluster.local/ns/d01/sa/gs1-api-core;Hash=c0306f7205d9ac84e9e5b312c942b9a3465da206292ff0c7c6a4f635ce15ae91;Subject="";URI=spiffe://cluster.local/ns/d01/sa/gs1-api-game-server];content-type:[application/grpc];x-request-id:[d9d13ed6-bb8b-9123-a75a-fe0e456a350c];x-b3-sampled:[1];b3:[];x-envoy-attempt-count:[1];

Debug log with grpc metadata from the application ServiceB when called by GRPCcurl:
2021/12/07 21:58:40 gRPC server metadata content-type:[application/grpc];x-envoy-attempt-count:[1];x-b3-traceid:[ebf7561051ac6433952010cd9cac40d3];x-b3-parentspanid:[c846478c2b88b686];x-b3-sampled:[1];:authority:[gs1-api-core:50101];user-agent:[grpcurl/v1.8.5 grpc-go/1.37.0];x-request-id:[a48ace99-0d2f-9fa9-9aad-83d5fe03c579];x-forwarded-proto:[http];x-forwarded-client-cert:[By=spiffe://cluster.local/ns/d01/sa/gs1-api-core;Hash=c0306f7205d9ac84e9e5b312c942b9a3465da206292ff0c7c6a4f635ce15ae91;Subject="";URI=spiffe://cluster.local/ns/d01/sa/gs1-api-game-server];x-b3-spanid:[817e1c7f84e7f9ec];

Istio installed via helm version 1.12.0 with the following settings:
global:
  tracer:
    zipkin:
      address: jaeger-collector.istio-system:9411
meshConfig:
  accessLogFile: /dev/stdout
  accessLogFormat: |
    [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-B3-SAMPLED)%" "%REQ(X-REQUEST-ID)%" "%REQ(X-B3-TRACEID)%" "%REQ(X-B3-SPANID)%" "%REQ(X-B3-PARENTSPANID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%
pilot:
  traceSampling: 100

docker.io/istio/proxyv2:1.12.0
docker.io/istio/pilot:1.12.0
jaegertracing/all-in-one:1.28.0
At this point, I am lost and I am not sure why istio is not taking care of gRPC part of the trace and don't replace headers.

This looks weird.. Can you enable service B proxy trace log with istioctl proxy-config log POD --level=trace, Capture the log of the request from ingress and grpc curl directly from service A. Want to compare what is difference between those two request, which might attribute to why service B proxy does not initiate a span.

@bianpengyuan  thanks for a quick response. Here we go:
Ingress Log:
[2021-12-08T08:53:58.386Z] "POST /init HTTP/1.1" 200 - via_upstream - "-" 828 754 4185 97 "78.133.70.82, 34.102.217.248,35.191.16.180" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" "1" "b5c688cd-0846-9cab-9432-fe6704aa7a3c" "29639b27a30880e5ff53863e6d6b6bab" "ff53863e6d6b6bab" "-" "gs1-gs-x.domain.tld" "10.100.33.12:8005" outbound|80||gs1-api-game-server.d01.svc.cluster.local 10.100.36.3:59276 10.100.36.3:80 35.191.16.180:51409 - play-for-real

ServiceA:
[2021-12-08T08:51:12.838Z] "POST /init HTTP/1.1" 200 - via_upstream - "-" 828 754 708 694 "78.133.70.82, 34.102.217.248,35.191.18.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36" "1" "86dbe583-64d9-982f-abdf-90ea5b0411fa" "e70ae8c0491b8e46cb7909086c8a732a" "369bafcd44cd96f7" "cb7909086c8a732a" "gs1-gs-x.domain.tld" "10.100.36.8:8005" inbound|8005|| 127.0.0.6:36657 10.100.36.8:8005 35.191.18.0:0 outbound_.80_._.gs1-api-game-server.d01.svc.cluster.local default

[2021-12-08T08:51:13.036Z] "POST /core_service_v1.CoreService/GetBalance HTTP/2" 200 - via_upstream - "-" 55 17 58 58 "-" "grpc-go/1.36.1" "1" "86dbe583-64d9-982f-abdf-90ea5b0411fa" "e70ae8c0491b8e46cb7909086c8a732a" "369bafcd44cd96f7" "cb7909086c8a732a" "gs1-api-core:50101" "10.100.32.2:8007" outbound|50101||gs1-api-core.d01.svc.cluster.local 10.100.36.8:37150 10.100.66.69:50101 10.100.36.8:50170 - default

ServiceB log attached.
ServiceB-proxy.log
Same with CurlgRPC:
ServiceA log:
[2021-12-08T09:07:21.369Z] "POST /core_service_v1.CoreService/GetBalance HTTP/2" 200 - via_upstream - "-" 63 17 34 30 "-" "grpcurl/v1.8.5 grpc-go/1.37.0" "1" "a48ace99-0d2f-9fa9-9aad-83d5fe03c579" "ebf7561051ac6433952010cd9cac40d3" "222d8415b7cc1280" "952010cd9cacdddd" "gs1-api-core:50101" "10.100.32.2:8007" outbound|50101||gs1-api-core.d01.svc.cluster.local 10.100.36.8:36550 10.100.66.69:50101 10.100.36.8:53974 - default

Service B log attached.
ServiceB-curlgrpc.log

It seems it is fixed by removing b3 and b3flags empty headers. If I don't propogate them from ServiceA to ServiceB trace works and glues togather. Any ideas why? I remember advice from #11696 not to propagate empty headers.

Ah b3 and b3-flags actually implies sample decision: https://github.com/openzipkin/b3-propagation#debug-flag and envoy doc: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#config-http-conn-man-headers-x-b3-flags

Ah b3 and b3-flags actually implies sample decision: https://github.com/openzipkin/b3-propagation#debug-flag and envoy doc: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#config-http-conn-man-headers-x-b3-flags

so the root reason is empty b3 headers broken trace context?

Yes, it seems so. Is it expected behaviour?

so the root reason is empty b3 headers broken trace context?

Yes.

Yes, it seems so. Is it expected behaviour?

Yes. Close since this is expected. Thanks!

authors: bianpengyuan movergan zirain


text:
[release-1.17] improve accesslog mode e2e tests

Manual cherrypick required.
#45504 failed to apply on top of branch "release-1.17":
Applying: improve accesslog mode e2e tests
Applying: retry
Using index info to reconstruct a base tree...
M	tests/integration/telemetry/api/accesslogs_test.go
Falling back to patching base and 3-way merge...
Auto-merging tests/integration/telemetry/api/accesslogs_test.go
CONFLICT (content): Merge conflict in tests/integration/telemetry/api/accesslogs_test.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 retry
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

https://github.com/istio/istio/pull/45532 fixes this issue.

duplicated

authors: istio-testing lei-tang zirain


text:
VM Sidecar drop connections on large number of them

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
We installed sidecar on a virtual machine with MySQL. Our architecture requires a large number of connections to the database (up to 10,000)
But when we reach 1700-2000 connections, we can no longer connect and get errors like this

2023-11-03T10:23:44.119694Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:104	Caught Aborted, suspect faulting address 0x3dd000368cc	thread=223451

2023-11-03T10:23:44.119762Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:91	Backtrace (use tools/stack_decode.py to get line numbers):	thread=223451

2023-11-03T10:23:44.119802Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:92	Envoy version: a1ff538a63890e27dd2add4b2680ba8dc49293ca/1.27.1-dev/Clean/RELEASE/BoringSSL	thread=223451

2023-11-03T10:23:44.119551Z	critical	envoy assert external/envoy/source/common/network/socket_interface_impl.cc:72	assert failure: SOCKET_VALID(result.return_value_). Details: socket(2) failed, got error: Too many open filesthread=223446

ulimit -n
40000
Version
version 1.19.3
OS rocky linux 9

Additional Information
No response

There is a limit in your os, try to check ulimit -a

open files                          (-n) 40000

4000 limits all the process in the os

increase limit, error disappeared but problem not
ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 58746
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 524288
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 58746
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
[root@db1-test ~]# lsof | wc -l
347505
[root@db1-test ~]# netstat -atn|wc -l
1613

IIRC, in istio we set connection pool max connections to MAXUint32, so there is not connection number limit. But there is a max idle time per idle connection. So check if this idle timeout cause connection close or there is a connection number limit on MySQL setting


text:
Add api key

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
First add the api attributs into install/kubernetes/templates/istio-mixer.yaml.tmpl, then run ./install/updateVersion.sh
Release note:

NONE

@apicl: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
43671b10b3e54f9d9647af6959a728331856dabc
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

I don't know the purpose of any of these files, so I'll pass on reviewing this :-)

/lgtm

PTAL

/lgtm

@apicl: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
65b5d1e2c32e830151e0cdf3af46d1adb189f445
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

I'd like someone familiar with all these files to comment first before approving...

/assign @costinm

/lgtm

/retest

/lgtm

@apicl: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
e6a10b2acad1c99a847b07c04c8fdab92e699266
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@apicl: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
e6a10b2acad1c99a847b07c04c8fdab92e699266
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@apicl: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-presubmit.sh
e6a10b2acad1c99a847b07c04c8fdab92e699266
link
/test istio-presubmit

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/lgtm

authors: apicl douglas-reid geeknoid istio-testing manlinl


text:
[release-1.6] unit tests for ServiceEntry update fix

This is an automated cherry-pick of #23354

@googlebot I consent

@istio-testing: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio_release-1.6
b2da85f3a9f1f16e78a094331c4f4af008b1e474
link
/test unit-tests_istio_release-1.6

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@istio-testing: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

unit-tests_istio_release-1.6
b2da85f3a9f1f16e78a094331c4f4af008b1e474
link
/test unit-tests_istio_release-1.6

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

authors: brian-avery istio-testing jacob-delgado rshriram


text:
[kiali] pilot version endpoint is gone, use the new istiod replacement

fixes: https://github.com/istio/istio/issues/23677
pilot service version endpoint is gone in 1.6 - replaced with analogous version endpoint in the istiod service. Kiali needs to switch to using the new istiod endpoint.

In response to a cherrypick label: new pull request created: #23679

authors: istio-testing jmazzitelli


text:
multicluster registries respect endpointmode

Currently only the "main" registry has the endpoint mode set. This adds that for multicluster remote registries.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Pull Request Attributes
Please check any characteristics that apply to this pull request.
[ ] Does not have any changes that may affect Istio users.

@stevenctl: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

release-notes_istio
88e9071cedfa2e10ef923be70372e21abe041875
link
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn I actually don't want this to merge yet. I have a strong feeling the endpointslice tests will fail. I saw that happening in https://github.com/istio/istio/pull/27493 even with this change. That PR is just to get the pilot/mc tests green so we get to the point where that's blocking. I've had it passing a few times now consistently to have it break later when incompatible thing are added.

Looks like ingress gateways in remote clusters fail to get discovery:
https://prow.istio.io/view/gs/istio-prow/pr-logs/pull/istio_istio/27531/integ-pilot-multicluster-tests_istio/1605

@stevenctl: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

gencheck_istio
8d64296bee64befb70fdd4366186d8aa470ac273
link
/test gencheck_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

authors: howardjohn istio-testing stevenctl


text:
ext-authz: return error for invalid extension provider only when it is being used

Fixes #30824
If some extension provider is not used by any authz policy, it should not cause errors even if it might be invalid.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[x] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Pull Request Attributes
Please check any characteristics that apply to this pull request.
[x] Does not have any changes that may affect Istio users.

@yangminzhu: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

integ-pilot-k8s-tests_istio
b19a0db87b840a6e6b4e590eaddd346d079a26c9
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-pilot-k8s-tests_istio

is the error in the original issue coming from the validation webhook when applying the meshconfig updaets? I didn't see other unit test updates other than validation-test.go

is the error in the original issue coming from the validation webhook when applying the meshconfig updaets? I didn't see other unit test updates other than validation-test.go

@incfly not from validation webhook but directly from the authz plugin in pilot, it emits the error every time it is called when pushing xDS configs to proxy.

In response to a cherrypick label: new pull request created: #30959

authors: incfly istio-testing yangminzhu


text:
Fuzzing: Add bootstrap fuzzer

Adds fuzzer that creates a fuzzed bootstrap.PilotArgs{} and passes it on to bootstrap.NewServer()
To help us figure out who should review this PR, please put an X in all the areas that this PR affects.

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[x] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[x] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Please check any characteristics that apply to this pull request.

[x] Does not have any user-facing changes. This may include CLI changes, API changes, behavior changes, performance improvements, etc.

Hi @AdamKorcz. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test

@AdamKorcz: PR needs rebase.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

authors: AdamKorcz hzxuzhonghu istio-testing