text:
"gRPC config stream closed: 13" closes all proxy connexions

Bug description
Istio 1.3.0 Ingress gateway (may also happen on any sidecar) is logging the error [2019-09-25 15:35:47.514][22][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, every few minutes, from 5 to 25 mins usually.
While the issue https://github.com/istio/istio/issues/8270 is stating that this is just an "info" log and should not be linked to any issue, I do see all my current connexions going through the gateway to be terminated.
Turning logs to info on the Gateway :
istio-ingressgateway-7dc54f8787-w7gng istio-proxy [2019-09-25 15:35:47.514][22][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
istio-ingressgateway-7dc54f8787-w7gng istio-proxy [2019-09-25 15:35:47.740][22][info][filter] [src/istio/utils/logger.cc:83] Logger active
istio-ingressgateway-7dc54f8787-w7gng istio-proxy [2019-09-25 15:35:47.742][22][info][filter] [src/istio/utils/logger.cc:83] Logger active
istio-ingressgateway-7dc54f8787-w7gng istio-proxy [2019-09-25 15:35:47.742][22][info][upstream] [external/envoy/source/server/lds_api.cc:60] lds: add/update listener '0.0.0.0_443'
istio-ingressgateway-7dc54f8787-w7gng istio-proxy {"duration":"1463305","downstream_local_address":"10.230.128.165:443","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"grpc-go/1.23.0","response_flags":"DC","start_time":"2019-09-25T15:12:08.431Z","method":"POST","request_id":"aba73ff8-8418-4979-b8ac-cfd9ccab561e","upstream_host":"10.230.128.103:1081","x_forwarded_for":"10.230.128.4","requested_server_name":"useredged.alerting.devops.services","bytes_received":"2241","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|1081||useredged.alerting.svc.cluster.local","downstream_remote_address":"10.230.128.4:34886","authority":"useredged.alerting.devops.services:443","path":"/ra.useredgesvc.UserEdgeSVC/BiFeed","protocol":"HTTP/2","upstream_service_time":"-","upstream_local_address":"-"}

The logs here are showing a "change" in the listener on port 443, which obviously closes all the opened TCP connexions.
At the same time, Pilot is also logging ads errors while communicating to the IngressGateway.
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[x] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
gRPC config stream closed: 13 error should not lead to a loss of connexions
Steps to reproduce the bug
Deployed Istio 1.3.0 using helm chart. start a "streaming" GRPC connexion through the ingress gateway suing HTTPS
Version (include the output of istioctl version --remote and kubectl version)
istioctl version --remote
client version: 1.3.0
control plane version: 1.3.0

kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Helm chart using helm template
Environment where bug was observed (cloud vendor, OS, etc)
Azure AKS cluster 1.14.6

The Proxy listener change on the Ingress Gateway is triggered when Pilot timeout it's connexions with each proxy.
It sounds to be linked to keepaliveMaxServerConnectionAge parameter on Pilot.
Looking at the docs at it states : https://istio.io/docs/reference/commands/pilot-discovery/
Maximum duration a connection will be kept open on the server before a graceful close. (default `2562047h47m16.854775807s`)

Why is mine set to 30s ? (deployed from Helm)
What is a real good value for this parameter ?
Why does this timeout + reconnect is seeing a change in the listener on port 443 ?
Any idea/help welcomed...

Just dumped the listener config before / after the relaod using :
istioctl -n istio-system proxy-config listener  istio-ingressgateway-b95849b6d-w5mmf -o json > ~/tmp/before`
.. wait for config change
istioctl -n istio-system proxy-config listener  istio-ingressgateway-b95849b6d-w5mmf -o json > ~/tmp/after

diff  ~/tmp/before ~/tmp/after

There is no change...

Why is mine set to 30s ? (deployed from Helm)

30 seconds or 30m? The default is 30m so if its 30s I suspect you may have configured it yourself somehow?
This may be linked to https://github.com/istio/istio/issues/12162? If you are in a cluster you can experiment, you could try setting --set pilot.env.PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true, or just set the environment variable on the pilot deployment. Note this will result in a huge increase in Pilot CPU usage, so I wouldn't try it in production

Sorry it's 30 minutes, not seconds :)
I'm going to try your option and let you know. Thanks

setting DISABLE_XDS_MARSHALING_TO_ANY=true solved the issue.
CPU and memory is impacted, as suggested :

Is there something wrong with my install or this is a regression ?
Thanks.

@howardjohn if I understand the issue correctly, every 30m Pilot will reset connections made to xDS servers by the Envoy proxy which can cause the proxy to drop all configuration and reload.
In the interim period the service can have disruption as there might be no listeners/routes/clusters/endpoint config at the Envoy proxies between a disconnect and config getting re-populated.
Is this accurate?

it's exactly that @nrjpoddar  !

In the interim period the service can have disruption as there might be no listeners/routes/clusters/endpoint config at the Envoy proxies between a disconnect and config getting re-populated.

No, there should be no time period with no config -- it will hotswap in the new config so there should be no downtime.

What I see is the Proxy (envoy) drops and re-create the listeners, so, yes, it does have to end all the opened TCP connexions...
Maybe Envoy should compare the configs then decide to stop the listener if needed... I really feel it's an issue on Envoy side, but as this does not happen when setting PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true, I feel like maybe Istio is sending some config it should not...

No, there should be no time period with no config -- it will hotswap in the new config so there should be no downtime.

That makes sense on the desired and hopefully implemented behavior. Why does this PILOT_DISABLE_XDS_MARSHALING_TO_ANY env matter in this case? I understand how it can be linked to CPU spikes but not destroying/re-creating listeners.

What I see is the Proxy (envoy) drops and re-create the listeners, so, yes, it does have to end all the opened TCP connexions...Maybe Envoy should compare the configs then decide to stop the listener if needed... I really feel it's an issue on Envoy side, but as this does not happen when setting PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true, I feel like maybe Istio is sending some config it should not...

Envoy does do this comparing by computing the hash of the config. What is possible is that when Marshaling to Any it is not doing this in a stable manner (eg unordered list). See https://github.com/envoyproxy/envoy/issues/6252

Thanks @howardjohn for the info. Looks like we will need to uplift https://github.com/istio/envoy/tree/release-1.3 to get the upstream Envoy PR and then change istio/proxy release-1.3 to the new SHA. I'm assuming we want this addressed in 1.3.x?

Yes if possible. I had thought it was already fixed -- I think there were some attempts months back to fix this but apparently didn't fix all cases if you are still seeing this.

Ok, I can take a stab at merging the envoy changes and go from there. Do we cherry-pick just the PR commit we need from envoyproxy or just move to the latest master in the Istio fork?

Not actually sure.. @rlenglet @duderino how do we pull in envoy updates?

@rlenglet @duderino how do we pull in envoy updates?

For 1.3, just backport directly into istio/envoy:release-1.3.
istio/envoy:master is currently not used and is far behind. Just ignore it.

Let me know when you have something merged (with docker images would be even simpler for me) so I can test it.
Many thanks for your help on this.

For 1.3, just backport directly into istio/envoy:release-1.3.

My question is specific to release-1.3 branch, when you say backport are we just cherry-picking relevant commits/PR or are should we be doing merge of the current envoyproxy/envoy:master into istio/envoy:release-1.3?

@rlenglet  ^
Can we check if this is in master already? If so maybe @prune998 or someone in #17139 (basically the same issue) try out the daily build and see if it is resolved?

Can we check if this is in master already?

istio/envoy:master is even more outdated than istio/envoy:release-1.3 afaict

I just realized that istio/envoy:release-1.3 is based of envoyproxy/envoy-wasm and not envoyproxy/envoy so there is another refresh that needs to happen.

My question is specific to release-1.3 branch, when you say backport are we just cherry-picking relevant commits/PR or are should we be doing merge of the current envoyproxy/envoy:master into istio/envoy:release-1.3?

Please cherry-pick.

thank you all for helping get to the bottom of this!! Let's consolidate this into https://github.com/istio/istio/issues/17139 since it is the same issue

Hi @prune998 please follow the comments in this thread: https://github.com/istio/istio/issues/17139#issuecomment-537728166 and try it out in your environment to see if the issue is addressed.
Feel free to comment/re-open if that's not the case.

will do in an hour. Thanks.

Seems to solve the issue. I commentend in the other thread. Maybe we can close this issue ?
https://github.com/istio/istio/issues/17139#issuecomment-538048560

update istio to 1.3.3 but still get errors: my grpc client <--> server still get some erorr_msg:OS Error and tcp Reset, and the logs of istio-proxy still has:
[2019-10-15 07:05:59.038][24][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-10-15 07:39:07.474][24][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-10-15 08:11:41.801][24][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,

@nrjpoddar @rlenglet  could you assign project member on https://github.com/istio/istio/issues/27513?
People still encounter this same kind of issue in production from time to time, even after the fix has been applied.
This lead me to believe that other root cause(s) is(are) still out there.
At lease these versions are still involved:
1.6.8
1.6.14
1.7.3
1.9.2
1.10

Same question：
istio 1.6.14
warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13,

php error
fwrite(): send of 527 bytes failed with errno=32 Broken pipe

Don't use Istio 1.6, it's over 2 years old and EOL. If you do, please don't
post issues for it.
On Thu, May 26, 2022 at 7:15 PM leez.lee @.***> wrote:

Same question：
istio 1.6.14
warning	envoy config	[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13,
php error
fwrite(): send of 527 bytes failed with errno=32 Broken pipe
—
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/17383#issuecomment-1139214768, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEYGXMVXERQZXBHZ3LX4JLVMAV2VANCNFSM4I2OZ2PQ
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

authors: BenjaminHuang ckcd glyslxq howardjohn nrjpoddar prune998 rlenglet


text:
Kiali is unable to capture the traffic for bookinfo app.

Bug description
Kiali is unable to capture the traffic for bookinfo app on Openshift 3.11 on IBM Cloud.
Deployed the bookinfo app and expose the "productpage service" and generated traffic using the productpage svc route , but in Kilai nothing is showing up.
But If I expose a istio ingress gateway service and generate the traffic some portion of traffic showsup in kiali.
Question: Is it necessary to use istio gateway to see the traffic in Kiali?
Affected product area (please put an X in all that apply)
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
Kiali should show the traffic for bookinfo app.
Steps to reproduce the bug
Deploy bookinfo On Openshift 3.11 (on IBM Cloud) and access the app using istio ingress svc (by exposing it securely)
Version (include the output of istioctl version --remote and kubectl version)
bash-4.2$ istioctl version --remote
client version: 1.3.2
citadel version: 1.3.2
egressgateway version: 1.3.2
galley version: 1.3.2
ingressgateway version: 1.3.2
pilot version: 1.3.2
policy version: 1.3.2
sidecar-injector version: 1.3.2
telemetry version: 1.3.2
kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-09-12T23:41:09Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"
How was Istio installed?
Using Helm (demo profile)
Environment where bug was observed (cloud vendor, OS, etc)
IBM Cloud
Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

@howardjohn Do we have any insight on this? I have tried this in multiple cluster and got same result.
Thanks
Chayan

Did you use CNI plugin?
Also, what k8s level is openshift 3.11?  You want to make sure it met our k8s requirements: https://istio.io/docs/setup/platform-setup/

With istio 1.3.3, on OpenShift 3.11.88, i confirm i have the same issue.

@linsun Please find my answer below:
Did you use CNI plugin: I hope yes. I have installed the istio with below command.
"helm template install/kubernetes/helm/istio --name istio --namespace istio-system --values install/kubernetes/helm/istio/values-istio-demo.yaml --set istio_cni.enabled=true | kubectl apply -f - "
Also as per @sethokayba on openshift 3.11 CNI plugin is enabled by default.-->> https://istio.io/docs/setup/additional-setup/cni/
Version:
openshift v3.11.146
kubernetes v1.11.0+d4cacc0
Also all the platform related configuration are in place . I  am able to spin up the bookinfo app as well.

Note: k8s 1.11 is not on the list of recommended k8s from https://istio.io/docs/setup/platform-setup/

@brian-avery @knrc could you shed some light on if openshift v3.11 works with Istio 1.3?
https://istio.io/docs/setup/platform-setup/openshift/
Also, is this instruction only for openshift on prem?  https://docs.openshift.com/container-platform/3.11/servicemesh-install/servicemesh-install.html#service-mesh-install_prerequisites
cc @jwendell

We have tested Istio 1.1 (as shipped in Maistra) with OpenShift 3.11.
Istio 1.3 (and 1.4) were tested on OpenShift 4.1 and 4.2 respectively.
For a good out-of-box experience using Istio on OpenShift we recommend using Maistra: https://maistra.io.

@jwendell I just installed istio-1.1.9 (OpenShift on IBM Cloud). and deployed bookinfo app.
As our FW allows only 443 , I have reconfigured the bookinfo gateway with HTTPS something like this.
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: `/etc/istio/ingressgateway-certs/tls.crt

And able to access the app using istio-ingressgateways external IP.
Problem: Kiali cannot be accessible via route , I did a port forwarding to access. but the problem remains same. Kiali cannot catch all the traffic.

@jwendell To use Maistra ... I think we will have to configure the nodes and registry ... how do I configure them in openshift on IBM Cloud where you don't have os label access?

@chayan88 nodes are already configured for you, so you don't need to worry about it.

@chayan88  i believe your issue is sidecar isn't capture any traffic.  So the istio cni doesn't do its job to program the traffic redirect for the pod.

hello,
I just installed istio 1.1.15, which is declared as tested with kubernetes 1.11, and the problem is the same.
I used:

OpenShift 3.11.88 (kubernetes 1.11)
Istio 1.1.15
Cni-plugin for sidecar injection.

The traffic is not hitting the other services on kiali :

@chayan88
I tried to go with the same config as in my last comment, and found that kiali is showing 100% of generated traffic but using a different example. https://rinormaloku.com/istio-an-introduction/

Can you please try this sample app instead of bookinfo app and give us your results ?

@sethokayba If you dont mind could you please let me know what is the exact command you run for the installation along with CNI plugin?
I am running this command
helm template install/kubernetes/helm/istio --name istio --namespace istio-system     --values install/kubernetes/helm/istio/values-istio-demo.yaml --set istio_cni.enabled=true | kubectl apply -f -

@sethokayba this time with 1.3.3 istio upstream with different app mentioned by you worked for me.

on OpenShift 3.11, you don't need to install the CNI Plugin, it is already included, you need just to pass the --set istio_cni.enabled=true to activate it.
For now, the thing to do, is to find the difference between "sentiment" and "bookinfo" configs. What makes "sentiment" work and what don't for "bookinfo"

source: gharchive/issue

authors: chayan88 jwendell linsun sethokayba

repo: istio/istio

license_type: permissive

license_source: github-api


text:
External nginx accesses K8S by http with large request body(30k) slowly(3s)

Bug Description
                    _________(4)_____
                   /                 \
client--(1)-->nginx---(2)--->k8s----->Node
   \___________(3)__________/

(3)   :50ms
(1)(4):56ms

(1)(2):3000ms
k8s contains：istio,when request body smaller ，then request time smaller

response is null,Is Request problem

more time spend on socket,every time 200ms,I didn't set istio buffer

2021-11-25 19:39:40.619 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Registered read interest for [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]]
2021-11-25 19:39:40.819 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read direct from socket: [1398]
2021-11-25 19:39:40.819 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Received [,"81983067558090138","-4323871760126837319","633774505488365412","7410418044286815305","1371140723808735600","-2300376376544271968","2951147133693625867","6113232255313590202","-8342399442765923318","5980318295795919693","-2988920748733493526","6160840572437824416","-8020402543256672930","3961168924224050717","-700318907431244624","5415961205304082761","-36559229184096336","4390078983419048162","8769729974953551161","-9057025827780544916","-1199436727386034092","-1263509277303341834","-4081906218303952560","-5682905977705060701","666921464191633397","-6460959057039590840","7471082418880651473","7871791953886676892","6727841351839621384","5506991634409644656","-4484534021905967839","4305040709395019047","-1260132012630055537","-1819570144916250930","1609463874643525238","7551834216209262190","7285038908050053030","5317682422057814920","2522419124396923359","-2540667765411146359","743672998318898550","9046260284578394893","724275536151032318","115765274654216660","5188164022643938119","-6451617894879454337","-6890888214546540971","818626724735905480","1757522606312002806","8780877389698297439","-6633831760562671836","8910869093054625458","3144379710255846777","8788820082583774201","8{"head":{"syscode":"09"},"data":{"hotelId":"868","rateIds":["4973205683719548978","-4099288786701839056","-3969989209876160007","3559846983539160197","-6932499306999151351","9138967400820904148","-5]
2021-11-25 19:39:40.819 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Before fill(): parsingHeader: [false], parsingRequestLine: [false], parsingRequestLinePhase: [0], parsingRequestLineStart: [0], byteBuffer.position(): [2848], byteBuffer.limit(): [2848], end: [1450]
2021-11-25 19:39:40.820 DEBUG 1 --- [nio-8091-exec-3] o.a.tomcat.util.net.SocketWrapperBase    : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read from buffer: [0]
2021-11-25 19:39:40.820 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Registered read interest for [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]]
2021-11-25 19:39:41.150 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read direct from socket: [1398]
2021-11-25 19:39:41.150 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Received [914995835448333056","-7413226614932867529","2670550698550639927","-3106788153424550569","-250806289955148448","-7882675913105294601","5288902085839417153","-527888297470586900","-8249859143653463294","-1774241003411533976","-6704658909002110832","-6493726291670431098","4693491961516155652","4461292153699823155","-5799299420075900057","-3251845292851977934","2696953795558670558","416382867471506740","-2680581381529102089","-5187766432694922350","-6698968481452401732","-2904275601304516317","3127840137160051854","5973300315068039435","-4639391966170195654","2157081861811093202","8209051988203425379","1400260112915551246","-5701202508708143566","-2679823026882699032","-6296296280489335319","7525984157845816620","-60730528949773072","-493646545915949761","-8864132490744479704","-6669105368159484921","4992788689965530011","5419269121532184710","157478893815489387","-4524042107480924536","-1785736545008882682","-5643913863179435715","-7268275592654618191","-1702171933480319210","3266893929293178452","2573511154189867989","-2224821418959639381","-1818240278191763245","7095828665920069875","3432453452263079049","-2741262583984451656","-1170055589888280799","-6344656619332770519","-5928118160581792368","7844048292554242149","9012438083656840169","-4610229378941227399","7421817029076177317","-7468236095439022179","-1165378763217974864","4547339396770527725","-7782965699486765515","8630]
2021-11-25 19:39:41.151 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Before fill(): parsingHeader: [false], parsingRequestLine: [false], parsingRequestLinePhase: [0], parsingRequestLineStart: [0], byteBuffer.position(): [2848], byteBuffer.limit(): [2848], end: [1450]
2021-11-25 19:39:41.151 DEBUG 1 --- [nio-8091-exec-3] o.a.tomcat.util.net.SocketWrapperBase    : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read from buffer: [0]
2021-11-25 19:39:41.151 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Registered read interest for [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]]
2021-11-25 19:39:41.241 DEBUG 1 --- [nio-8091-Poller] org.apache.tomcat.util.net.NioEndpoint   : timeout completed: keys processed=2; now=1637840381240; nextExpiration=1637840381216; keyCount=1; hasEvents=false; eval=false
2021-11-25 19:39:41.241 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read direct from socket: [1398]
2021-11-25 19:39:41.241 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Received [6464364780342","8074124601565984569","6285227121009348017","3513523466137622038","1061903693472851151","4104057819048849956","3508217889377482475","-3905843838370921473","9083812035673580510","-3642687813240354486","-7498036014308271758","-9183711927411692909","5586905287882974815","6384115912063628391","6678934405078772133","7815968079493626299","5981726668753780596","2259003220071473294","-5944088608470356739","-6611545439930607722","-5958369191464484988","4903790238762823130","6502902871117999937","2851751787248505276","-7404375986315551858","3936607722009567371","-8164703222665684962","-5220615339717968322","-1164054654585988933","8092687942970374611","-5990240940564905890","-2831792170081826732","1406965309161269627","-4582639944397352837","-4217151089792229698","-1126290517261652875","3421627543576981625","8686651354594809820","380469382638252315","-8193223709142872559","7994389951355470384","-5017815150269597963","2470269372712901589","-6996234257002840275","-6192589922676554207","6038934111078608094","-1979363852452376893","5397663682078692462","7680405538503424808","-8214141846036268529","4492605710704695106","7063027223912878708","-421153419231149753","6062276872794815796","-6186118561390288072","1842058827082177799","4135148976892491530","8268927621656821347","-4742606129507250210","6995019737045885085","-5057941164944435393","3227980012894914059","-897823872596461]
2021-11-25 19:39:41.241 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Before fill(): parsingHeader: [false], parsingRequestLine: [false], parsingRequestLinePhase: [0], parsingRequestLineStart: [0], byteBuffer.position(): [2848], byteBuffer.limit(): [2848], end: [1450]
2021-11-25 19:39:41.241 DEBUG 1 --- [nio-8091-exec-3] o.a.tomcat.util.net.SocketWrapperBase    : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read from buffer: [0]
2021-11-25 19:39:41.242 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Registered read interest for [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]]
2021-11-25 19:39:41.422 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read direct from socket: [1398]
2021-11-25 19:39:41.422 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Received [0595","-7810902863469510944","-5142713273074591100","-7190881468515238397","-4060176464784817004","4123105121567010470","-4469008737337445376","-9198923340942970922","8356235890617948609","-1421866384558967671","-671532079933482511","1854573922302848406","9203497255234133352","-2370978650286677343","-2826765998949179867","2109541439525261651","6164366683000739724","-7184250126069213968","-3514743025148620909","7154036686411699326","-2161035924671036199","-3033610341975008574","-5852408681718630469","-2241038547644991300","-8796657736640740881","6875515174026214396","-2036733060277522805","1060543643403304202","8535490693759338988","7511712202523968713","-569521235145535980","4964186812617553515","5805710446222299039","-958712231540519737","9101676554154611927","-5020388330225828647","-7127067540271587971","1907157068019997384","1738688880428452333","-2585920109550679843","-4095102220532424470","7046292810730601876","-7681720542288854490","7614673558757444440","2945975524123669595","8701390891742394164","-7923411416687601427","8617875004604195168","-5340865503662864662","6514116782494587755","-8002474245116430597","2337828758904176331","-2479571798137376570","4882794834610872698","6124306239466500716","-3372408073816862803","7703255949719029900","3766403963595108245","-9176813528465771619","-4245223917071223406","81983067558090138","-4323871760126837319","633774505488365412","]
2021-11-25 19:39:41.422 DEBUG 1 --- [nio-8091-exec-3] o.a.coyote.http11.Http11InputBuffer      : Before fill(): parsingHeader: [false], parsingRequestLine: [false], parsingRequestLinePhase: [0], parsingRequestLineStart: [0], byteBuffer.position(): [2848], byteBuffer.limit(): [2848], end: [1450]
2021-11-25 19:39:41.423 DEBUG 1 --- [nio-8091-exec-3] o.a.tomcat.util.net.SocketWrapperBase    : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read from buffer: [0]
2021-11-25 19:39:41.423 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Registered read interest for [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]]
2021-11-25 19:39:41.628 DEBUG 1 --- [nio-8091-exec-3] org.apache.tomcat.util.net.NioEndpoint   : Socket: [org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@4f2dfc7:org.apache.tomcat.util.net.NioChannel@b7b5aa0:java.nio.channels.SocketChannel[connected local=/127.0.0.1:8091 remote=/127.0.0.1:32876]], Read direct from socket: [1398]
Version
k8s:v1.18.5
istio:1.5.9

Additional Information
No response

Can you provide envoy access logs？
Nginx use http1.0 protocol to backend by default, Envoy will return the response directly but will not close the link immediately, Nginx thinks that the response ends when the connection is closed.
https://github.com/envoyproxy/envoy/issues/6392

[Envoy (Epoch 0)] [2021-11-27 04:24:27.424][28][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:361] [C20] new connection
[Envoy (Epoch 0)] [2021-11-27 04:24:27.424][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:473] [C20] socket event: 3
[Envoy (Epoch 0)] [2021-11-27 04:24:27.424][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:561] [C20] write ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.428][28][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C20] handshake expecting read
[Envoy (Epoch 0)] [2021-11-27 04:24:27.428][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:511] [C20] read ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.428][28][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C20] handshake expecting read
[Envoy (Epoch 0)] [2021-11-27 04:24:27.432][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:473] [C20] socket event: 3
[Envoy (Epoch 0)] [2021-11-27 04:24:27.432][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:561] [C20] write ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:176] [C20] handshake complete
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:511] [C20] read ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: -1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:155] [C20] ssl read 0 bytes into 0 slices
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:473] [C20] socket event: 3
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:561] [C20] write ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:511] [C20] read ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: 2303
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: -1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:155] [C20] ssl read 2303 bytes into 1 slices
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:518] [C20] parsing 2303 bytes
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:703] [C20] message begin
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:263] [C20] new stream
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:459] [C20] completed header: key=host value=www.xxxxxxxxxxxxxx.com
[Envoy (Epoch 0)] [2021-11-27 04:24:27.433][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:459] [C20] completed header: key=x-forwarded-prefix value=/Services/xxxxx-cache-business-api/
.................
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:615] [C20] onHeadersCompleteBase complete for headers
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:459] [C20] completed header: key=x-envoy-original-path value=/Services/xxxxx-cache-business-api/cache/query
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:803] [C20] Server: onHeadersComplete size=24
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:732] [C20][S1708058540123959888] request headers complete (end_stream=false):
':authority', 'www.xxxxxxxxxxxxxx.com'
':path', '/cache/query'
':method', 'POST'
'x-forwarded-prefix', '/Services/xxxxx-cache-business-api/'
'x-real-ip', '173.18.152.15'
'x-forward-for', '173.18.152.15'
'x-forward-proto', 'http'
'x-nginx-proxy', 'true'
'content-length', '1052447'
'accept', '*/*'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36'
'content-type', 'application/json'
'origin', 'http://173.18.144.117:15460'
'referer', 'http://173.18.144.117:15460/swagger-ui/'
'accept-encoding', 'gzip, deflate'
'accept-language', 'zh-CN,zh;q=0.9,en;q=0.8,zh-TW;q=0.7'
'x-forwarded-for', '10.244.0.0'
'x-forwarded-proto', 'http'
'x-envoy-internal', 'true'
'x-request-id', '56b8d2d7-5bdd-4f35-9e5b-70fdbf0fd1ed'
'x-envoy-decorator-operation', 'xxxxx-cache-business-api-svc.xxxxx-cache.svc.cluster.local:80/Services/xxxxx-cache-business-api/*'
'x-istio-attributes', 'CjoKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIeEhxob3RlbC1jYWNoZS1idXNpbmVzcy1hcGktc3ZjCi4KHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEg0SC2hvdGVsLWNhY2hlCk8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTc0ODRkZmM2ODgtcWd0OHYuaXN0aW8tc3lzdGVtClgKGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBI8Ejpob3RlbC1jYWNoZS1idXNpbmVzcy1hcGktc3ZjLmhvdGVsLWNhY2hlLnN2Yy5jbHVzdGVyLmxvY2FsClYKF2Rlc3RpbmF0aW9uLnNlcnZpY2UudWlkEjsSOWlzdGlvOi8vaG90ZWwtY2FjaGUvc2VydmljZXMvaG90ZWwtY2FjaGUtYnVzaW5lc3MtYXBpLXN2Yw=='
'x-b3-traceid', '1c8866549d663564dafecd982f17fab3'
'x-b3-spanid', 'dafecd982f17fab3'
'x-b3-sampled', '0'
'x-envoy-original-path', '/Services/xxxxx-cache-business-api/cache/query'

[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][debug][filter] [src/envoy/http/mixer/filter.cc:47] Called Mixer::Filter : Filter
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][debug][filter] [src/envoy/http/mixer/filter.cc:148] Called Mixer::Filter : setDecoderFilterCallbacks
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1906] [C20][S1708058540123959888] setting buffer limit to 21000000
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:989] [C20][S1708058540123959888] decode headers called: filter=0x562740b329c0 status=1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:874] [C20] body size=775
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1128] [C20][S1708058540123959888] decode data called: filter=0x562740b329c0 status=1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.434][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:539] [C20] parsed 2303 bytes
[Envoy (Epoch 0)] [2021-11-27 04:24:27.624][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:473] [C20] socket event: 3
[Envoy (Epoch 0)] [2021-11-27 04:24:27.624][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:561] [C20] write ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:511] [C20] read ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: 1398
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: -1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:155] [C20] ssl read 1398 bytes into 1 slices
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:518] [C20] parsing 1398 bytes
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:874] [C20] body size=1398
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1128] [C20][S1708058540123959888] decode data called: filter=0x562740b329c0 status=1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.625][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:539] [C20] parsed 1398 bytes
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:473] [C20] socket event: 3
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:561] [C20] write ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:511] [C20] read ready
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: 1398
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: -1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:155] [C20] ssl read 1398 bytes into 1 slices
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:518] [C20] parsing 1398 bytes
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:874] [C20] body size=1398
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1128] [C20][S1708058540123959888] decode data called: filter=0x562740b329c0 status=1
[Envoy (Epoch 0)] [2021-11-27 04:24:27.834][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:539] [C20] parsed 1398 bytes
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:473] [C20] socket event: 3
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:561] [C20] write ready
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][connection] [external/envoy/source/common/network/connection_impl.cc:511] [C20] read ready
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: 1398
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:80] [C20] ssl read returns: -1
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:155] [C20] ssl read 1398 bytes into 1 slices
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:518] [C20] parsing 1398 bytes
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:874] [C20] body size=1398
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1128] [C20][S1708058540123959888] decode data called: filter=0x562740b329c0 status=1
[Envoy (Epoch 0)] [2021-11-27 04:24:28.037][28][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:539] [C20] parsed 1398 bytes

some node slowly,recreate node and set 2 server for traffic  replace master


text:
Failed ： istioctl isntall  --set profile=demo

Bug Description
Istiod : Readiness probe failed
Ingress gateways,Egress gateways  cannot create pod
Version
istioctl version
istioctl 1.13.0

kubectl version --short
Client Version: v1.21.10
Server Version: v1.21.10

Additional Information
my kubernetes cluster model:
Master Node: master1,master2
Work Node: node1, node2
cluster model create by
kubeadm init --apiserver-advertise-address 192.168.80.221 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.21.10 --pod-network-cidr  "192.168.0.0/16"   --service-cidr  "10.96.0.0/12"  --control-plane-endpoint "k8smaster1"
k8s CNI is calico

can you shared more info?
any log from istiod and also kubectl describe rs -nistio-system  <istio-ingressgatew-xxxxx replace with the real name>

Infomation More：
【1】[root@k8smaster1 istio-1.13.0]# kubectl describe pod istiod-7586c7dfd8-4qtjk  -n istio-system
Events:
Type     Reason     Age                  From               Message

Normal   Scheduled  25m                  default-scheduler  Successfully assigned istio-system/istiod-7586c7dfd8-4qtjk to k8smaster2
Normal   Pulling    25m                  kubelet            Pulling image "docker.io/istio/pilot:1.13.0"
Normal   Pulled     24m                  kubelet            Successfully pulled image "docker.io/istio/pilot:1.13.0" in 1m0.599004058s
Normal   Created    24m                  kubelet            Created container discovery
Normal   Started    24m                  kubelet            Started container discovery
Warning  Unhealthy  35s (x481 over 24m)  kubelet            Readiness probe failed: Get "http://192.168.137.65:8080/ready": dial tcp 192.168.137.65:8080: connect: connection refused
【2】 [root@k8smaster1 istio-1.13.0]# kubectl describe pod istio-ingressgateway-dc9c8f588-rwzbd  -n istio-system
Events:
Type     Reason       Age                   From               Message

Normal   Scheduled    21m                   default-scheduler  Successfully assigned istio-system/istio-ingressgateway-dc9c8f588-rwzbd to k8snode2
Warning  FailedMount  19m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[ingressgateway-ca-certs kube-api-access-tl7sr istio-envoy istio-data config-volume podinfo ingressgateway-certs istiod-ca-cert istio-token]: timed out waiting for the condition
Warning  FailedMount  16m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-token kube-api-access-tl7sr podinfo istiod-ca-cert istio-envoy config-volume ingressgateway-certs ingressgateway-ca-certs istio-data]: timed out waiting for the condition
Warning  FailedMount  14m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-data istio-envoy config-volume ingressgateway-certs ingressgateway-ca-certs istiod-ca-cert istio-token podinfo kube-api-access-tl7sr]: timed out waiting for the condition
Warning  FailedMount  12m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[kube-api-access-tl7sr istiod-ca-cert istio-token podinfo ingressgateway-ca-certs istio-envoy config-volume istio-data ingressgateway-certs]: timed out waiting for the condition
Warning  FailedMount  9m56s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istiod-ca-cert ingressgateway-certs kube-api-access-tl7sr istio-data podinfo istio-envoy config-volume istio-token ingressgateway-ca-certs]: timed out waiting for the condition
Warning  FailedMount  7m40s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[ingressgateway-certs ingressgateway-ca-certs istiod-ca-cert istio-envoy istio-token kube-api-access-tl7sr config-volume istio-data podinfo]: timed out waiting for the condition
Warning  FailedMount  5m23s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[config-volume istio-token ingressgateway-ca-certs istio-envoy istio-data ingressgateway-certs kube-api-access-tl7sr podinfo istiod-ca-cert]: timed out waiting for the condition
Warning  FailedMount  3m7s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[ingressgateway-certs podinfo ingressgateway-ca-certs kube-api-access-tl7sr istiod-ca-cert istio-data istio-envoy config-volume istio-token]: timed out waiting for the condition
Warning  FailedMount  2m41s (x17 over 21m)  kubelet            MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
Warning  FailedMount  39s (x2 over 49s)     kubelet            (combined from similar events): MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
【3】root@k8smaster1 istio-1.13.0]# kubectl describe pod istio-egressgateway-6cf5fb4756-p4bzf  -n istio-system
Events:
Type     Reason       Age                 From               Message

Normal   Scheduled    21m                 default-scheduler  Successfully assigned istio-system/istio-egressgateway-6cf5fb4756-p4bzf to k8snode1
Warning  FailedMount  19m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istiod-ca-cert egressgateway-ca-certs config-volume istio-data kube-api-access-hlnk6 istio-envoy istio-token podinfo egressgateway-certs]: timed out waiting for the condition
Warning  FailedMount  17m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[podinfo egressgateway-ca-certs kube-api-access-hlnk6 istio-envoy istiod-ca-cert istio-token istio-data config-volume egressgateway-certs]: timed out waiting for the condition
Warning  FailedMount  15m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[egressgateway-certs istiod-ca-cert istio-token istio-data egressgateway-ca-certs istio-envoy podinfo kube-api-access-hlnk6 config-volume]: timed out waiting for the condition
Warning  FailedMount  12m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[egressgateway-ca-certs kube-api-access-hlnk6 istiod-ca-cert config-volume istio-token egressgateway-certs istio-data istio-envoy podinfo]: timed out waiting for the condition
Warning  FailedMount  10m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[podinfo kube-api-access-hlnk6 istiod-ca-cert istio-envoy config-volume istio-token istio-data egressgateway-certs egressgateway-ca-certs]: timed out waiting for the condition
Warning  FailedMount  8m22s               kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istiod-ca-cert istio-token istio-data istio-envoy config-volume egressgateway-ca-certs podinfo egressgateway-certs kube-api-access-hlnk6]: timed out waiting for the condition
Warning  FailedMount  6m6s                kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-envoy istiod-ca-cert kube-api-access-hlnk6 istio-data podinfo egressgateway-certs egressgateway-ca-certs config-volume istio-token]: timed out waiting for the condition
Warning  FailedMount  3m50s               kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[config-volume istio-data egressgateway-certs istiod-ca-cert podinfo kube-api-access-hlnk6 istio-envoy istio-token egressgateway-ca-certs]: timed out waiting for the condition
Warning  FailedMount  93s                 kubelet            (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-envoy istiod-ca-cert egressgateway-certs egressgateway-ca-certs kube-api-access-hlnk6 istio-token podinfo config-volume istio-data]: timed out waiting for the condition
Warning  FailedMount  78s (x18 over 21m)  kubelet            MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found

can you shared more info? any log from istiod and also kubectl describe rs -nistio-system <istio-ingressgatew-xxxxx replace with the real name>

【More Details】
Infomation More：
【1】[root@k8smaster1 istio-1.13.0]# kubectl describe pod istiod-7586c7dfd8-4qtjk  -n istio-system
Events:
Type     Reason     Age                  From               Message

Normal   Scheduled  25m                  default-scheduler  Successfully assigned istio-system/istiod-7586c7dfd8-4qtjk to k8smaster2
Normal   Pulling    25m                  kubelet            Pulling image "docker.io/istio/pilot:1.13.0"
Normal   Pulled     24m                  kubelet            Successfully pulled image "docker.io/istio/pilot:1.13.0" in 1m0.599004058s
Normal   Created    24m                  kubelet            Created container discovery
Normal   Started    24m                  kubelet            Started container discovery
Warning  Unhealthy  35s (x481 over 24m)  kubelet            Readiness probe failed: Get "http://192.168.137.65:8080/ready": dial tcp 192.168.137.65:8080: connect: connection refused
【2】 [root@k8smaster1 istio-1.13.0]# kubectl describe pod istio-ingressgateway-dc9c8f588-rwzbd  -n istio-system
Events:
Type     Reason       Age                   From               Message

Normal   Scheduled    21m                   default-scheduler  Successfully assigned istio-system/istio-ingressgateway-dc9c8f588-rwzbd to k8snode2
Warning  FailedMount  19m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[ingressgateway-ca-certs kube-api-access-tl7sr istio-envoy istio-data config-volume podinfo ingressgateway-certs istiod-ca-cert istio-token]: timed out waiting for the condition
Warning  FailedMount  16m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-token kube-api-access-tl7sr podinfo istiod-ca-cert istio-envoy config-volume ingressgateway-certs ingressgateway-ca-certs istio-data]: timed out waiting for the condition
Warning  FailedMount  14m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-data istio-envoy config-volume ingressgateway-certs ingressgateway-ca-certs istiod-ca-cert istio-token podinfo kube-api-access-tl7sr]: timed out waiting for the condition
Warning  FailedMount  12m                   kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[kube-api-access-tl7sr istiod-ca-cert istio-token podinfo ingressgateway-ca-certs istio-envoy config-volume istio-data ingressgateway-certs]: timed out waiting for the condition
Warning  FailedMount  9m56s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istiod-ca-cert ingressgateway-certs kube-api-access-tl7sr istio-data podinfo istio-envoy config-volume istio-token ingressgateway-ca-certs]: timed out waiting for the condition
Warning  FailedMount  7m40s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[ingressgateway-certs ingressgateway-ca-certs istiod-ca-cert istio-envoy istio-token kube-api-access-tl7sr config-volume istio-data podinfo]: timed out waiting for the condition
Warning  FailedMount  5m23s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[config-volume istio-token ingressgateway-ca-certs istio-envoy istio-data ingressgateway-certs kube-api-access-tl7sr podinfo istiod-ca-cert]: timed out waiting for the condition
Warning  FailedMount  3m7s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[ingressgateway-certs podinfo ingressgateway-ca-certs kube-api-access-tl7sr istiod-ca-cert istio-data istio-envoy config-volume istio-token]: timed out waiting for the condition
Warning  FailedMount  2m41s (x17 over 21m)  kubelet            MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
Warning  FailedMount  39s (x2 over 49s)     kubelet            (combined from similar events): MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found
【3】root@k8smaster1 istio-1.13.0]# kubectl describe pod istio-egressgateway-6cf5fb4756-p4bzf  -n istio-system
Events:
Type     Reason       Age                 From               Message

Normal   Scheduled    21m                 default-scheduler  Successfully assigned istio-system/istio-egressgateway-6cf5fb4756-p4bzf to k8snode1
Warning  FailedMount  19m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istiod-ca-cert egressgateway-ca-certs config-volume istio-data kube-api-access-hlnk6 istio-envoy istio-token podinfo egressgateway-certs]: timed out waiting for the condition
Warning  FailedMount  17m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[podinfo egressgateway-ca-certs kube-api-access-hlnk6 istio-envoy istiod-ca-cert istio-token istio-data config-volume egressgateway-certs]: timed out waiting for the condition
Warning  FailedMount  15m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[egressgateway-certs istiod-ca-cert istio-token istio-data egressgateway-ca-certs istio-envoy podinfo kube-api-access-hlnk6 config-volume]: timed out waiting for the condition
Warning  FailedMount  12m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[egressgateway-ca-certs kube-api-access-hlnk6 istiod-ca-cert config-volume istio-token egressgateway-certs istio-data istio-envoy podinfo]: timed out waiting for the condition
Warning  FailedMount  10m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[podinfo kube-api-access-hlnk6 istiod-ca-cert istio-envoy config-volume istio-token istio-data egressgateway-certs egressgateway-ca-certs]: timed out waiting for the condition
Warning  FailedMount  8m22s               kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istiod-ca-cert istio-token istio-data istio-envoy config-volume egressgateway-ca-certs podinfo egressgateway-certs kube-api-access-hlnk6]: timed out waiting for the condition
Warning  FailedMount  6m6s                kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-envoy istiod-ca-cert kube-api-access-hlnk6 istio-data podinfo egressgateway-certs egressgateway-ca-certs config-volume istio-token]: timed out waiting for the condition
Warning  FailedMount  3m50s               kubelet            Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[config-volume istio-data egressgateway-certs istiod-ca-cert podinfo kube-api-access-hlnk6 istio-envoy istio-token egressgateway-ca-certs]: timed out waiting for the condition
Warning  FailedMount  93s                 kubelet            (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[istio-envoy istiod-ca-cert egressgateway-certs egressgateway-ca-certs kube-api-access-hlnk6 istio-token podinfo config-volume istio-data]: timed out waiting for the condition
Warning  FailedMount  78s (x18 over 21m)  kubelet            MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found

this might be the root cause:
Warning FailedMount 78s (x18 over 21m) kubelet MountVolume.SetUp failed for volume "istiod-ca-cert" : configmap "istio-ca-root-cert" not found

please verify is that configmap exists in istio-system namespace, also you can check is there anything wrong in istiod pod

【The problem is solved】

@1511563371  Hello, how to solve the problem? helpme please

authors: 1511563371 mpuertao zirain


text:
[release-1.16] Fix readiness probe when sidecar injector is disabled

Manual cherrypick required.
#46003 failed to apply on top of branch "release-1.16":
Applying: Fix readiness probe when sidecar injector is disabled
Applying: Refactor location of sidecar injector readiness
Using index info to reconstruct a base tree...
M	pilot/pkg/bootstrap/server.go
M	pilot/pkg/bootstrap/sidecarinjector.go
Falling back to patching base and 3-way merge...
Auto-merging pilot/pkg/bootstrap/sidecarinjector.go
CONFLICT (content): Merge conflict in pilot/pkg/bootstrap/sidecarinjector.go
Auto-merging pilot/pkg/bootstrap/server.go
CONFLICT (content): Merge conflict in pilot/pkg/bootstrap/server.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 Refactor location of sidecar injector readiness
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

The original PR was not cherry-picked to 1.16.

authors: hanxiaop istio-testing


text:
End user JWT authentication is not working in istio-1.0 release build

Setting up end user JWT auth following the Istio 1.0 docs always results in Envoy returning 503 Service Unavailable
I'm aware of the documentation bug and created the Gateway and VirtualService resource to setup ingress routing. After exposing "httpbin" service I am able to reach it via ingressgateway without any token as expected:
curl $INGRESS_HOST/headers -s -o /dev/null -w "%{http_code}\n"
200

Creating the authentication policy with the SVC_ACCOUNT & JWKS set like this:
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "example-3"
spec:
  targets:
  - name: httpbin
  peers:
  - mtls:
  origins:
  - jwt:
      issuer: $SVC_ACCOUNT
      jwksUri: $JWKS
  principalBinding: USE_ORIGIN
EOF

always returns 503 even after adding the bearer token:
curl --header "Authorization: Bearer $TOKEN" $INGRESS_HOST/headers -s -o /dev/null -w "%{http_code}\n"
503

If I update the policy spec and remove the peers mtls section like this:
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "example-3"
spec:
  targets:
  - name: httpbin
  origins:
  - jwt:
      issuer: $SVC_ACCOUNT
      jwksUri: $JWKS
  principalBinding: USE_ORIGIN
EOF

I always get 401 unauthorized even with a valid token:
curl --header "Authorization: Bearer $TOKEN" $INGRESS_HOST/headers -s -o /dev/null -w "%{http_code}\n"
401

Kubernetes version: "1.9.6"
Istio version: https://gcsweb.istio.io/gcs/istio-prerelease/daily-build/release-1.0-20180720-09-15/
Output of istioctl version:
Version: release-1.0-20180720-09-15
GitRevision: 86dc1496ef02591ce2f6b2490b161841733e1ffc

mTLS is globally turned off as mentioned in the task docs
Might be related to the issue: https://github.com/istio/istio/issues/6668

@qiwzhang  I looked at the proxy log. Looks like JWT failed with KID_ALG_UNMATCH status. Do you know what does it mean?
Btw, I update the doc to correct ingress settings. Thanks for pointing it out.

Nvm. It's because I create a test JWT token using new private key. Apparently the public key set was not updated after the new private key created. I'm not sure it is because pilot didn't refresh it, or envoy didn't change the cache. Will debug more next week.
On the other hand, the tutorial should work if the public key set was not changed. I'll do more clean up to avoid confusion. Stay tune.

previously we depends on periodically push to distribute public key to sidecars when detecting change, looks like periodically push is disabled now by default(see below code comment) ? will debug to confirm; if that's the case, TODO for proactively push needs to fixed...
https://github.com/istio/istio/blob/405f47f54d19e6d998aa3e863ec9e2bc23c157f1/pilot/pkg/proxy/envoy/v2/discovery.go#L92

@diemtvu and @quanjielin I looked at the Envoy proxy config (via localhost:15000/config_dump) and observed that the config for JWT filter is never passed from Pilot to the Envoy proxy. I was expecting to  see a JWT filter and it's config as documented here if JWT based Authentication policy is enabled.
Are you observing the same behavior?

What service account do you use? Make sure it's a valid one in your
project. You can also verify that the JWKS url with that service account
accessible and return something. (JWKS url is "
https://www.googleapis.com/service_accounts/v1/jwk/", follow by your
service account, e.g
https://www.googleapis.com/service_accounts/v1/jwk/example@my-awesome-project
.iam.gserviceaccount.com")
On Mon, Jul 23, 2018 at 8:48 AM Neeraj Poddar notifications@github.com
wrote:

@diemtvu https://github.com/diemtvu and @quanjielin
https://github.com/quanjielin I looked at the Envoy proxy config (via
localhost:15000/config_dump) and observed that the config for JWT filter is
never passed from Pilot to the Envoy proxy. I was expecting to see a JWT
filter and it's config as documented here
https://github.com/istio/proxy/tree/master/src/envoy/http/jwt_auth#add-this-filter-to-the-filter-chain
if JWT based Authentication policy is enabled.
Are you observing the same behavior?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7290#issuecomment-407105515, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AX99cUN0YA1SMdRvMwqLKaa_JBK50mwrks5uJfBngaJpZM4VZSjV
.

--
Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

Thanks for your reply. I have the correct JWKS url as documented which includes the service account. The service account is a valid account from my project in GCP.
I think Pilot from Istio version release-1.0-20180720-09-15 is not pushing JWT filter config to Envoys.

That's weird. I was testing from head, and just redone it again
with istio-release-1.0-20180723-09-15, and it all worked fine. Can you try
again with this release and let me know if it works?
If possible, can you share me your cluster. I'll try to debug from here.
Thanks.
Diem
On Mon, Jul 23, 2018 at 1:36 PM Neeraj Poddar notifications@github.com
wrote:

Thanks for your reply. I have the correct JWKS url as documented which
includes the service account. The service account is a valid account from
my project in GCP.
I think Pilot from Istio version release-1.0-20180720-09-15 is not
pushing JWT filter config to Envoys.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7290#issuecomment-407192542, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AX99cYXZcSSj3SeVEahvABJOFBpIC0dOks5uJjPJgaJpZM4VZSjV
.

--
Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

I'm not able to reproduce the behavior with istio-release-1.0-20180723-09-15

I am using the 1.0.0 release from dockerhub, and am having the same issue. There is somehow no mention of jwt in the envoy config dump.

Can you check if galley is running?
On Thu, Aug 30, 2018 at 5:39 AM Arttii notifications@github.com wrote:

I am using the 1.0.0 release from dockerhub, and am having the same issue.
There is somehow no mention of jwt in the envoy config dump.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7290#issuecomment-417304391, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AX99cbrfD9D_lyId7xt-k6q2B31pJ3iqks5uV9z4gaJpZM4VZSjV
.

--
Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

The deployment is green, but the only logs I see are
2018-08-30T14:24:32.733722Z	info	server-side configuration validation enabled
2018-08-30T14:24:32.735497Z	info	istio-galley validatingwebhookconfiguration (re)loaded:
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  creationTimestamp: null
  labels:
    component: istio-galley
    layer: base
    release: beta
  name: istio-galley
  namespace: istio-system
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: istio-galley
    uid: 0aef639b-9572-11e8-a125-067877dfc83a
webhooks:
- clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1VENDQWMyZ0F3SUJBZ0lRRDZrZVZmOWxHN1YrSTBXbS9mRmo0akFOQmdrcWhraUc5dzBCQVFzRkFEQWMKTVJvd0dBWURWUVFLRXhGck9ITXVZMngxYzNSbGNpNXNiMk5oYkRBZUZ3MHhPREEzTWpReE5EVTVOREJhRncweApPVEEzTWpReE5EVTVOREJhTUJ3eEdqQVlCZ05WQkFvVEVXczRjeTVqYkhWemRHVnlMbXh2WTJGc01JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW5Ob1pGQmozVVpjSEJ6d2R1NzdmdHdvL2VkUDYKWDBEOTNEeGQ5eDI2Tkl5d3ZOWkNHYXFGR0ZRY0FOL3o0L3FFN0tveTJYNXV0bitLOFVYUndNcnZqYVNtdUZqRQpxL2xKS1VsZ1hFSkkzVHhjT3hIakdscHRXZk1XRTRZdTVmdlplZHphTUExZlFBYjUrbjZSSmpzSXIwY0I0a0FXCjE2cWQvRkJYZi9ob05SN1pIdlZpSHgxTEdSZ1FYaW9PWU5PbnJ1MnB1bFlFajZKUmtqaVhlVDlOakFwbU42WjEKWmp2WXNLUXdjKzFzdnpRNllid0l1ZDNQbTJpbUhKclRycUZPZDlyOXVNdTFBS3FHTW91UHluaWVvMzQ1U1VtcApQTEtYYWgxTVFOTEhjc3FTSWtkeHZGRkdEODF1bjU3ZytUMDVjYTE2S3huWVFnNmhNdmZ3WmxUNkl3SURBUUFCCm95TXdJVEFPQmdOVkhROEJBZjhFQkFNQ0FnUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBUnBvdlRSYVdWVjZPemY0d0Uxc0h1WWJ4NmtpUnFnaGY2OGFIL0JPRnV2eFpCcEVubkc5Twp6NVRsTmFqSTZONWlaTnAwdEx4bXFXM2lVUUt0Y3JHWVZqZENPVHhxVEZKUlBQQ3JhSFRXZDZFL0VmY2U2NXVFClQzNXNrd0VQbUlidDdtTitVVjl1Z1Zjb1A1YlI0bmR0bzRESWtKNmdJUlNOakMxbEgrOXhCRnNWOC9NNC9mM0UKUEpibFRTZzRyeVpFYjJFZ1MwL29ZampwSndWS3llN0NKR0crcVpWMG8xNFNqQ2hCVG9nbExKZEJtSUNrTlpDNApmU0w3VGF0YzRham9zUTh6Tm50Y09KUVNhNXlHS2I0MHhyNTNJaXZkTnFoNWhmSWVUakU3dWxIczMvbkVKRlFyCnArUXZnMG9zc2VVS1poeGptbWlTNDUxZmQ4aVpLUTdtNHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    service:
      name: istio-galley
      namespace: istio-system
      path: /admitpilot
  failurePolicy: Fail
  name: pilot.validation.istio.io
  namespaceSelector: {}
  rules:
  - apiGroups:
    - config.istio.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - httpapispecs
    - httpapispecbindings
    - quotaspecs
    - quotaspecbindings
  - apiGroups:
    - rbac.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
  - apiGroups:
    - authentication.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*'
  - apiGroups:
    - networking.istio.io
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
    - destinationrules
    - envoyfilters
    - gateways
    - virtualservices
- clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1VENDQWMyZ0F3SUJBZ0lRRDZrZVZmOWxHN1YrSTBXbS9mRmo0akFOQmdrcWhraUc5dzBCQVFzRkFEQWMKTVJvd0dBWURWUVFLRXhGck9ITXVZMngxYzNSbGNpNXNiMk5oYkRBZUZ3MHhPREEzTWpReE5EVTVOREJhRncweApPVEEzTWpReE5EVTVOREJhTUJ3eEdqQVlCZ05WQkFvVEVXczRjeTVqYkhWemRHVnlMbXh2WTJGc01JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW5Ob1pGQmozVVpjSEJ6d2R1NzdmdHdvL2VkUDYKWDBEOTNEeGQ5eDI2Tkl5d3ZOWkNHYXFGR0ZRY0FOL3o0L3FFN0tveTJYNXV0bitLOFVYUndNcnZqYVNtdUZqRQpxL2xKS1VsZ1hFSkkzVHhjT3hIakdscHRXZk1XRTRZdTVmdlplZHphTUExZlFBYjUrbjZSSmpzSXIwY0I0a0FXCjE2cWQvRkJYZi9ob05SN1pIdlZpSHgxTEdSZ1FYaW9PWU5PbnJ1MnB1bFlFajZKUmtqaVhlVDlOakFwbU42WjEKWmp2WXNLUXdjKzFzdnpRNllid0l1ZDNQbTJpbUhKclRycUZPZDlyOXVNdTFBS3FHTW91UHluaWVvMzQ1U1VtcApQTEtYYWgxTVFOTEhjc3FTSWtkeHZGRkdEODF1bjU3ZytUMDVjYTE2S3huWVFnNmhNdmZ3WmxUNkl3SURBUUFCCm95TXdJVEFPQmdOVkhROEJBZjhFQkFNQ0FnUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBUnBvdlRSYVdWVjZPemY0d0Uxc0h1WWJ4NmtpUnFnaGY2OGFIL0JPRnV2eFpCcEVubkc5Twp6NVRsTmFqSTZONWlaTnAwdEx4bXFXM2lVUUt0Y3JHWVZqZENPVHhxVEZKUlBQQ3JhSFRXZDZFL0VmY2U2NXVFClQzNXNrd0VQbUlidDdtTitVVjl1Z1Zjb1A1YlI0bmR0bzRESWtKNmdJUlNOakMxbEgrOXhCRnNWOC9NNC9mM0UKUEpibFRTZzRyeVpFYjJFZ1MwL29ZampwSndWS3llN0NKR0crcVpWMG8xNFNqQ2hCVG9nbExKZEJtSUNrTlpDNApmU0w3VGF0YzRham9zUTh6Tm50Y09KUVNhNXlHS2I0MHhyNTNJaXZkTnFoNWhmSWVUakU3dWxIczMvbkVKRlFyCnArUXZnMG9zc2VVS1poeGptbWlTNDUxZmQ4aVpLUTdtNHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    service:
      name: istio-galley
      namespace: istio-system
      path: /admitmixer
  failurePolicy: Fail
  name: mixer.validation.istio.io
  namespaceSelector: {}
  rules:
  - apiGroups:
    - config.istio.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - rules
    - attributemanifests
    - circonuses
    - deniers
    - fluentds
    - kubernetesenvs
    - listcheckers
    - memquotas
    - noops
    - opas
    - prometheuses
    - rbacs
    - servicecontrols
    - solarwindses
    - stackdrivers
    - statsds
    - stdios
    - apikeys
    - authorizations
    - checknothings
    - listentries
    - logentries
    - metrics
    - quotas
    - reportnothings
    - servicecontrolreports
    - tracespans

Is your service on TCP or HTTP port? JWT is available for HTTP port only.
Best.
On Thu, Aug 30, 2018 at 8:34 AM Arttii notifications@github.com wrote:

The deployment is green, but the only logs I see are
2018-08-30T14:24:32.733722Z	info	server-side configuration validation enabled
2018-08-30T14:24:32.735497Z	info	istio-galley validatingwebhookconfiguration (re)loaded:
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
creationTimestamp: null
labels:
component: istio-galley
layer: base
release: beta
name: istio-galley
namespace: istio-system
ownerReferences:

apiVersion: extensions/v1beta1
blockOwnerDeletion: true
controller: true
kind: Deployment
name: istio-galley
uid: 0aef639b-9572-11e8-a125-067877dfc83a
webhooks:
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1VENDQWMyZ0F3SUJBZ0lRRDZrZVZmOWxHN1YrSTBXbS9mRmo0akFOQmdrcWhraUc5dzBCQVFzRkFEQWMKTVJvd0dBWURWUVFLRXhGck9ITXVZMngxYzNSbGNpNXNiMk5oYkRBZUZ3MHhPREEzTWpReE5EVTVOREJhRncweApPVEEzTWpReE5EVTVOREJhTUJ3eEdqQVlCZ05WQkFvVEVXczRjeTVqYkhWemRHVnlMbXh2WTJGc01JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW5Ob1pGQmozVVpjSEJ6d2R1NzdmdHdvL2VkUDYKWDBEOTNEeGQ5eDI2Tkl5d3ZOWkNHYXFGR0ZRY0FOL3o0L3FFN0tveTJYNXV0bitLOFVYUndNcnZqYVNtdUZqRQpxL2xKS1VsZ1hFSkkzVHhjT3hIakdscHRXZk1XRTRZdTVmdlplZHphTUExZlFBYjUrbjZSSmpzSXIwY0I0a0FXCjE2cWQvRkJYZi9ob05SN1pIdlZpSHgxTEdSZ1FYaW9PWU5PbnJ1MnB1bFlFajZKUmtqaVhlVDlOakFwbU42WjEKWmp2WXNLUXdjKzFzdnpRNllid0l1ZDNQbTJpbUhKclRycUZPZDlyOXVNdTFBS3FHTW91UHluaWVvMzQ1U1VtcApQTEtYYWgxTVFOTEhjc3FTSWtkeHZGRkdEODF1bjU3ZytUMDVjYTE2S3huWVFnNmhNdmZ3WmxUNkl3SURBUUFCCm95TXdJVEFPQmdOVkhROEJBZjhFQkFNQ0FnUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBUnBvdlRSYVdWVjZPemY0d0Uxc0h1WWJ4NmtpUnFnaGY2OGFIL0JPRnV2eFpCcEVubkc5Twp6NVRsTmFqSTZONWlaTnAwdEx4bXFXM2lVUUt0Y3JHWVZqZENPVHhxVEZKUlBQQ3JhSFRXZDZFL0VmY2U2NXVFClQzNXNrd0VQbUlidDdtTitVVjl1Z1Zjb1A1YlI0bmR0bzRESWtKNmdJUlNOakMxbEgrOXhCRnNWOC9NNC9mM0UKUEpibFRTZzRyeVpFYjJFZ1MwL29ZampwSndWS3llN0NKR0crcVpWMG8xNFNqQ2hCVG9nbExKZEJtSUNrTlpDNApmU0w3VGF0YzRham9zUTh6Tm50Y09KUVNhNXlHS2I0MHhyNTNJaXZkTnFoNWhmSWVUakU3dWxIczMvbkVKRlFyCnArUXZnMG9zc2VVS1poeGptbWlTNDUxZmQ4aVpLUTdtNHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
service:
name: istio-galley
namespace: istio-system
path: /admitpilot
failurePolicy: Fail
name: pilot.validation.istio.io
namespaceSelector: {}
rules:

apiGroups:

config.istio.io
apiVersions:
v1alpha2
operations:
CREATE
UPDATE
resources:
httpapispecs
httpapispecbindings
quotaspecs
quotaspecbindings

apiGroups:

rbac.istio.io
apiVersions:
'*'
operations:
CREATE
UPDATE
resources:
'*'

apiGroups:

authentication.istio.io
apiVersions:
'*'
operations:
CREATE
UPDATE
resources:
'*'

apiGroups:

networking.istio.io
apiVersions:
'*'
operations:
CREATE
UPDATE
resources:
destinationrules
envoyfilters
gateways
virtualservices

clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1VENDQWMyZ0F3SUJBZ0lRRDZrZVZmOWxHN1YrSTBXbS9mRmo0akFOQmdrcWhraUc5dzBCQVFzRkFEQWMKTVJvd0dBWURWUVFLRXhGck9ITXVZMngxYzNSbGNpNXNiMk5oYkRBZUZ3MHhPREEzTWpReE5EVTVOREJhRncweApPVEEzTWpReE5EVTVOREJhTUJ3eEdqQVlCZ05WQkFvVEVXczRjeTVqYkhWemRHVnlMbXh2WTJGc01JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW5Ob1pGQmozVVpjSEJ6d2R1NzdmdHdvL2VkUDYKWDBEOTNEeGQ5eDI2Tkl5d3ZOWkNHYXFGR0ZRY0FOL3o0L3FFN0tveTJYNXV0bitLOFVYUndNcnZqYVNtdUZqRQpxL2xKS1VsZ1hFSkkzVHhjT3hIakdscHRXZk1XRTRZdTVmdlplZHphTUExZlFBYjUrbjZSSmpzSXIwY0I0a0FXCjE2cWQvRkJYZi9ob05SN1pIdlZpSHgxTEdSZ1FYaW9PWU5PbnJ1MnB1bFlFajZKUmtqaVhlVDlOakFwbU42WjEKWmp2WXNLUXdjKzFzdnpRNllid0l1ZDNQbTJpbUhKclRycUZPZDlyOXVNdTFBS3FHTW91UHluaWVvMzQ1U1VtcApQTEtYYWgxTVFOTEhjc3FTSWtkeHZGRkdEODF1bjU3ZytUMDVjYTE2S3huWVFnNmhNdmZ3WmxUNkl3SURBUUFCCm95TXdJVEFPQmdOVkhROEJBZjhFQkFNQ0FnUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBUnBvdlRSYVdWVjZPemY0d0Uxc0h1WWJ4NmtpUnFnaGY2OGFIL0JPRnV2eFpCcEVubkc5Twp6NVRsTmFqSTZONWlaTnAwdEx4bXFXM2lVUUt0Y3JHWVZqZENPVHhxVEZKUlBQQ3JhSFRXZDZFL0VmY2U2NXVFClQzNXNrd0VQbUlidDdtTitVVjl1Z1Zjb1A1YlI0bmR0bzRESWtKNmdJUlNOakMxbEgrOXhCRnNWOC9NNC9mM0UKUEpibFRTZzRyeVpFYjJFZ1MwL29ZampwSndWS3llN0NKR0crcVpWMG8xNFNqQ2hCVG9nbExKZEJtSUNrTlpDNApmU0w3VGF0YzRham9zUTh6Tm50Y09KUVNhNXlHS2I0MHhyNTNJaXZkTnFoNWhmSWVUakU3dWxIczMvbkVKRlFyCnArUXZnMG9zc2VVS1poeGptbWlTNDUxZmQ4aVpLUTdtNHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
service:
name: istio-galley
namespace: istio-system
path: /admitmixer
failurePolicy: Fail
name: mixer.validation.istio.io
namespaceSelector: {}
rules:

apiGroups:

config.istio.io
apiVersions:
v1alpha2
operations:
CREATE
UPDATE
resources:
rules
attributemanifests
circonuses
deniers
fluentds
kubernetesenvs
listcheckers
memquotas
noops
opas
prometheuses
rbacs
servicecontrols
solarwindses
stackdrivers
statsds
stdios
apikeys
authorizations
checknothings
listentries
logentries
metrics
quotas
reportnothings
servicecontrolreports
tracespans

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7290#issuecomment-417362952, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AX99cXCj9OEzA6ch1aimigrVyk8ATYNsks5uWAXugaJpZM4VZSjV
.

--
Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

The Service is running an http server.
I have the following service:
---
apiVersion: v1
kind: Service
metadata:
  name: {{.component}}-service-private
  namespace: {{.namespace}}
  labels:
    release: {{.release}}
    component:  {{.component}}
    version: {{.version}}
spec:
  ports:
    - protocol: TCP
      port: {{.containerPort}}
      targetPort: {{.containerPort}}
      name: service-port
  selector:
    component: {{.component}}

And the following definitions for ingress:
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{.component}}-vsvc
  namespace:  {{.namespace}}
spec:
  hosts:
  - "{{.component}}.{{.externalDomain}}"
  gateways:
  - istio-k8s-ingress.istio-system.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: {{.containerPort}}
        host: {{.component}}-service-private
---
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: {{.component}}-magic-policy
  namespace:  {{.namespace}}
spec:
  targets:
  - name: {{.component}}-service-private
  peers:
  - mtls:
      mode: STRICT
  origins:
  - jwt:
      issuer: "gitlab"
      jwksUri: "gitlab/oauth/discovery/keys"
  principalBinding: USE_ORIGIN

---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: {{.component}}-svc-rule
  namespace:  {{.namespace}}
spec:
  host: "{{.component}}-service-private.{{.namespace}}.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

jkwsUri must start with "http://" or "https://". You should see validation
error with the policy above (please let me know if you don't).
On Thu, Aug 30, 2018 at 10:14 AM Arttii notifications@github.com wrote:

The Service is running an http server.
I have the following service:

apiVersion: v1
kind: Service
metadata:
name: {{.component}}-service-private
namespace: {{.namespace}}
labels:
release: {{.release}}
component:  {{.component}}
version: {{.version}}
spec:
ports:
- protocol: TCP
port: {{.containerPort}}
targetPort: {{.containerPort}}
name: service-port
selector:
component: {{.component}}
And the following definitions for ingress:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: {{.component}}-vsvc
namespace:  {{.namespace}}
spec:
hosts:

"{{.component}}.{{.externalDomain}}"
gateways:
istio-k8s-ingress.istio-system.svc.cluster.local
http:
match:

uri:
prefix: /
route:
destination:
port:
number: {{.containerPort}}
host: {{.component}}-service-private

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: {{.component}}-magic-policy
namespace:  {{.namespace}}
spec:
targets:

name: {{.component}}-service-private
peers:
mtls:
mode: STRICT
origins:
jwt:
issuer: "gitlab"
jwksUri: "gitlab/oauth/discovery/keys"
principalBinding: USE_ORIGIN

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: {{.component}}-svc-rule
namespace:  {{.namespace}}
spec:
host: "{{.component}}-service-private.{{.namespace}}.svc.cluster.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7290#issuecomment-417396708, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AX99cfKucgZ8NPJrTk-mysYOHFqo4v_Oks5uWB1kgaJpZM4VZSjV
.

--
Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

Same here using 1.0.1 build.  I have read all the Istio 0.8 and 1.0 docs on authentication and I am trying to implement origin authentication with a KeyCloak openid provider. I have tried repeatedly to get Istio to talk to KeyCloak and validate token that I have issued, using the following authentication config:
`
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
annotations:
name: jwt-example
spec:
origins:

jwt:
issuer: "http://192.168.100.201/auth/realms/freeipa"
jwksUri: "http://192.168.100.201/auth/realms/freeipa/protocol/openid-connect/certs"
audiences:

test
principalBinding: USE_ORIGIN
targets:

name: httpbin
peers:
mtls:
mode: STRICT
`

My ingress gateways are working without these configs and I see no warnings or errors in the pilot/discovery containers. The services are setup without the "protocol: TCP" that is mentioned above. The KeyCloak install is able to issue tokens using curl and is running on a load-balanced IP accessible to the Istio pods. The client 'test' was created in KeyCloak to and the token requested from that to map to this authentication policy.
I can use JWT with the minimal example on github as shown below but I don't see any errors showing when I add the policy or warnings to indicate where authentication is failing. I see a "upstream connect error or disconnect/reset before headers" but can't see how to locate the failing component. Any tips?
2018-09-03T11:38:38.800817Z     info    ads     XDS: Pushing 2018-09-03T11:38:38Z Services: 23, VirtualServices: 2, ConnectedEndpoints: 3 2018-09-03T11:38:38.801573Z     info    ads     Cluster init time 706.917µs 2018-09-03T11:38:38Z 2018-09-03T11:38:38.801630Z     info    ads     PushAll done 2018-09-03T11:38:38Z 26.051µs 2018-09-03T11:38:38.801847Z     info    ads     CDS: PUSH 2018-09-03T11:38:38Z for istio-egressgateway-6f8ff5df7b-6sxrh.istio-system "10.233.64.77:35496", Clusters: 42, Services 23 2018-09-03T11:38:38.802842Z     info    ads     LDS: PUSH for node:istio-egressgateway-6f8ff5df7b-6sxrh.istio-system addr:"10.233.64.77:35496" listeners:0 2018-09-03T11:38:38.803265Z     info    ads     CDS: PUSH 2018-09-03T11:38:38Z for httpbin-99bf57b99-t72gp.http1 "10.233.64.73:42386", Clusters: 43, Services 23 2018-09-03T11:38:38.808501Z     info    ads     CDS: PUSH 2018-09-03T11:38:38Z for istio-ingressgateway-6646c9d9b7-7b7nd.istio-system "10.233.64.74:41956", Clusters: 42, Services 23 2018-09-03T11:38:38.810160Z     info    ads     ADS: RDS: PUSH for node: istio-ingressgateway-6646c9d9b7-7b7nd.istio-system addr:10.233.64.74:41956 routes:1 2018-09-03T11:38:38.817339Z     info    ads     LDS: PUSH for node:istio-ingressgateway-6646c9d9b7-7b7nd.istio-system addr:"10.233.64.74:41956" listeners:1 gc 309 @7605.233s 0%: 0.079+8.1+5.3 ms clock, 0.31+5.0/7.9/0.086+21 ms cpu, 14->14->7 MB, 15 MB goal, 4 P 2018-09-03T11:38:38.828515Z     info    ads     ADS: RDS: PUSH for node: httpbin-99bf57b99-t72gp.http1 addr:10.233.64.73:42386 routes:11 2018-09-03T11:38:38.862195Z     info    ads     LDS: PUSH for node:httpbin-99bf57b99-t72gp.http1 addr:"10.233.64.73:42386" listeners:33 2018-09-03T11:38:38.862258Z     info    ads     Push finished: 63.853594ms {     "ProxyStatus": {},     "Start": "2018-09-03T11:38:38.798400504Z",     "End": "2018-09-03T11:38:38.862223644Z" } gc 310 @7605.570s 0%: 0.015+7.7+6.1 ms clock, 0.061+2.1/7.4/12+24 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 311 @7635.570s 0%: 0.043+7.7+7.4 ms clock, 0.17+3.2/7.5/2.4+29 ms cpu, 13->13->7 MB, 14 MB goal, 4 P scvg50: inuse: 13, idle: 8, sys: 22, released: 0, consumed: 22 (MB) gc 312 @7665.570s 0%: 0.018+8.0+7.6 ms clock, 0.073+0.63/7.8/9.3+30 ms cpu, 13->13->7 MB, 14 MB goal, 4 P gc 313 @7695.572s 0%: 0.22+13+3.7 ms clock, 0.88+4.3/13/16+15 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 314 @7725.572s 0%: 0.053+9.3+5.9 ms clock, 0.21+2.9/8.0/7.9+23 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 315 @7755.573s 0%: 0.036+8.6+5.4 ms clock, 0.14+1.6/8.3/13+21 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 316 @7786.469s 0%: 0.043+8.7+6.2 ms clock, 0.17+1.0/8.2/3.8+24 ms cpu, 13->13->7 MB, 14 MB goal, 4 P scvg51: inuse: 13, idle: 8, sys: 22, released: 0, consumed: 22 (MB) gc 317 @7815.571s 0%: 0.11+9.7+5.8 ms clock, 0.45+4.5/7.8/16+23 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 318 @7845.572s 0%: 0.096+5.6+5.2 ms clock, 0.38+0/5.4/14+21 ms cpu, 13->13->7 MB, 14 MB goal, 4 P gc 319 @7875.572s 0%: 0.033+13+3.7 ms clock, 0.13+3.9/13/16+14 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 320 @7905.572s 0%: 0.033+13+4.2 ms clock, 0.13+1.1/12/25+16 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 321 @7935.618s 0%: 0.086+11+5.1 ms clock, 0.34+0.36/10/11+20 ms cpu, 13->14->7 MB, 14 MB goal, 4 P scvg52: 1 MB released scvg52: inuse: 13, idle: 8, sys: 22, released: 1, consumed: 20 (MB) gc 322 @7965.839s 0%: 0.033+9.0+4.8 ms clock, 0.13+0.34/7.4/12+19 ms cpu, 13->14->7 MB, 14 MB goal, 4 P gc 323 @7995.570s 0%: 0.015+12+6.4 ms clock, 0.062+0.63/12/18+25 ms cpu, 13->13->7 MB, 14 MB goal, 4 P gc 324 @8025.571s 0%: 0.049+13+6.5 ms clock, 0.19+1.1/12/15+26 ms cpu, 13->13->7 MB, 14 MB goal, 4 P gc 325 @8055.570s 0%: 0.058+16+4.1 ms clock, 0.23+0.59/8.7/10+16 ms cpu, 13->13->7 MB, 14 MB goal, 4 P gc 326 @8085.585s 0%: 0.017+5.9+5.4 ms clock, 0.068+1.9/5.7/6.9+21 ms cpu, 13->14->7 MB, 14 MB goal, 4 P scvg53: 0 MB released scvg53: inuse: 13, idle: 8, sys: 22, released: 2, consumed: 19 (MB) gc 327 @8115.571s 0%: 0.035+7.0+6.1 ms clock, 0.14+2.1/6.7/6.3+24 ms cpu, 13->13->7 MB, 14 MB goal, 4 P gc 328 @8146.320s 0%: 0.093+9.6+4.5 ms clock, 0.37+1.8/8.3/11+18 ms cpu, 13->13->7 MB, 14 MB goal, 4 P
`
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "jwt-example"
spec:
targets:

name: httpbin
origins:
jwt:
issuer: "testing@secure.istio.io"
jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.0/security/tools/jwt/samples/jwks.json"
principalBinding: USE_ORIGIN
`

Did you get this working?
I am facing the same issue. I could get the sample https://istio.io/help/ops/security/end-user-auth/ working only if I have the jwks endpoint in a public url (like https://gist.githubusercontent.com/mefarazath/0757201b14de6f2a3c01b62d39648367/raw/1a37b704825f64f0763dfc02dfabc1af44fb344b/test-jwt.jwks)
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: "jwt-example"
spec:
  targets:
  - name: httpbin
  origins:
  - jwt:
      issuer: "https://localhost:9443/oauth2/token"
      jwksUri: "https://gist.githubusercontent.com/mefarazath/0757201b14de6f2a3c01b62d39648367/raw/1a37b704825f64f0763dfc02dfabc1af44fb344b/test-jwt.jwks"
  principalBinding: USE_ORIGIN

I tried running an oauth2 server within the k8s cluster and setting the jwks uri using, service name, NodePort etc. but could not get it to work... In the error scenarios I see the below in the logs
[2018-09-12 12:26:12.571][17][warning][config] bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener 172.17.0.13_8000: Proto constraint validation failed (JwtAuthenticationValidationError.Rules[i]: ["embedded message failed validation"] | caused by JwtRuleValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): rules {
  issuer: "https://localhost:9443/oauth2/token"
  local_jwks {
    inline_string: ""
  }
  forward: true
  forward_payload_header: "istio-sec-948f5aaa926a8a4d63e3ad5bf058ce381a69561e"
}
allow_missing_or_failed: true

This error means pilot cannot fetch the public key set from the given
JwksUri. If you use https, your server certificate must be signed by a CA
that is verifiable. We use CA bundle that come with standard linux distro,
so it should work all common PKI. In other words, self-signing certificate
will not work.
Hope this helps.
Thanks.
Diem
On Wed, Sep 12, 2018 at 5:36 AM Farasath Ahamed notifications@github.com
wrote:

Did you get this working?
I am facing the same issue. I could get the sample
https://istio.io/help/ops/security/end-user-auth/ working only if I have
the jwks endpoint in a public url (like
https://gist.githubusercontent.com/mefarazath/0757201b14de6f2a3c01b62d39648367/raw/1a37b704825f64f0763dfc02dfabc1af44fb344b/test-jwt.jwks
)
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
name: "jwt-example"
spec:
targets:

name: httpbin
origins:
jwt:
issuer: "https://localhost:9443/oauth2/token"
jwksUri: "https://gist.githubusercontent.com/mefarazath/0757201b14de6f2a3c01b62d39648367/raw/1a37b704825f64f0763dfc02dfabc1af44fb344b/test-jwt.jwks"
principalBinding: USE_ORIGIN

I tried running an oauth2 server within the k8s cluster and setting the
jwks uri using, service name, NodePort etc. but could not get it to work...
In the error scenarios I see the below in the logs
[2018-09-12 12:26:12.571][17][warning][config] bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:70] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener 172.17.0.13_8000: Proto constraint validation failed (JwtAuthenticationValidationError.Rules[i]: ["embedded message failed validation"] | caused by JwtRuleValidationError.LocalJwks: ["embedded message failed validation"] | caused by DataSourceValidationError.InlineString: ["value length must be at least " '\x01' " bytes"]): rules {
issuer: "https://localhost:9443/oauth2/token"
local_jwks {
inline_string: ""
}
forward: true
forward_payload_header: "istio-sec-948f5aaa926a8a4d63e3ad5bf058ce381a69561e"
}
allow_missing_or_failed: true
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/7290#issuecomment-420632018, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AX99cf2ZdjL_8wBO3sNiguifAmu01dNXks5uaP_jgaJpZM4VZSjV
.

--
Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

@diemtvu Exactly!
The jwks uri I had in the policy was a HTTPS endpoint secured with a self-signed cert. Switched to using the http endpoint and everything worked fine for me :smile:
Thanks a lot!

authors: Arttii diemtvu howels mefarazath nrjpoddar quanjielin