text:
httpproxy fail when use istio

(This is used to report product bugs, please visit https://discuss.istio.io for questions on using Istio)
Bug description
I have a app writed in java, and the app access X site use a proxy http://www.myproxy.com:8080.
I start the app use "java -Dhttps.proxyHost=www.myproxy.com -Dhttps.proxyPort=8080 -jar myapp".
It runs well in k8s without istio.
when i inject the istio sidecar into the app (without any serviceEntry), then it can not access X site by proxy http://www.myproxy.com:8080, it's response status is 404.
Expected behavior
the app can access external site by specify proxy
Version (include the output of istioctl version --remote and kubectl version)
kubectl version:
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
istioctl version --remote:
client version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.4-10-g9b6d31b"}
citadel version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b-dirty", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.4-10-g9b6d31b"}
galley version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b-dirty", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.4-10-g9b6d31b"}
ingressgateway version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.4-10-g9b6d31b"}
pilot version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b-dirty", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.4-10-g9b6d31b"}
policy version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b-dirty", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.4-10-g9b6d31b"}
sidecar-injector version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b-dirty", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.4-10-g9b6d31b"}
telemetry version: version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b-dirty", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.4-10-g9b6d31b"}
How was Istio installed?
install use helm
Environment where bug was observed (cloud vendor, OS, etc)
CentOS Linux release 7.6.1810 (Core)

kubectl get configmap istio -n istio-system -o yaml | grep -o "mode: ALLOW_ANY"
mode: ALLOW_ANY

use a ServiceEntry allow X site and www.myproxy.com:8080 also faild

@mofee
In our experimental setup, we are perhaps seeing a manifestation of this issue.
a) Does your app attempt to connect to the proxy only once, i.e. soon after the Pod is Running?
b) If your app connects to the proxy on-demand or periodically sometime after the Pod has been Running, then what happens to these connection attempts?
c) I do not believe that this issue has to do anything with connecting to a proxy.
In our setup, we have a Pod which attempts to curl GET sample data from a test server. All attempts succeed to get the data from the server succeed.
We see an issue in a variant of the same Pod which has the istio-sidecar manually injected. The first attempt always fails because curl is unable to connect to the test server (connection refused). However, subsequent attempts succeed.
With access logging enabled, in istio-proxy, I see that the number of outbound GET requests logged correspond to the number of successful attempts by curl.
I am debugging this further, and shall post the findings/results.

Attached are:
(a) the logs from the pod/container that attempts to curl GET sample/test data from a server
(b) logs from the proxy container (with --proxyLogLevel=trace, and hence a bit noisy)
[server/host names and public IP addresses changed to preserve identities, but length preserved]

TCP stats and iptables stats are below.
Container logs show that the first connection attempt has failed, while the subsequent 4 attempts succeed. The first connection attempt is made even before the proxy container logs that an active listener for 15001 has been added.
Proxy container logs show that the listener is aware of four requests (post a listener has been active on 15001).

Tcp:
13 active connections openings
82 passive connection openings
2 failed connection attempts
0 connection resets received
4 connections established
546 segments received
715 segments send out
1 segments retransmited
0 bad segments received.
2 resets sent
root@istio-curler-pod:~# iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 95 packets, 5700 bytes)
pkts bytes target     prot opt in     out     source               destination
Chain INPUT (policy ACCEPT 95 packets, 5700 bytes)
pkts bytes target     prot opt in     out     source               destination
Chain OUTPUT (policy ACCEPT 50 packets, 4232 bytes)
pkts bytes target     prot opt in     out     source               destination
13   780 ISTIO_OUTPUT  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
Chain POSTROUTING (policy ACCEPT 56 packets, 4592 bytes)
pkts bytes target     prot opt in     out     source               destination
Chain ISTIO_IN_REDIRECT (0 references)
pkts bytes target     prot opt in     out     source               destination
0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            redir ports 15001
Chain ISTIO_OUTPUT (1 references)
pkts bytes target     prot opt in     out     source               destination
0     0 ISTIO_REDIRECT  all  --  *      lo      0.0.0.0/0           !127.0.0.1
7   420 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            owner UID match 1337
0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            owner GID match 1337
0     0 RETURN     all  --  *      *       0.0.0.0/0            127.0.0.1
6   360 ISTIO_REDIRECT  all  --  *      *       0.0.0.0/0            0.0.0.0/0
curler.3.log
curler.3.proxy.log

@g-p-n , thanks for your reply.
I also test use a curl pod, the image from https://hub.docker.com/r/appropriate/curl, and be injected the istio sidecar with  --proxyLogLevel=trace.
Then i sue cmd:
curl -x http://hkproxy.global.test.com:8080 https://www.google.com
and the response is
curl: (56) Received HTTP code 404 from proxy after CONNECT
the istio-proxy trace log file is here
I can see the envoy add some headers in http connect, i don't know if it is the cause
`[2019-06-25 07:12:48.248][26][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:363] [C5] parsing 120 bytes
[2019-06-25 07:12:48.248][26][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:476] [C5] message begin
[2019-06-25 07:12:48.249][26][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:243] [C5] new stream
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:331] [C5] completed header: key=Host value:443=www.google.com
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:331] [C5] completed header: key=User-Agent value=curl/7.59.0
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:442] [C5] headers complete
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:331] [C5] completed header: key=Proxy-Connection value=Keep-Alive
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:463] [C5] message complete
[2019-06-25 07:12:48.249][26][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:580] [C5][S5680453476879491337] request headers complete (end_stream=true):
':authority', 'www.google.com:443'
':path', 'www.google.com:443'
':method', 'CONNECT'
'user-agent', 'curl/7.59.0'
'proxy-connection', 'Keep-Alive'
[2019-06-25 07:12:48.249][26][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1040] [C5][S5680453476879491337] request end stream
[2019-06-25 07:12:48.249][26][debug][filter] [src/envoy/http/mixer/filter.cc:39] Called Mixer::Filter : Filter
[2019-06-25 07:12:48.249][26][debug][filter] [src/envoy/http/mixer/filter.cc:146] Called Mixer::Filter : setDecoderFilterCallbacks
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1200] [C5][S5680453476879491337] encode headers called: filter=0x2ecfd60 status=0
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1200] [C5][S5680453476879491337] encode headers called: filter=0x2ecfcc0 status=0
[2019-06-25 07:12:48.249][26][debug][filter] [src/envoy/http/mixer/filter.cc:133] Called Mixer::Filter : encodeHeaders 0
[2019-06-25 07:12:48.249][26][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1200] [C5][S5680453476879491337] encode headers called: filter=0x2ecfa40 status=0
[2019-06-25 07:12:48.249][26][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1305] [C5][S5680453476879491337] encoding headers via codec (end_stream=true):
':status', '404'
'date', 'Tue, 25 Jun 2019 07:12:48 GMT'
'server', 'envoy'
`

@mofee
Thank you for the traces. Shall take a look at it.
Will I be able to use http://hkproxy.global.test.com:8080 to reproduce the issue locally?

@g-p-n  http://hkproxy.global.test.com:8080 is our internal  http proxy, can not be accessed for you.

@mofee
Never mind about my request to use your proxy server.
I able to reproduce the issue.

@g-p-n
Never mind, but it's a  internal ip [192.168.195.9]
i have tried visiting a http website
curl -x http://hkproxy.global.test.com:8080  http://www.website.org/
the curl not print the error message, but also can not received the response, nothing print out.

@mofee
Your logs with trace enabled helped.
Here is my investigation. In summary, I see that envoy does not support CONNECT.
We could wait to hear from the istio/proxy/envoy experts.
This is the output from curl.
*   Trying AA.BB.CCC.DDD...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x561e653037a0)
* Connected to someproxy.com (AA.BB.CCC.DDD) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
> Host: www.google.com:443
> User-Agent: curl/7.64.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< date: Tue, 25 Jun 2019 13:41:53 GMT
date: Tue, 25 Jun 2019 13:41:53 GMT
< server: envoy
server: envoy
< content-length: 0
content-length: 0
<

* Received HTTP code 404 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 404 from proxy after CONNECT

Here is the corresponding output of the parsing from istio-proxy.
[external/envoy/source/common/http/conn_manager_impl.cc:580] [C5][S5680453476879491337] request headers complete (end_stream=true):
':authority', 'www.google.com:443'
':path', 'www.google.com:443'
':method', 'CONNECT'
'user-agent', 'curl/7.59.0'
'proxy-connection', 'Keep-Alive'

In this case of a request via a proxy, the first HTTP command/method issued is CONNECT. Also, the "path" above is really not a relative path.
Hence, I believe we are running into this code segment, which is sending an HTTP 404 as the response to the client (in this case curl). I suspect the same is happening to your Java application too.
  // Currently we only support relative paths at the application layer. We expect the codec to have
  // broken the path into pieces if applicable. NOTE: Currently the HTTP/1.1 codec only does this
  // when the allow_absolute_url flag is enabled on the HCM.
  // https://tools.ietf.org/html/rfc7230#section-5.3 We also need to check for the existence of
  // :path because CONNECT does not have a path, and we don't support that currently.
  if (!request_headers_->Path() || request_headers_->Path()->value().getStringView().empty() ||
      request_headers_->Path()->value().getStringView()[0] != '/') {
    const bool has_path =
        request_headers_->Path() && !request_headers_->Path()->value().getStringView().empty();
    connection_manager_.stats_.named_.downstream_rq_non_relative_path_.inc();
    sendLocalReply(Grpc::Common::hasGrpcContentType(*request_headers_), Code::NotFound, "", nullptr,
                   is_head_request_, absl::nullopt,
                   has_path ? StreamInfo::ResponseCodeDetails::get().AbsolutePath
                            : StreamInfo::ResponseCodeDetails::get().MissingPath);
    return;
  }

in https://github.com/envoyproxy/envoy/blob/master/source/common/http/conn_manager_impl.cc.
Hence, this log in istio-proxy seen in my setup.
[2019-06-25 13:41:53.689][23][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1137] [C6][S2763635160165889090] Sending local reply with details absolute_path_rejected

@mofee
Digging further, it looks like envoy can perform as a HTTP proxy. Here are the links to the envoy documentation and configuration.
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http_proxy#arch-overview-http-dynamic-forward-proxy
https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/dynamic_forward_proxy_filter#config-http-filters-dynamic-forward-proxy
However, the envoy documentation clearly states that this feature is to be considered as alpha and not production ready.
I haven't tried this out myself.

does the application have to access X site only using myproxy? can it access the site using the local sidecar's HTTP proxy ?

@g-p-n  thank you, i will try follow the links
@rshriram , for some reasons, our apps can not access the internet except by the http proxy

today, i tried to config the envoy dynamic forward proxy, but it required that

filter should be configured with the name envoy.filters.http.dynamic_forward_proxy
2.Both filter and cluster must be configured together and point to the same DNS cache parameters for Envoy to operate as an HTTP dynamic forward proxy

i can config the http filter via the istio envoy filter, how can i config cluster  use istio?

authors: g-p-n mofee rshriram


text:
Remove test parameters from istioctl

Recently, many new testing arguments now appear in istioctl
istioctl --help
Shows the old stuff, then
      --test.bench regexp               run only benchmarks matching regexp
      --test.benchmem                   print memory allocations for benchmarks
      --test.benchtime d                run each benchmark for duration d (default 1s)
      --test.blockprofile file          write a goroutine blocking profile to file
      --test.blockprofilerate rate      set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
      --test.count n                    run tests and benchmarks n times (default 1)
      --test.coverprofile file          write a coverage profile to file
      --test.cpu list                   comma-separated list of cpu counts to run each test with
      --test.cpuprofile file            write a cpu profile to file
      --test.failfast                   do not start new tests after the first test failure
      --test.list regexp                list tests, examples, and benchmarks matching regexp then exit
      --test.memprofile file            write an allocation profile to file
      --test.memprofilerate rate        set memory allocation profiling rate (see runtime.MemProfileRate)
      --test.mutexprofile string        write a mutex contention profile to the named file after execution
      --test.mutexprofilefraction int   if >= 0, calls runtime.SetMutexProfileFraction() (default 1)
      --test.outputdir dir              write profiles to dir
      --test.parallel n                 run at most n tests in parallel (default 8)
      --test.run regexp                 run only tests and examples matching regexp
      --test.short                      run smaller test suite to save time
      --test.testlogfile file           write test action log to file (for use only by cmd/go)
      --test.timeout d                  panic test binary after duration d (default 0, timeout disabled) (default 0s)
      --test.trace file                 write an execution trace to file
      --test.v                          verbose: print additional output

I noticed the same on pilot-discovery. Not something people really use, but still shouldn't be there

If you go to the istioctl/cmd/istioctl directory, run "go build", and then run "./istioctl --hep", you will not see the test-related options.

John, mind taking a look at this please?
Thanks.

Yep, think I found the root cause, will send a fix soon

We definitely are not building binaries with -race, if this somehow snuck in since CPU/memory usage would be extremely high

source: gharchive/issue

authors: esnible geeknoid howardjohn

repo: istio/istio

license_type: permissive

license_source: github-api


text:
Ingress SDS missing event to push key/cert pair to proxy

Bug description
In rare cases we're seeing that SDS is missing an event entirely that's causing only the root cert to get pushed but not the key/cert pair. Looking through the logs we can see that the root cert gets pushed out once the secret is eventually found:
2019-11-12T22:05:53.335991Z	warn	secretFetcherLog	Cannot find secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert, searching for fallback secret gateway-fallback
2019-11-12T22:05:53.336191Z	error	secretFetcherLog	cannot find secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert and cannot find fallback secret gateway-fallback
2019-11-12T22:05:53.336244Z	warn	cacheLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-112, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert, EVENT: SecretFetcher cannot find secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert from cache
2019-11-12T22:05:53.336308Z	warn	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-112, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert, EVENT: waiting for ingress gateway secret for proxy "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local"

2019-11-12T22:05:53.336165Z	warn	secretFetcherLog	Cannot find secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, searching for fallback secret gateway-fallback
2019-11-12T22:05:53.336362Z	error	secretFetcherLog	cannot find secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b and cannot find fallback secret gateway-fallback
2019-11-12T22:05:53.336382Z	warn	cacheLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-113, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, EVENT: SecretFetcher cannot find secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b from cache
2019-11-12T22:05:53.336391Z	warn	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-113, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, EVENT: waiting for ingress gateway secret for proxy "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local"

2019-11-12T22:05:59.991770Z	error	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, EVENT: NotifyProxy failed. No connection with id "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111" can be found
2019-11-12T22:05:59.991797Z	error	cacheLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, EVENT: failed to notify secret change for proxy: no connection with id "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111" can be found
2019-11-12T22:05:59.991894Z	info	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-112, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert, EVENT: pushed root cert to proxy

You'll see that no key/cert pair gets pushed. This causes TLS/mTLS gateways to break completely.
The way to recover from this is to delete the secret and add it back again. This results in the following logs:
2019-11-12T23:09:37.570981Z	info	secretFetcherLog	secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b is deleted
2019-11-12T23:09:37.571063Z	error	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, EVENT: NotifyProxy failed. No connection with id "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111" can be found
2019-11-12T23:09:37.571075Z	error	cacheLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, EVENT: failed to notify secret change for proxy: no connection with id "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-111" can be found
2019-11-12T23:09:37.571085Z	info	secretFetcherLog	secret 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert is deleted
2019-11-12T23:09:37.571104Z	error	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-110, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert, EVENT: NotifyProxy failed. No connection with id "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-110" can be found
2019-11-12T23:09:37.571115Z	error	cacheLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-110, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert, EVENT: failed to notify secret change for proxy: no connection with id "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-110" can be found
2019-11-12T23:10:14.200253Z	info	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-112, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b-cacert, EVENT: pushed root cert to proxy

2019-11-12T23:10:14.200251Z	info	sdsServiceLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-113, RESOURCE NAME: 975cfa2b-9e3f-4c4b-8eb6-2c810aad271b, EVENT: pushed key/cert pair to proxy

Expected behavior
SDS always pushing the required crypto to the proxy.
Steps to reproduce the bug
Unknown, I've found this very hard to reproduce. It has only affected one user of ours AFAIK.
Version (include the output of istioctl version --remote and kubectl version)
client version: unknown
control plane version: 1.3.0

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:41:55Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.11-eks-5876d6", GitCommit:"5876d6b7429820450950ade17fe7b4bf5ccada7f", GitTreeState:"clean", BuildDate:"2019-09-24T20:54:25Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Imagine having a Makefile that does this where $ISTIO_VERSION is 1.3.0:
@curl -s -L https://git.io/getLatestIstio | ISTIO_VERSION=$(ISTIO_VERSION) sh -
@cat istio/istio-namespace.yaml > istio/istio.yaml
@cat istio/istio-namespace.yaml > istio/istio-crds.yaml
@helm template istio-$(ISTIO_VERSION)/install/kubernetes/helm/istio-init --name istio-init --namespace istio-system >> istio/istio-crds.yaml
@helm template istio-$(ISTIO_VERSION)/install/kubernetes/helm/istio --name istio --namespace istio-system \
    --set sidecarInjectorWebhook.rewriteAppHTTPProbe=true \
    --set servicegraph.enabled=true \
    --set gateways.istio-ingressgateway.sds.enabled=true \
    --set tracing.enabled=true \
    --set kiali.enabled=true \
    --set "kiali.dashboard.jaegerURL=http://jaeger-query:16686" \
    --set "kiali.dashboard.grafanaURL=http://grafana:3000" \
    --set grafana.enabled=true >> istio/istio.yaml
@cat istio-$(ISTIO_VERSION)/install/kubernetes/helm/istio-init/files/crd-1* > istio/istio-delete-crds.yaml
@rm -rf istio-$(ISTIO_VERSION)

A kubectl apply is later done on the istio-crds.yaml and istio.yaml files.
Environment where bug was observed (cloud vendor, OS, etc)
EKS

I vaguely recall this being a known bug that has been fixed in one of the 1.3 patch releases. I highly recommend you upgrade to 1.3.5 which has a number of high severity fixed.
@JimmyCYJ can verify if this was fixed

Thanks! I couldn't find any reference to this in the release notes. @JimmyCYJ can you advise?

Thanks @howardjohn! Yes, there is a bug in Citadel agent in 1.3.0 release. If your Cidatel agent prints logs such as "warn	secretFetcherLog	unexpected server key/cert change in secret kyma-gateway-certs", then it indicates that Citadel agent hits this bug.
The release notes is here https://istio.io/news/2019/announcing-1.3.4/#minor-enhancements.
Tracking issue is here https://github.com/istio/istio/issues/17409.
Please upgrade to 1.3.5.

Thanks @JimmyCYJ. We're not see those log anywhere in our Citadel instance. I also don't see failed load server cert/key pair in our logs. Could it be another issue we're hitting?

Thanks @JimmyCYJ. I'm not seeing logs like that anywhere in our setup. We are seeing these logs from our SDS container though:
2019-11-13T22:46:36.022211Z	error	cacheLog	CONNECTION ID: router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-106, RESOURCE NAME: 6dd40d9f-0f30-481f-9c12-086715e685d9, EVENT: failed to notify secret change for proxy: no connection with id "router~192.168.54.237~istio-ingressgateway-66b78f978b-njr66.istio-system~istio-system.svc.cluster.local-106" can be found

Does it look like a different issue to you?

I'm testing this against a cluster that's been upgraded to 1.3.5, but I find the original issue is very hard to reproduce. Is there something specific that I can exercise to hit the case above where SDS loses track of the connection?

I'm closing this because @JimmyCYJ provide a suggestion to upgrade to 1.3.5
Please feel free to re-open if you can reproduce after upgrading to 1.3.5

I still see this issue with 1.3.5. Can we reopen this? Here's how I'm able to repro:
Create a gateway that uses a secret like so:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: apa0401f02e6d286f2f0678eff4046e11571e58
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*.apa0401f02e6d286f2f0678eff4046e11571e58.myurl.com'
    port:
      name: https-mutual
      number: 443
      protocol: HTTPS
    tls:
      credentialName: apa0401f02e6d286f2f0678eff4046e11571e58
      mode: MUTUAL

Create the secret that the gateway uses:
apiVersion: v1
data:
  cacert: cacertdatahere
  cert: certdatahere
  key: secretdatahere
kind: Secret
metadata:
  name: apa0401f02e6d286f2f0678eff4046e11571e58
  namespace: istio-system
type: Generic

Now delete the secret and gateway and create them again using the exact same values. This causes SDS to fail to push things over to Envoy. In 1.3.5 I see that even the root cert doesn't make it to Envoy so the TLS endpoint is completely broken:
2019-11-19T05:45:20.765756Z	warn	secretFetcherLog	Cannot find secret apa0401f02e6d286f2f0678eff4046e11571e58-cacert, searching for fallback secret gateway-fallback
2019-11-19T05:45:20.765784Z	error	secretFetcherLog	cannot find secret apa0401f02e6d286f2f0678eff4046e11571e58-cacert and cannot find fallback secret gateway-fallback
2019-11-19T05:45:20.765791Z	warn	cacheLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-18, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58-cacert, EVENT: SecretFetcher cannot find secret apa0401f02e6d286f2f0678eff4046e11571e58-cacert from cache
2019-11-19T05:45:20.765800Z	warn	sdsServiceLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-18, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58-cacert, EVENT: waiting for ingress gateway secret for proxy "router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local"

2019-11-19T05:45:20.769565Z	warn	secretFetcherLog	Cannot find secret apa0401f02e6d286f2f0678eff4046e11571e58, searching for fallback secret gateway-fallback
2019-11-19T05:45:20.769693Z	error	secretFetcherLog	cannot find secret apa0401f02e6d286f2f0678eff4046e11571e58 and cannot find fallback secret gateway-fallback
2019-11-19T05:45:20.769723Z	warn	cacheLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-17, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58, EVENT: SecretFetcher cannot find secret apa0401f02e6d286f2f0678eff4046e11571e58 from cache
2019-11-19T05:45:20.769739Z	warn	sdsServiceLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-17, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58, EVENT: waiting for ingress gateway secret for proxy "router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local"

2019-11-19T05:45:41.187776Z	error	sdsServiceLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-16, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58, EVENT: NotifyProxy failed. No connection with id "router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-16" can be found
2019-11-19T05:45:41.187808Z	error	cacheLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-16, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58, EVENT: failed to notify secret change for proxy: no connection with id "router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-16" can be found
2019-11-19T05:45:41.187980Z	error	sdsServiceLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-15, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58-cacert, EVENT: NotifyProxy failed. No connection with id "router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-15" can be found
2019-11-19T05:45:41.187996Z	error	cacheLog	CONNECTION ID: router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-15, RESOURCE NAME: apa0401f02e6d286f2f0678eff4046e11571e58-cacert, EVENT: failed to notify secret change for proxy: no connection with id "router~100.96.4.216~istio-ingressgateway-6985dc855-qgc4z.istio-system~istio-system.svc.cluster.local-15" can be found

@aaronjwood

I've also seen this issue with 1.3.5.

I see the same for 1.4.0 (kube 1.16.2):
istioctl version
client version: 1.4.0
control plane version: 1.4.0
kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
While SIMPLE tls for ingress works fine, MUTUAL tls realized according to task https://istio.io/docs/tasks/traffic-management/ingress/secure-ingress-sds/ behaves like in the issue

@aaronjwood could you turn on debug logging at ingress-sds? You can do it via /ctrlz endpoint. Please share a copy of the log, or send me a copy by slack channel. From the piece of log you shared, ingress-sds does not detect the secret apa0401f02e6d286f2f0678eff4046e11571e58. I need to figure out why it fails to detect the secret.

Okay I ran through this scenario again with cacheLog, sdsServiceLog, and secretFetcher debug logs turned on:
2019-12-12T00:26:25.294660Z	debug	sdsServiceLog	start staled connection cleanup job
2019-12-12T00:26:25.294729Z	debug	sdsServiceLog	remove staled clients {ConnectionID:router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-6 ResourceName:awood1018db10718dc0e1994886056a23fd7b9b1-cacert}
2019-12-12T00:26:25.294748Z	debug	sdsServiceLog	remove staled clients {ConnectionID:router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-4 ResourceName:awood1018db10718dc0e1994886056a23fd7b9b1}
2019-12-12T00:26:25.294756Z	debug	sdsServiceLog	remove staled clients {ConnectionID:router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-3 ResourceName:awood1018db10718dc0e1994886056a23fd7b9b1-cacert}
2019-12-12T00:26:25.294768Z	debug	sdsServiceLog	remove staled clients {ConnectionID:router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5 ResourceName:awood1018db10718dc0e1994886056a23fd7b9b1}
2019-12-12T00:26:25.294660Z	debug	sdsServiceLog	start staled connection cleanup job
2019-12-12T00:26:58.677632Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-7, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: add a new connection
2019-12-12T00:26:58.677691Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-7, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: received first SDS request from proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local", version info "", error details nil

2019-12-12T00:26:58.677711Z	debug	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-7, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: calling SecretFetcher to search for secret awood1018db10718dc0e1994886056a23fd7b9b1
2019-12-12T00:26:58.677717Z	debug	secretFetcherLog	SecretFetcher search for secret awood1018db10718dc0e1994886056a23fd7b9b1
2019-12-12T00:26:58.677724Z	debug	secretFetcherLog	load secret awood1018db10718dc0e1994886056a23fd7b9b1 from secret fetcher: false
2019-12-12T00:26:58.677847Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: add a new connection
2019-12-12T00:26:58.677897Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: received first SDS request from proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local", version info "", error details nil

2019-12-12T00:26:58.677912Z	debug	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: calling SecretFetcher to search for secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert
2019-12-12T00:26:58.677918Z	debug	secretFetcherLog	SecretFetcher search for secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert
2019-12-12T00:26:58.677924Z	debug	secretFetcherLog	load secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert from secret fetcher: false
2019-12-12T00:26:58.684456Z	warn	secretFetcherLog	Cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1, searching for fallback secret gateway-fallback
2019-12-12T00:26:58.684523Z	error	secretFetcherLog	cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1 and cannot find fallback secret gateway-fallback
2019-12-12T00:26:58.684539Z	warn	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-7, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: SecretFetcher cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1 from cache
2019-12-12T00:26:58.684549Z	warn	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-7, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: waiting for ingress gateway secret for proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local"

2019-12-12T00:26:58.684601Z	warn	secretFetcherLog	Cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert, searching for fallback secret gateway-fallback
2019-12-12T00:26:58.684608Z	error	secretFetcherLog	cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert and cannot find fallback secret gateway-fallback
2019-12-12T00:26:58.684612Z	warn	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: SecretFetcher cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert from cache
2019-12-12T00:26:58.684619Z	warn	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: waiting for ingress gateway secret for proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local"

2019-12-12T00:27:13.477775Z	debug	secretFetcherLog	secret awood1018db10718dc0e1994886056a23fd7b9b1 is added as a server certificate
2019-12-12T00:27:13.477822Z	debug	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: secret cache is updated
2019-12-12T00:27:13.477864Z	error	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: NotifyProxy failed. No connection with id "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5" can be found
2019-12-12T00:27:13.477881Z	error	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: failed to notify secret change for proxy: no connection with id "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5" can be found
2019-12-12T00:27:13.477900Z	debug	secretFetcherLog	secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert is added as a client CA cert (from a compound Secret)
2019-12-12T00:27:13.477922Z	debug	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-6, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: secret cache is updated
2019-12-12T00:27:13.477948Z	error	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-6, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: NotifyProxy failed. No connection with id "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-6" can be found
2019-12-12T00:27:13.477957Z	error	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-6, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: failed to notify secret change for proxy: no connection with id "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-6" can be found
2019-12-12T00:28:33.326025Z	info	secretFetcherLog	secret awood1018db10718dc0e1994886056a23fd7b9b1 is deleted
2019-12-12T00:28:33.326115Z	debug	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: secret cache is deleted
2019-12-12T00:28:33.326201Z	error	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: NotifyProxy failed. No connection with id "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5" can be found
2019-12-12T00:28:33.326284Z	error	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1, EVENT: failed to notify secret change for proxy: no connection with id "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-5" can be found
2019-12-12T00:28:33.326325Z	info	secretFetcherLog	secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert is deleted
2019-12-12T00:28:33.326356Z	debug	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: secret cache is deleted
2019-12-12T00:28:33.326413Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: received push channel request for proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local"
2019-12-12T00:28:33.326447Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: close connection for proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local"
2019-12-12T00:28:33.326551Z	info	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-8, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: connection is terminated: rpc error: code = Canceled desc = context canceled
2019-12-12T00:28:33.400910Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-9, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: add a new connection
2019-12-12T00:28:33.400968Z	debug	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-9, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: received first SDS request from proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local", version info "", error details nil

2019-12-12T00:28:33.400991Z	debug	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-9, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: calling SecretFetcher to search for secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert
2019-12-12T00:28:33.400997Z	debug	secretFetcherLog	SecretFetcher search for secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert
2019-12-12T00:28:33.401004Z	debug	secretFetcherLog	load secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert from secret fetcher: false
2019-12-12T00:28:33.407724Z	warn	secretFetcherLog	Cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert, searching for fallback secret gateway-fallback
2019-12-12T00:28:33.407750Z	error	secretFetcherLog	cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert and cannot find fallback secret gateway-fallback
2019-12-12T00:28:33.407757Z	warn	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-9, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: SecretFetcher cannot find secret awood1018db10718dc0e1994886056a23fd7b9b1-cacert from cache
2019-12-12T00:28:33.409307Z	warn	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-9, RESOURCE NAME: awood1018db10718dc0e1994886056a23fd7b9b1-cacert, EVENT: waiting for ingress gateway secret for proxy "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local"

At a quick glance it looks like the cleanup job which is taking out stalled clients seems to be the culprit.

I should instead say that the recycle logic here https://github.com/istio/istio/blob/669bf354e27d928bbedc49feb41417fd8c4311b0/security/pkg/nodeagent/sds/sdsservice.go#L269 and/or here https://github.com/istio/istio/blob/669bf354e27d928bbedc49feb41417fd8c4311b0/security/pkg/nodeagent/sds/sdsservice.go#L356 seems to be creating some sort of logical race...

Also, for some context the reason why you see that the secret does not exist for some time in the logs (even in the scenarios that work correctly) is due to the behavior of our own software. What we do is we first program the gateway config into the cluster and then later (between 30 seconds and 1 min usually) program the secret into the cluster.

Saw an interesting scenario today:
2019-12-13T18:21:37.813823Z	error	sdsServiceLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-13, RESOURCE NAME: awood01ded641ce22a5805cb501f6413c9fd0c4, EVENT: NotifyProxy failed. No connection with id "router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-13" can be found
2019-12-13T18:21:37.813875Z	error	cacheLog	CONNECTION ID: router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-13, RESOURCE NAME: awood01ded641ce22a5805cb501f6413c9fd0c4, EVENT: failed to notify secret change for proxy: no connection with id "router~100.102.204.50~istio-ingressgateway-66

The connection its trying to use called router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-13 was recycled yesterday:
2019-12-12T01:41:25.294628Z	debug	sdsServiceLog	start staled connection cleanup job
2019-12-12T01:41:25.294675Z	debug	sdsServiceLog	remove staled clients {ConnectionID:router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-13 ResourceName:awood01ded641ce22a5805cb501f6413c9fd0c4}
2019-12-12T01:41:25.294691Z	debug	sdsServiceLog	remove staled clients {ConnectionID:router~100.102.204.50~istio-ingressgateway-66b78f978b-dmnp8.istio-system~istio-system.svc.cluster.local-12 ResourceName:awood01ded641ce22a5805cb501f6413c9fd0c4-cacert}

Hi,
It happens to me too with istio 1.4.2
in the ingress gateway :
ingress sds :
2019-12-26T05:25:36.481345Z     info    sdsServiceLog   CONNECTION ID: router~10.233.65.7~istio-ingressgateway-6677d6c5f5-6z7wq.istio-system~istio-system.svc.cluster.local-1, RESOURCE NAME: adcc-cert, EVENT: connection is terminated: rpc error: code = Canceled desc = context canceled
2019-12-26T05:25:36.481415Z     error   sdsServiceLog   Remote side closed connection
istio-proxy :
2019-12-26T07:45:46.407147Z     error   Epoch 0 exited with error: signal: segmentation fault (core dumped)
2019-12-26T07:45:46.407181Z     info    1 active epochs running
it seems like the istio-proxy inside the ingress crashed and didn't recover.( i looked for  info    Envoy proxy is ready)
can someone advise?
thanks,
perry

Note that SDS in 1.5 is not quite ready yet: https://github.com/istio/istio/issues/22443

@howardjohn should we reopen this ticket?

I face the same problem in version 1.5.

https://github.com/istio/istio/pull/24817 is the fix for the issue.

@JimmyCYJ what release will this be targeted for? Will it be backported anywhere?

what release will this be targeted for? Will it be backported anywhere?

It will be shipped with release-1.7. Which will be announced on August 11th (https://github.com/istio/istio/wiki/Istio-Release-1.7)
@williamaronli could you cherrypick this fix into 1.5 and 1.6 as well?

will cherry-pick to 1.5 and 1.6 this week

authors: JimmyCYJ aaronjwood dkalani howardjohn incfly jamesmoriarty jullianoacqio perry9186 psychoveter williamaronli


text:
Node agent / FlexVolume / Workload API work

Looks like we don't need FlexVolume, i propose we close this issue, @saumoh , any objection?

I am closing it for now and feel free to reopen it.


text:
Unable to connect Elasticache Redis from inside of a istio sidecar enabled pod

Bug description
I have deployed a EKS cluster and I am using Istio . I am trying to connect Elasticache Redis cluster from inside of a pod and that pod has Istio sidecar enabled. I have encryption disabled in Redis cluster. It is working as expected when I am connecting from a non-istio pod. Command to connect redis, I am using ,
redis-cli -h redis.xxx.amazonaws.com
redis.xxx.amazonaws.com:6379> ping
Error: Connection reset by peer

I have created a DestionationRule , ServiceEntry and VirtualService in that namespace from where I am trying to connect.
ServiceEntry :-
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: elasticache-svc-redis
  namespace: <NAME>
spec:
  hosts:
    - "redis.xxx.amazonaws.com"
  location: MESH_EXTERNAL
  ports:
    - number: 6379
      name: redis
      protocol: TCP
  resolution: DNS

DestinationRule :-
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
    name: redis-elasticcache
    namespace: <NAME>
spec:
    host: "*.amazonaws.com"
    trafficPolicy:
        tls:
            mode: DISABLE

VirtualService :-
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: redis-test-cluster
  namespace: <NAME>
spec:
  hosts:
  - redis.xxx.amazonaws.com
  http:
  - timeout: 30s
    route:
    - destination:
        host: redis.xxx.amazonaws.com

Observations
I can connect through netcat but not telnet ,
telnet redis.xxx.amazonaws.com 6379
Trying 10.30.10.177...
Connected to redis.xxx.amazonaws.com.
Escape character is '^]'.
Connection closed by foreign host.

Netcat :-
nc -v redis.xxx.amazonaws.com 6379
redis.xxx.amazonaws.com (10.30.10.177:6379) open

Expected behavior
redis-cli -h redis.xxx.amazonaws.com
redis.xxx.amazonaws.com:6379> ping
PONG

Steps to reproduce the bug
Go inside of a pod ,
kubectl exec -it <pod_name> -n <name_spcae> -c <default_container> /bin/bash
Run the command,
redis-cli -h XXXXX.amazonaws.com
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
Version :- istioctl version --remote
client version: 1.5.1
internal-popcraftio-ingressgateway version:
citadel version: 1.4.3
galley version: 1.4.3
ingressgateway version: 1.4.3
pilot version: 1.4.3
policy version: 1.4.3
sidecar-injector version: 1.4.3
telemetry version: 1.4.3
office-popcraftio-ingressgateway version:
data plane version: 1.4.3 (88 proxies)

kubectl version :-
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4", GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean", BuildDate:"2020-03-12T21:03:42Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-502bfb", GitCommit:"502bfb383169b124d87848f89e17a04b9fc1f6f0", GitTreeState:"clean", BuildDate:"2020-02-07T01:31:02Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

helm version :-
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}
How was Istio installed?
Cluster was created using helm chart .
Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS Version 1.14

See https://github.com/istio/istio/issues/14743 https://github.com/istio/istio/issues/12079 https://github.com/istio/istio/issues/14138 https://github.com/istio/istio/issues/15429 https://github.com/istio/istio/issues/16078 <-- see this issue for possible solution

We had the exact same problem. EKS 1.14, Istio 1.6.5
In our case the problem was in a sort of a race condition between envoy and app containers. App couldn't make connection to  Elasticache redis until envoy sidecar is fully up and synced.
Our app is on .Net and we are using StackExchange.Redis client lib.
Adding resolveDns=true,connectRetry=100,connectTimeout=5000,abortConnect=false solved the issue for us. (https://stackexchange.github.io/StackExchange.Redis/Configuration.html#configuration-options)
Look at these for a detailed explanation:
https://github.com/kubernetes/kubernetes/issues/65502
https://banzaicloud.com/blog/k8s-sidecars/

Adding annotation in Deployments as below solved the isssue.
      annotations:
        traffic.sidecar.istio.io/excludeOutboundPorts: "6379"

Same here in 2023 with istio 1.18.2 able to resolve by:
annotations:
        traffic.sidecar.istio.io/excludeOutboundPorts: "6379"

we have MTLS in strict mode.
Having:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: ggg
  namespace: ggg
spec:
  hosts:
    - ggg.cache.amazonaws.com
  location: MESH_EXTERNAL
  ports:
    - name: tcp-redis
      number: 6379
      protocol: TCP
  resolution: DNS

not helps
Also we accessing amazon elasticache

authors: Hronom abhijitio adiseshan esnible meshok0


text:
External requests fail, and "Problem with the SSL CA cert" when sending an HTTPS request from the istio-proxy

I installed istio and injected the sidecar into one of my services. My issue is that with the sidecar container being in place, I cannot send any requests to external services from my actual container, and I'm trying to investigate what the problem can be.
If I enter the istio-proxy container with bash, and execute curl http://example.com, then it works, but if I do curl https://example.com, then I receive
curl: (77) Problem with the SSL CA cert (path? access rights?)

Is this expected? If not, what can cause this?
The actual issue that I'm investigating is that if I'm trying to do the same curl command (curl http://example.com) in my actual application container, I get
root@myservice-6f548df4d5-v4kmj:/# curl http://example.com
curl: (7) Failed to connect to example.com port 80: Connection refused

I have already followed the guide about setting up the rule for egress traffic, and I applied the following rul:
apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: examplecom-egress-allow-rule
spec:
  destination:
    service: example.com
  ports:
    - port: 80
      protocol: http
    - port: 443
      protocol: https

Should external requests work properly?
It'd be great if you could give me some pointers about where to start troubleshooting this issue.

As a workaround, I think you'd want to flip it around and use
  CIDR=$(gcloud container clusters describe ${K8S_CLUSTER} ${GCP_OPTS:-} --format "value(servicesIpv4Cidr)")

and set that as IP_RANGES_INCLUDE env to only intercept cluster ips instead of all traffic

Closing this in favor of https://github.com/istio/istio/issues/4378

Hi Guys, I am facing similar issues with

error reading ca cert file /etc/ssl/certs/ca-certificates.crt (Error while reading file.)

istio-proxy@consul1-v1-688df668c9-8cs25:/$ curl --verbose https://config.somedomain.com

Rebuilt URL to: https://config.somedomain.com/
Trying 10.21.200.120...
Connected to config.somedomain.com (10.21.200.120) port 443 (#0)
error reading ca cert file /etc/ssl/certs/ca-certificates.crt (Error while reading file.)
Closing connection 0
curl: (77) Problem with the SSL CA cert (path? access rights?)
istio-proxy@consul1-v1-688df668c9-8cs25:/$

config.somedomain.com is one of our internal URL.
Really appreciate If you have any inputs on this.
Kube version: v1.10.5
Istio version: 1.0

authors: ayj ldemailly markvincze vpiduri


text:
Istio 1.5.6, istiod not creating istio-ca-root-cert configmap for namespaces where istio injection is enabled

HI Guys,
with reference to the below comment, the config map creation issue where istio injection is enabled on the namespace should have fixed in istio 1.5.6 , but that's not the case as this has hit us badly again in production today.
Based on the confirmation. we have migrated all our environments from Istio 1.5.4 to Istio 1.5.6 just to fix this burning issue, but the issue still exists
Whenever a tenant onboard we create a namespace for the tenant and all the deployments of that tenant happens in his own namespace and whenever istiod stops creating istio-ca-certs for new namespaces, the deployments for all these new tenants will fail
the stop gap solution is to restart the istiod pod and it will resolve the issue, but it's a bad hack and it's not a good customer experience
Cloud Provider: GCP
Managed Kubernetes: Yes, we are using GKE
Kubernetes Cluster Version:  1.16.10-gke.8
No of Nodes: 120
Istio Version: 1.5.6
No of Namespaces: 350 currently & growing
Can you guys please suggest, how can this problem be fixed asap as it's happening on our production environments

"@rajivml @Mdirfan6 Actually this is fixed in 1.5.6, please see the release announcement
https://istio.io/latest/news/releases/1.5.x/announcing-1.5.6/.
Originally posted by @knight42 in https://github.com/istio/istio/issues/22463#issuecomment-648777698
"

I have verified the istio-leader configmap, it's pointing to the right istiod pod which is of istio 1.5.6 version itself

Please let us know if any additional logs are needed, after we restarted istiod problem went away and in couple of hours again the problem started, will changing the istiod pod replicas to 1 will temporarily solve the issue if it's an leader election issue among istiod pods ?

any update on this ? this issue is happening consistently on our prod environment at-least once a day , any suggestion on how to fix this for good would be helpful

cc @howardjohn @tariq1890 @knight42

Can you post the istiod logs? does this issue happen always/permanently or only after some period of time? I mean if you restart istiod, then create a new namespace, will it update the configmap?

it's happening many times a day and when we are restarting istiod, it's creating missing configmaps in the namespaces where istio injection is enabled and we have temporarily scheduled a cron like this to restart istiod when ever configmap creation is not happening .
I will share the logs shortly , I have just now asked our IT admin to create an sink (GCP bucket) to upload the istiod logs as gcp console is giving an option to download only 300 lines of logs
#!/bin/bash

echo "Validating istio setup"
temp_namespace="validate-istio"
#create temp namespace
kubectl create namespace $temp_namespace
echo "$temp_namespace namespace created. Labelling namespace ... "
sleep 10
kubectl label namespace $temp_namespace istio-injection=enabled
sleep 30
#validating istiod
counter=1
while [ $counter -le 3 ];
do
  # check configmap
  response=$(kubectl -n ${temp_namespace} get configmap | grep istio-ca-root-cert | awk '{print $1}')
  echo "$response"
  if [ \( ! -z "${response}" \) -a \( "${response}" = "istio-ca-root-cert" \) ]; then
    echo "Istio Setup successfully completed and working fine."
    kubectl delete namespace $temp_namespace
    echo "$temp_namespace namespace deleted"
    exit 0
  else
    kubectl -n istio-system rollout restart deployment istiod
    echo "${counter} iteration restarting istio ..."
    sleep 30
  fi
  counter=$((counter+1))
done

echo "Istio setup validation failed"
kubectl delete namespace $temp_namespace
echo "$temp_namespace namespace deleted"

@howardjohn plz find the logs downloadable at below link, attached logs worth of couple of hours
https://uipath-my.sharepoint.com/:u:/p/rajiv_chodisetti/EQUC5shCdeZPvg2Kt9PpKw8BPJd5N9p1zv49VkeZaFt0sA?e=3smp8J
https://uipath-my.sharepoint.com/:u:/p/rajiv_chodisetti/ERa8gKX6yFBDtdCw6bc1XuIB_IS8tlYg893h0zdqO6sYpQ?e=nwoD9l

Hi, is anyone looking into it

@myidpt @howardjohn since you worked on https://github.com/istio/istio/issues/22463, could you help to take a look at this issue? It is not clear to me what is the root cause, and whether the root cause to this issue is the same as the one fixed at #22463.

This issue is still consistently happening on all our environments i.e alpha, staging and prod where istio 1.5.6 is being used and we put a cronjob to restart istiod pod whenever we run into such issue which is not an elegant solution. So, appreciate any support here, if more logs are required please let me know

@howardjohn Hi John, do we observe similar issues on Istio 1.6 or later versions?

Not that I know of, probably fixed by #22598. We can re-open if we see it in 1.7+

@howardjohn Nope, this fix was live in 1.5.6 and use 1.5.6 in production and this issue happens on our production on a day to day basis and we have a cron setup to restart istiod when this happens https://github.com/istio/istio/pull/22598

@howardjohn Nope, this fix was live in 1.5.6 and use 1.5.6 in production and this issue happens on our production on a day to day basis and we have a cron setup to restart istiod when this happens https://github.com/istio/istio/pull/22598

authors: JimmyCYJ howardjohn myidpt rajivml


text:
Istio doesn't appear to favour pods on the same node

Bug description
I've been investigating some tail latency problems recently; and stumbled across this behaviour which I believe to be wrong.
Given the following pods:
❯ k get pods -o wide
NAME                                 READY   STATUS    RESTARTS   AGE    IP              NODE                                                  NOMINATED NODE   READINESS GATES
solr-0                               1/1     Running   0          2d5h   10.226.16.6     gke-delivery-platform-solrw-20200210-57ae432e-w1n3    <none>           1/1
solr-1                               1/1     Running   0          2d5h   10.226.3.6      gke-delivery-platform-solrw-20200210-2e0a15aa-qj1i    <none>           1/1
solr-2                               1/1     Running   0          2d5h   10.226.20.6     gke-delivery-platform-solrw-20200210-10df47ec-2762    <none>           1/1
solr-3                               1/1     Running   0          2d5h   10.226.13.6     gke-delivery-platform-solrw-20200210-57ae432e-t99f    <none>           1/1
solr-4                               1/1     Running   0          2d5h   10.226.19.6     gke-delivery-platform-solrw-20200210-2e0a15aa-uoar    <none>           1/1
solr-5                               1/1     Running   0          2d5h   10.226.10.8     gke-delivery-platform-solrw-20200210-10df47ec-zytc    <none>           1/1
solr-overwatch-549bc4bcc-mxjfx       2/2     Running   0          77m    10.226.16.9     gke-delivery-platform-solrw-20200210-57ae432e-w1n3    <none>           <none>

And the following service:
  name: karl
  namespace: search-solr
  resourceVersion: "357571791"
  selfLink: /api/v1/namespaces/search-solr/services/karl
  uid: 883ed57b-42c1-4d12-88c8-b7100452b9ee
spec:
  clusterIP: 10.224.11.190
  ports:
  - name: http-server
    port: 80
    protocol: TCP
    targetPort: http-server
  selector:
    app: solr
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Which resolves to these endpoints:
❯ k get ep karl -o wide
NAME   ENDPOINTS                                                  AGE
karl   10.226.10.8:80,10.226.13.6:80,10.226.16.6:80 + 3 more...   9m35s

When making requests from overwatch, which resides on the same node as solr-0, I would expect traffic to remain on that node, however it does not.  Here are the TCP dumps showing me repeatedly hitting solr-0 from overwatch:
[atcloud@solr-overwatch-549bc4bcc-mxjfx app]$ curl -s -L "http://karl.search-solr/solr/at-uk/select?q=*:*&rows=0" | jq .

oot@gke-delivery-platform-solrw-20200210-57ae432e-w1n3:~# tcpdump -AAAA port 80 and src 10.226.16.9 | grep GET
14:49:36.960170 IP 10.226.16.9.56506 > 10.226.20.6.http: Flags [P.], seq 1921231721:1921233968, ack 3409949414, win 331, length 2247: HTTP: GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
......Pr..i.?..P..KB...GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
14:49:37.528387 IP 10.226.16.9.36418 > 10.226.13.6.http: Flags [P.], seq 3930777270:3930779517, ack 4207939322, win 331, length 2247: HTTP: GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
....B.P.J......P..K;...GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
14:49:37.853118 IP 10.226.16.9.53456 > 10.226.10.8.http: Flags [P.], seq 248354423:248356670, ack 3089487598, win 331, length 2247: HTTP: GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
....P...w.%..P..K8...GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
14:49:38.528211 IP 10.226.16.9.56482 > 10.226.20.6.http: Flags [P.], seq 359421875:359424122, ack 1105199185, win 331, length 2247: HTTP: GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
......P.lW.A..QP..KB...GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
14:49:38.798516 IP 10.226.16.9.36448 > 10.226.13.6.http: Flags [P.], seq 410833569:410835816, ack 2891369657, win 331, length 2247: HTTP: GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
....`.P.|...V..P..K;...GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1
14:49:39.061700 IP 10.226.16.9.50168 > 10.226.19.6.http: Flags [P.], seq 955976578:955978825, ack 181264837, win 331, length 2247: HTTP: GET /solr/at-uk/select?q=*:*&rows=0 HTTP/1.1

[ ] Docs
[ ] Installation
[x] Networking
[x] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
[ ] Upgrade
Expected behavior
Pods scheduled with other pods using nodeAffinity should route locally
Steps to reproduce the bug
🤷
Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)
1.9.1
How was Istio installed?
Environment where the bug was observed (cloud vendor, OS, etc)
Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue.
Refer cluster state archive for more details.

Few things which are atypical about this particular app:

The target workload is a Statefulset
The target workload does not have istio

Pods scheduled with other pods using nodeAffinity should route locally

Why is this expected? My understanding is this isn't how Kubernetes works without Istio, and its not how Istio is documented (based on locality info, not cluster/node) - is that not accurate?
Its a valid feature request to route to node-local but I wouldn't expect it to happen today or by default?

@howardjohn two years using kubernetes i've worked on that assumption, but you're absolutely right; after digging - its not how k8s routes either.  Apologies.
In that case, this is a feature request!

authors: Stono howardjohn


text:
Gateway SSL Istio 1.0 prelim

Gateway SSL does not work when using 1.0 prelim.
Service is accessible when using 80 but not when using 443.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: app-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
      - "myhost"
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
        privateKey: /etc/istio/ingressgateway-certs/tls.key
      hosts:
      - "myhost"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: app
spec:
  hosts:
  - "myhost"
  gateways:
  - app-gateway # <---- bind to gateway
  http:
  - match:
    - uri:
        prefix: /app
    route:
    - destination:
        host: app

Ingress logs:
[2018-07-20 19:36:21.445][18][warning][config] external/envoy/source/server/listener_manager_impl.cc:259] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)
[2018-07-20 19:36:21.445][18][info][upstream] external/envoy/source/server/lds_api.cc:73] lds: add/update listener '0.0.0.0_443'
Certs have been created following example:
kubectl get secrets -n istio-system istio-ingressgateway-certs
NAME                         TYPE                DATA      AGE
istio-ingressgateway-certs   kubernetes.io/tls   2         23m

Curl response when using ssl. curl -v https://myhost/app/
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb13c000000)
> GET /app/ HTTP/1.1
> Host: myhost
> User-Agent: curl/7.50.3
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 503
< date: Fri, 20 Jul 2018 19:52:46 GMT
< server: envoy

@robertfreund
same experience. on top, on listener update gateway segfaults
[2018-07-21 03:08:33.925][121][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:125] Caught Segmentation fault, suspect faulting address 0x0
[2018-07-21 03:08:33.925][121][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:94] Backtrace thr<0> obj</usr/local/bin/envoy> (If unsymbolized, use tools/stack_decode.py):
[2018-07-21 03:08:33.929][121][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] thr<0> #0 0xcc0528 Envoy::SignalAction::sigHandler()
[2018-07-21 03:08:33.929][121][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] thr<0> obj</lib/x86_64-linux-gnu/libpthread.so.0>
[2018-07-21 03:08:33.929][121][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:117] thr<0> #1 0x7f35a65ae38f (unknown)
[2018-07-21 03:08:33.929][121][critical][backtrace] bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:121] end backtrace thread 0
2018-07-21T03:08:33.995588Z	warn	Epoch 0 terminated with an error: signal: segmentation fault (core dumped)

@mk01 I think the segfault is unrelated potentially. Would be good to have a core dump and backtrace to understand the issue.
@robertfreund short names are not allowed in the gateway hosts. Try using a FQDN name. I have fixed the validation issue in #7301 so that you would get a validation error early on (when using istioctl).

Yeah, i am using FQDN. That was changed for posting. This actually all works using the old ingress. Im trying to upgrade from 0.7.

Started over from scratch and got this to work for both the example and my use case.
The confusion was in verifying service via the Host header.
It not enough with --resolve and --cacert you still need the Host header passed as a host header that includes the port will not match the host in the gateway.

Host: mydomain.com:31090 will not match mydomain.com.

Hi @robertfreund , facing the same issue, but couldn't understand the solution, please help me in understanding it.


text:
release: Update latest stable Istio CNI SHA

Make istio releases & nightlies pick up latest stable istio-cni.
Fixes: #13171

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: tiswanso
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
If they are not already assigned, you can assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tiswanso: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
011717cc7d26250c276862244f3213b0f47fb9f7
link
/test istio-integ-k8s-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@tiswanso: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-integ-k8s-tests.sh
011717cc7d26250c276862244f3213b0f47fb9f7
link
/test istio-integ-k8s-tests

prow/istio-pilot-multicluster-e2e.sh
011717cc7d26250c276862244f3213b0f47fb9f7
link
/test istio-pilot-multicluster-e2e

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/lgtm

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: howardjohn, tiswanso
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: costinm
If they are not already assigned, you can assign the PR to them by writing /assign @costinm in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Any movement on this or anything I/we can do to help?

/lgtm

/aprove

/approve
if i can type :)

for release-1.1 since it is sensitive to breakage, would be great to demonstrate some testing around the new CNI SHA before merge.

per offline discussion with Tim, this is already tested with release-1.1 in PR gates for CNI project.

source: gharchive/pull-request

authors: edude03 howardjohn istio-testing john-a-joyce linsun tiswanso


text:
remote clusters mesh networks reload

Please provide a description for what this PR is for.
And to help us figure out who should review this PR, please
put an X in all the areas that this PR affects.
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: hzxuzhonghu
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [hzxuzhonghu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

We found this bug when we set a invalid format meshnetworks during test, and updated it. But the remote cluster endpoints still have empty network label.

@hzxuzhonghu: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-lint.sh
96b8c70e10fb7f0f4cdcb5bb213585bd1a5b8a3d
link
/test istio-lint-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-lint.sh
96b8c70e10fb7f0f4cdcb5bb213585bd1a5b8a3d
link
/test istio-lint-master

prow/integ-mixer-k8s-presubmit-tests.sh
96b8c70e10fb7f0f4cdcb5bb213585bd1a5b8a3d
link
/test integ-mixer-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@hzxuzhonghu: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/integ-mixer-k8s-presubmit-tests.sh
3c1143d5961f4e6ec09270249abb24a8968a638a
link
/test integ-mixer-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: hzxuzhonghu, rshriram
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [hzxuzhonghu,rshriram]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

authors: hzxuzhonghu istio-testing


text:
[release-1.3] Fix rewriting of istio-cni Helm charts in release scripts

This is an automated cherry-pick of #16492

@istio-testing: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-new-install-k8s-presubmit-tests-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test integ-new-install-k8s-presubmit-tests-release-1.3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-new-install-k8s-presubmit-tests-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test integ-new-install-k8s-presubmit-tests-release-1.3

istio-unit-tests-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test istio-unit-tests-release-1.3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-new-install-k8s-presubmit-tests-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test integ-new-install-k8s-presubmit-tests-release-1.3

istio-unit-tests-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test istio-unit-tests-release-1.3

istio-racetest-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test istio-racetest-release-1.3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@istio-testing: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

integ-new-install-k8s-presubmit-tests-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test integ-new-install-k8s-presubmit-tests-release-1.3

istio-unit-tests-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test istio-unit-tests-release-1.3

istio-racetest-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test istio-racetest-release-1.3

istio-codecov-release-1.3
6815fdd20d31c2f7742cac675df5d316486f67d5
link
/test istio-codecov-release-1.3

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test istio-unit-tests-release-1.3

/test istio-racetest-release-1.3

/test istio-codecov-release-1.3

authors: istio-testing rlenglet


text:
Revert "Istiod agent"

Reverts istio/istio#18314

See https://github.com/istio/istio/issues/18779