text:
Adding Rate limiting ec2:DescribeInstances API along with Batching for high TPS

Prerequisite:
The AWS ec2:DescribeInstances API  is a low TPS batching capable API.
Summary :
As of Today the a K8s cluster can support up-to 5000 nodes. every nodes has to run kubelet process to join to cluster, while joining the cluster kubelet makes multiple calls to API Server and if it fails it does a retry with backoff where it can make 8-9 calls per object type.
Given the Authenticator is a webhook and API Server makes calls to Authenticator for every API call being made to verify the users or nodes role.
While verifying for nodes it needs to call to ec:DescribeInstances to get the PrivateDNSName for the node as only session with instanceId is available as well as we don't know whether this node belongs to the same aws account.
Problem:
For example lets assume we have 500 nodes , and a cluster have 20 object types.
Thinking of two situations:

When Master restarts or new Master comes up and old master goes down in a HA Master setup
When cluster has a sudden hike in number of nodes joining

For the master restart , every connection to the old master from every kubelet running on nodes will close like watches and all hence kubelet will try to re-connect and get the updated list and do watch connection while doing so it has to pass through Authenticator and authenticator does verification of instance with PrivateDNSNames belonging to the same aws account.
As of today not having any rate limiting  for the ec2:DescribeInstances make service unavailable as after 20-30 calls the aws ec2 API starts throttling because every node calls goes to API in the same interval and at multiple fold.
The number of calls per second for the failed to get authenticated would be
Number of nodes  * 20 * 8 = 500 * 20 * 8 = ~ 8000  TPS
In a minute it can go upto ~ 200 K - ~400K
As this throttling would not get resolved at the same time it takes almost 2-5 minutes to get resolve
So total number of throttling goes to a very high number which impacts other services running in the aws account or CNI or Kube-Controller-manager.
While the Authenticator tries to cache the PrivateDNSNames but that does not help in this case as the kubelet calls very fast and the master has restarted  so it does not have every node in cache.
Solution:

Adding rate Limiting to make sure it does not hit the high TPS
Adding rate limit will make the system very slow in the above mentioned situations as for the same node it would still make calls to ec2:DescribeInstances. So adding a look up map to check if we already have the in flight request for the ec2 instanceId and then wait for upto 5 second and keep looking into the cache to check if the Inflight request has completed and added the PrivateDNSName to cache.
Given having a large number of nodes it can still take minutes to process every single node description so adding a batching routine which when number if Inflight request becomes more than 5 it can start batching and every 200 milliseconds batches the instances and just makes one call to ec2 to get the details of instances and put the PrivateDNSNames into the cache.

Testing:

Added unit tests which emulates the behavior with 100 nodes.
Tested in a real EKS cluster with 500, 1000 and 3000 nodes to check the count of API as well as overall performance  of the system when we recycle the  masters.

/assign mhausler

make test and unit Test Results:
bash-5.0$ make test
go test -v -coverprofile=coverage.out -race sigs.k8s.io/aws-iam-authenticator/...
?   	sigs.k8s.io/aws-iam-authenticator/cmd/aws-iam-authenticator	[no test files]
=== RUN   TestUserARN
--- PASS: TestUserARN (0.00s)
PASS
coverage: 72.7% of statements
ok  	sigs.k8s.io/aws-iam-authenticator/pkg/arn	1.070s	coverage: 72.7% of statements
?   	sigs.k8s.io/aws-iam-authenticator/pkg/config	[no test files]
=== RUN   TestGetPrivateDNSName
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-1 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-1"
time="2020-02-20T22:04:25-08:00" level=info msg="Calling ec2:DescribeInstances for the InstanceId = ec2-1 "
--- PASS: TestGetPrivateDNSName (0.10s)
=== RUN   TestGetPrivateDNSNameWithBatching
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-1 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-1"
time="2020-02-20T22:04:25-08:00" level=info msg="Calling ec2:DescribeInstances for the InstanceId = ec2-1 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-3 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-3"
time="2020-02-20T22:04:25-08:00" level=info msg="Calling ec2:DescribeInstances for the InstanceId = ec2-3 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-5 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-5"
time="2020-02-20T22:04:25-08:00" level=info msg="Calling ec2:DescribeInstances for the InstanceId = ec2-5 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-6 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-6"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-10 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-10"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-8 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Calling ec2:DescribeInstances for the InstanceId = ec2-6 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-14 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-14"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-9 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-17 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-18 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-15 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-20 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-20"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-20 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-9"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-7 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-23 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-23"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-7"
time="2020-02-20T22:04:25-08:00" level=info msg="Calling ec2:DescribeInstances for the InstanceId = ec2-10 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-18"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-18 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-15"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-15 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-30 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-30"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-31 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-22 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-22"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-12 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-13 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-8"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-14 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-16 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-17"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-19 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-21 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-4 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-9 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-24 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-11 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-2 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-25 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-23 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-26 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-7 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-27 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-28 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-18 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-15 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-20 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-29 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-30 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-32 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-31"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-33 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-34 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-22 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-35 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-12"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-36 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-13"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-37 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-8 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-38 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-14 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-39 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-16"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-40 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-17 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-41 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-19"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-42 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-21"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-43 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-44 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-4"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-9 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-46 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-24"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-47 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-11"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-48 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-2"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-49 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-25"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-23 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-26"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-7 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-27"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-28"
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-18 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-45 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-15 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-29"
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-20 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-30 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-32"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-31 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-33"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-34"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-22 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-35"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-12 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-36"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-13 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-37"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-8 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-38"
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-14 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-39"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-16 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-40"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-17 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-41"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-19 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-42"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-21 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-43"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-44"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-4 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-9 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-46"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-24 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-47"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-11 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-48"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-2 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-49"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-25 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-23 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-26 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-7 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-27 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-28 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-45"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-29 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-30 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-32 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-31 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-33 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-34 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-22 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-26 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-35 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-12 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-36 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-13 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-37 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-8 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-38 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-38 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-39 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-16 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-40 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-17 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-41 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-19 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-42 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-21 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-43 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-44 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-4 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-46 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-24 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-47 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-11 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-48 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-2 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-49 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-25 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-13 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-27 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-28 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-45 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-29 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-29 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-32 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-31 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-33 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-34 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-25 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-26 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-35 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-12 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-36 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-37 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-38 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-39 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-16 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-40 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-40 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-41 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-19 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-42 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-21 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-43 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-44 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-4 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-46 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-24 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-47 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-11 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-48 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-2 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-49 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-36 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-37 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-27 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-28 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-45 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-32 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-33 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-34 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-48 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-49 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-35 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-39 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-45 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-41 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-20 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-42 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-43 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-44 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-46 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-47 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-18 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-15 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-14 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-9 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-23 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-7 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-30 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-22 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-8 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-17 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-31 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-12 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-13 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-16 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-19 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-21 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-4 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-24 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-11 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-2 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-25 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-26 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-27 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-28 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-29 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-32 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-33 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-34 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-35 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-36 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-37 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-38 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-39 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-40 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-41 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-42 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-43 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-44 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-46 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-47 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-48 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-49 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-45 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-50 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-51 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-50"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-52 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-52"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-51"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-51 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-51 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-51 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-52 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-52 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-52 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-56 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-50 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-57 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-57"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-58 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-58"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-58 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-59 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-65 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-56"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-53 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-55 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-55"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-61 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-57 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-54 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-58 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-65"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-66 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-50 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-60 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-62 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-63 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-64 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-59"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-56 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-67 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-53"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-68 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-69 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-55 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-70 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-61"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-71 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-57 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-72 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-54"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-73 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-58 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-74 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-65 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-75 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-66"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-76 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-77 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-50 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-78 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-60"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-79 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-62"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-80 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-63"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-81 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-59 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-82 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-64"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-56 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-83 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-67"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-84 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-53 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-85 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-68"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-86 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-69"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-87 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-55 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-88 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-70"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-89 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-61 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-90 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-71"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-91 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-57 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-92 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-93 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-72"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-54 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-94 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-73"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-95 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-96 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-74"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-97 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-65 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-98 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-75"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-66 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-99 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-100 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-76"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-77"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-78"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-60 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-79"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-62 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-80"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-63 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-81"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-59 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-82"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-64 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-56 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-83"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-67 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-84"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-53 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-85"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-68 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-86"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-69 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-87"
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-55 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-88"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-70 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-89"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-61 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-90"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-71 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-91"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-92"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-93"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-72 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-54 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-94"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-73 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-95"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-96"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-74 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-97"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-98"
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-65 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-75 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-66 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-99"
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the requestQueue cache for the InstanceId = ec2-100"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-76 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-77 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-78 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-60 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-79 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-62 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-80 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-63 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-81 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-59 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-82 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-64 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-83 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-67 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg=" Making call to getPrivateDnsAndPublishToCache for batch of instances"
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-84 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-53 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-85 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-68 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-86 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-69 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-87 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-70 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-88 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-89 "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-61 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-90 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-71 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-91 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-92 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-93 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-72 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-54 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-94 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-73 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-95 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-96 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-74 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-97 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-98 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-75 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-66 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-99 "
time="2020-02-20T22:04:25-08:00" level=info msg="Writing to buffered channel for instance Id ec2-100 "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-76 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-77 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-78 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-60 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-79 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-62 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-80 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-63 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-81 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-82 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-64 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-83 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-67 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-84 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-51 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Making Batch Query to DescribeInstances for 44 instances "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-85 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-68 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-86 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-69 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-87 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-70 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-88 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-89 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-90 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-71 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-91 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-92 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-93 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-72 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-94 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-73 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-95 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-96 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-74 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-97 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-98 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-75 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-99 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Missed the cache for the InstanceId = ec2-100 Verifying if its already in requestQueue "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-76 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-77 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-78 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-80 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-79 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-81 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-82 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-83 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-98 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-84 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-99 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-52 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-85 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-86 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-100 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-87 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-88 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-89 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-90 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-91 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-92 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-93 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-94 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-95 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-96 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Found the InstanceId:= ec2-97 request In Queue waiting in 5 seconds loop "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-50 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-58 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-57 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-56 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-55 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-65 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-59 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-53 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-61 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-54 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-66 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-60 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-62 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-63 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-64 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-67 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-68 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-69 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-70 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-71 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-72 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-73 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-74 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-75 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-76 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-77 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-78 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-79 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-80 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-81 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-82 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-83 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-84 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-85 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-86 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-87 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-88 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-89 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-90 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-91 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-92 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-93 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-94 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-95 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-96 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-97 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-98 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-99 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Received the Instance Id := ec2-100 from buffered Channel for batch processing "
time="2020-02-20T22:04:25-08:00" level=info msg="Successfully got the batch result , output.NextToken is nil "
time="2020-02-20T22:04:25-08:00" level=info msg="Removing instances from request Queue after getting response from Ec2"
time="2020-02-20T22:04:25-08:00" level=info msg=" Making call to getPrivateDnsAndPublishToCache for batch of instances"
time="2020-02-20T22:04:25-08:00" level=info msg="Making Batch Query to DescribeInstances for 51 instances "
time="2020-02-20T22:04:26-08:00" level=info msg="Successfully got the batch result , output.NextToken is nil "
time="2020-02-20T22:04:26-08:00" level=info msg="Removing instances from request Queue after getting response from Ec2"
--- PASS: TestGetPrivateDNSNameWithBatching (0.59s)
PASS
coverage: 76.2% of statements
ok  	sigs.k8s.io/aws-iam-authenticator/pkg/ec2provider	1.883s	coverage: 76.2% of statements
=== RUN   TestNewRateLimitedClient
--- PASS: TestNewRateLimitedClient (4.93s)
PASS
coverage: 75.0% of statements
ok  	sigs.k8s.io/aws-iam-authenticator/pkg/httputil	6.029s	coverage: 75.0% of statements
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper	[no test files]
=== RUN   TestUserMapping
--- PASS: TestUserMapping (0.00s)
=== RUN   TestRoleMapping
--- PASS: TestRoleMapping (0.00s)
=== RUN   TestAWSAccount
--- PASS: TestAWSAccount (0.00s)
=== RUN   TestLoadConfigMap
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
--- PASS: TestLoadConfigMap (0.02s)
=== RUN   TestConfigMap
=== RUN   TestConfigMap/aws-auth.yaml
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
=== RUN   TestConfigMap/aws-auth-crazy-case-keys.yaml
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
=== RUN   TestConfigMap/aws-auth-open-source-case-keys.yaml
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
=== RUN   TestConfigMap/aws-auth-lower-case-top-keys.yaml
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
=== RUN   TestConfigMap/aws-auth-crazy-case-top-keys.yaml
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
=== RUN   TestConfigMap/aws-auth-space-out-of-place.yaml
time="2020-02-20T22:04:29-08:00" level=info msg="Received aws-auth watch event"
=== RUN   TestConfigMap/aws-auth-missing-bar.yaml
--- PASS: TestConfigMap (0.05s)
    --- PASS: TestConfigMap/aws-auth.yaml (0.01s)
    --- PASS: TestConfigMap/aws-auth-crazy-case-keys.yaml (0.01s)
    --- PASS: TestConfigMap/aws-auth-open-source-case-keys.yaml (0.01s)
    --- PASS: TestConfigMap/aws-auth-lower-case-top-keys.yaml (0.01s)
    --- PASS: TestConfigMap/aws-auth-crazy-case-top-keys.yaml (0.01s)
    --- PASS: TestConfigMap/aws-auth-space-out-of-place.yaml (0.01s)
    --- PASS: TestConfigMap/aws-auth-missing-bar.yaml (0.01s)
PASS
coverage: 55.7% of statements
ok  	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/configmap	1.436s	coverage: 55.7% of statements
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/apis/iamauthenticator	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/apis/iamauthenticator/v1alpha1	[no test files]
=== RUN   TestIAMIdentityMappingCreation
time="2020-02-20T22:04:29-08:00" level=info msg="creating event broadcaster"
time="2020-02-20T22:04:29-08:00" level=info msg="setting up event handlers"
--- PASS: TestIAMIdentityMappingCreation (0.00s)
PASS
coverage: 36.9% of statements
ok  	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/controller	1.430s	coverage: 36.9% of statements
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/clientset/versioned	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/clientset/versioned/fake	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/clientset/versioned/scheme	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/clientset/versioned/typed/iamauthenticator/v1alpha1	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/clientset/versioned/typed/iamauthenticator/v1alpha1/fake	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/informers/externalversions	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/informers/externalversions/iamauthenticator	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/informers/externalversions/iamauthenticator/v1alpha1	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/informers/externalversions/internalinterfaces	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/crd/generated/listers/iamauthenticator/v1alpha1	[no test files]
?   	sigs.k8s.io/aws-iam-authenticator/pkg/mapper/file	[no test files]
=== RUN   TestAuthenticateNonPostError
time="2020-02-20T22:04:31-08:00" level=error msg="unexpected request method" client="192.0.2.1:1234" method=GET path=/authenticate
--- PASS: TestAuthenticateNonPostError (0.00s)
=== RUN   TestAuthenticateNonPostErrorCRD
time="2020-02-20T22:04:31-08:00" level=error msg="unexpected request method" client="192.0.2.1:1234" method=GET path=/authenticate
--- PASS: TestAuthenticateNonPostErrorCRD (0.00s)
=== RUN   TestAuthenticateEmptyBody
time="2020-02-20T22:04:31-08:00" level=error msg="could not parse request body" client="192.0.2.1:1234" error=EOF method=POST path=/authenticate
--- PASS: TestAuthenticateEmptyBody (0.00s)
=== RUN   TestAuthenticateEmptyBodyCRD
time="2020-02-20T22:04:31-08:00" level=error msg="could not parse request body" client="192.0.2.1:1234" error=EOF method=POST path=/authenticate
--- PASS: TestAuthenticateEmptyBodyCRD (0.00s)
=== RUN   TestAuthenticateUnableToDecodeBody
time="2020-02-20T22:04:31-08:00" level=error msg="could not parse request body" client="192.0.2.1:1234" error="invalid character 'o' in literal null (expecting 'u')" method=POST path=/authenticate
--- PASS: TestAuthenticateUnableToDecodeBody (0.00s)
=== RUN   TestAuthenticateUnableToDecodeBodyCRD
time="2020-02-20T22:04:31-08:00" level=error msg="could not parse request body" client="192.0.2.1:1234" error="invalid character 'o' in literal null (expecting 'u')" method=POST path=/authenticate
--- PASS: TestAuthenticateUnableToDecodeBodyCRD (0.00s)
=== RUN   TestAuthenticateVerifierError
time="2020-02-20T22:04:31-08:00" level=warning msg="access denied" client="192.0.2.1:1234" error="There was an error" method=POST path=/authenticate
--- PASS: TestAuthenticateVerifierError (0.00s)
=== RUN   TestAuthenticateVerifierErrorCRD
time="2020-02-20T22:04:31-08:00" level=warning msg="access denied" client="192.0.2.1:1234" error="There was an error" method=POST path=/authenticate
--- PASS: TestAuthenticateVerifierErrorCRD (0.00s)
=== RUN   TestAuthenticateVerifierSTSError
time="2020-02-20T22:04:31-08:00" level=warning msg="access denied" client="192.0.2.1:1234" error="sts getCallerIdentity failed: There was an error" method=POST path=/authenticate
--- PASS: TestAuthenticateVerifierSTSError (0.00s)
=== RUN   TestAuthenticateVerifierSTSErrorCRD
time="2020-02-20T22:04:31-08:00" level=warning msg="access denied" client="192.0.2.1:1234" error="sts getCallerIdentity failed: There was an error" method=POST path=/authenticate
--- PASS: TestAuthenticateVerifierSTSErrorCRD (0.00s)
=== RUN   TestAuthenticateVerifierNotMapped
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid= arn= client="192.0.2.1:1234" method=POST path=/authenticate session= userid=
time="2020-02-20T22:04:31-08:00" level=warning msg="access denied" arn= client="192.0.2.1:1234" error="ARN is not mapped" method=POST path=/authenticate
--- PASS: TestAuthenticateVerifierNotMapped (0.00s)
=== RUN   TestAuthenticateVerifierNotMappedCRD
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid= arn= client="192.0.2.1:1234" method=POST path=/authenticate session= userid=
time="2020-02-20T22:04:31-08:00" level=warning msg="access denied" arn= client="192.0.2.1:1234" error="ARN is not mapped" method=POST path=/authenticate
--- PASS: TestAuthenticateVerifierNotMappedCRD (0.00s)
=== RUN   TestAuthenticateVerifierRoleMapping
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid=ABCDEF accountid=0123456789012 arn="arn:aws:iam::0123456789012:role/Test" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:role/Test" client="192.0.2.1:1234" groups="[sys:admin listers]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username=TestUser
--- PASS: TestAuthenticateVerifierRoleMapping (0.00s)
=== RUN   TestAuthenticateVerifierRoleMappingCRD
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:role/Test" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:role/Test" client="192.0.2.1:1234" groups="[sys:admin listers]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username=TestUser
--- PASS: TestAuthenticateVerifierRoleMappingCRD (0.00s)
=== RUN   TestAuthenticateVerifierUserMapping
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" groups="[sys:admin listers]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username=TestUser
--- PASS: TestAuthenticateVerifierUserMapping (0.00s)
=== RUN   TestAuthenticateVerifierUserMappingCRD
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" groups="[sys:admin listers]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username=TestUser
--- PASS: TestAuthenticateVerifierUserMappingCRD (0.00s)
=== RUN   TestAuthenticateVerifierAccountMappingForUser
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" groups="[]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username="arn:aws:iam::0123456789012:user/Test"
--- PASS: TestAuthenticateVerifierAccountMappingForUser (0.00s)
=== RUN   TestAuthenticateVerifierAccountMappingForUserCRD
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:user/Test" client="192.0.2.1:1234" groups="[]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username="arn:aws:iam::0123456789012:user/Test"
--- PASS: TestAuthenticateVerifierAccountMappingForUserCRD (0.00s)
=== RUN   TestAuthenticateVerifierAccountMappingForRole
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:assumed-role/Test/extra" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:role/Test" client="192.0.2.1:1234" groups="[]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username="arn:aws:iam::0123456789012:role/Test"
--- PASS: TestAuthenticateVerifierAccountMappingForRole (0.00s)
=== RUN   TestAuthenticateVerifierAccountMappingForRoleCRD
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:assumed-role/Test/extra" client="192.0.2.1:1234" method=POST path=/authenticate session= userid=Test
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:role/Test" client="192.0.2.1:1234" groups="[]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:Test" username="arn:aws:iam::0123456789012:role/Test"
--- PASS: TestAuthenticateVerifierAccountMappingForRoleCRD (0.00s)
=== RUN   TestAuthenticateVerifierNodeMapping
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:role/TestNodeRole" client="192.0.2.1:1234" method=POST path=/authenticate session=i-0c6f21bf1f24f9708 userid=TestNodeRole
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:role/TestNodeRole" client="192.0.2.1:1234" groups="[system:nodes system:bootstrappers]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:TestNodeRole" username="system:node:ip-172-31-27-14"
--- PASS: TestAuthenticateVerifierNodeMapping (0.00s)
=== RUN   TestAuthenticateVerifierNodeMappingCRD
time="2020-02-20T22:04:31-08:00" level=info msg="STS response" accesskeyid= accountid=0123456789012 arn="arn:aws:iam::0123456789012:role/TestNodeRole" client="192.0.2.1:1234" method=POST path=/authenticate session=i-0c6f21bf1f24f9708 userid=TestNodeRole
time="2020-02-20T22:04:31-08:00" level=info msg="access granted" arn="arn:aws:iam::0123456789012:role/TestNodeRole" client="192.0.2.1:1234" groups="[system:nodes system:bootstrappers]" method=POST path=/authenticate uid="aws-iam-authenticator:0123456789012:TestNodeRole" username="system:node:ip-172-31-27-14"
--- PASS: TestAuthenticateVerifierNodeMappingCRD (0.00s)
=== RUN   TestRenderTemplate
=== RUN   TestRenderTemplate/a-{{EC2PrivateDNSName}}-b
=== RUN   TestRenderTemplate/a-{{EC2PrivateDNSName}}-b#01
=== RUN   TestRenderTemplate/a-{{AccountID}}-b
=== RUN   TestRenderTemplate/a-{{SessionName}}-b
=== RUN   TestRenderTemplate/a-{{SessionName}}-b#01
=== RUN   TestRenderTemplate/a-{{SessionNameRaw}}-b
=== RUN   TestRenderTemplate/a-{{AccountID}}-{{SessionName}}-{{SessionNameRaw}}-b
--- PASS: TestRenderTemplate (0.00s)
    --- PASS: TestRenderTemplate/a-{{EC2PrivateDNSName}}-b (0.00s)
    --- PASS: TestRenderTemplate/a-{{EC2PrivateDNSName}}-b#01 (0.00s)
    --- PASS: TestRenderTemplate/a-{{AccountID}}-b (0.00s)
    --- PASS: TestRenderTemplate/a-{{SessionName}}-b (0.00s)
    --- PASS: TestRenderTemplate/a-{{SessionName}}-b#01 (0.00s)
    --- PASS: TestRenderTemplate/a-{{SessionNameRaw}}-b (0.00s)
    --- PASS: TestRenderTemplate/a-{{AccountID}}-{{SessionName}}-{{SessionNameRaw}}-b (0.00s)
PASS
coverage: 52.9% of statements
ok  	sigs.k8s.io/aws-iam-authenticator/pkg/server	1.315s	coverage: 52.9% of statements
=== RUN   TestCacheFilename
--- PASS: TestCacheFilename (0.00s)
=== RUN   TestNewFileCacheProvider_Missing
Cache file .kube/cache/aws-iam-authenticator/credentials.yaml does not exist.
--- PASS: TestNewFileCacheProvider_Missing (0.00s)
=== RUN   TestNewFileCacheProvider_BadPermissions
--- PASS: TestNewFileCacheProvider_BadPermissions (0.00s)
=== RUN   TestNewFileCacheProvider_Unlockable
--- PASS: TestNewFileCacheProvider_Unlockable (0.00s)
=== RUN   TestNewFileCacheProvider_Unreadable
--- PASS: TestNewFileCacheProvider_Unreadable (0.00s)
=== RUN   TestNewFileCacheProvider_Unparseable
--- PASS: TestNewFileCacheProvider_Unparseable (0.00s)
=== RUN   TestNewFileCacheProvider_Empty
--- PASS: TestNewFileCacheProvider_Empty (0.00s)
=== RUN   TestNewFileCacheProvider_ExistingCluster
--- PASS: TestNewFileCacheProvider_ExistingCluster (0.00s)
=== RUN   TestNewFileCacheProvider_ExistingARN
--- PASS: TestNewFileCacheProvider_ExistingARN (0.00s)
=== RUN   TestFileCacheProvider_Retrieve_NoExpirer
Cache file .kube/cache/aws-iam-authenticator/credentials.yaml does not exist.
No cached credential available.  Refreshing...
Unable to cache credential: ProviderNotExpirer: provider stubProvider does not support ExpiresAt()
--- PASS: TestFileCacheProvider_Retrieve_NoExpirer (0.00s)
=== RUN   TestFileCacheProvider_Retrieve_WithExpirer_Unlockable
Cache file .kube/cache/aws-iam-authenticator/credentials.yaml does not exist.
No cached credential available.  Refreshing...
Unable to write lock file .kube/cache/aws-iam-authenticator/credentials.yaml: lock stuck, needs wd-40
--- PASS: TestFileCacheProvider_Retrieve_WithExpirer_Unlockable (0.00s)
=== RUN   TestFileCacheProvider_Retrieve_WithExpirer_Unwritable
Cache file .kube/cache/aws-iam-authenticator/credentials.yaml does not exist.
No cached credential available.  Refreshing...
Unable to update credential cache .kube/cache/aws-iam-authenticator/credentials.yaml: can't write cache
--- PASS: TestFileCacheProvider_Retrieve_WithExpirer_Unwritable (0.00s)
=== RUN   TestFileCacheProvider_Retrieve_WithExpirer_Writable
Cache file .kube/cache/aws-iam-authenticator/credentials.yaml does not exist.
No cached credential available.  Refreshing...
Updated cached credential
--- PASS: TestFileCacheProvider_Retrieve_WithExpirer_Writable (0.00s)
=== RUN   TestFileCacheProvider_Retrieve_CacheHit
--- PASS: TestFileCacheProvider_Retrieve_CacheHit (0.00s)
=== RUN   TestSTSEndpoints
--- PASS: TestSTSEndpoints (0.00s)
=== RUN   TestVerifyTokenPreSTSValidations
--- PASS: TestVerifyTokenPreSTSValidations (0.01s)
=== RUN   TestVerifyHTTPError
2020/02/20 22:04:31 RoundTripper returned a response & error; ignoring response
--- PASS: TestVerifyHTTPError (0.00s)
=== RUN   TestVerifyHTTP403
--- PASS: TestVerifyHTTP403 (0.00s)
=== RUN   TestVerifyBodyReadError
--- PASS: TestVerifyBodyReadError (0.00s)
=== RUN   TestVerifyUnmarshalJSONError
--- PASS: TestVerifyUnmarshalJSONError (0.00s)
=== RUN   TestVerifyInvalidCanonicalARNError
--- PASS: TestVerifyInvalidCanonicalARNError (0.00s)
=== RUN   TestVerifyInvalidUserIDError
--- PASS: TestVerifyInvalidUserIDError (0.00s)
=== RUN   TestVerifyNoSession
--- PASS: TestVerifyNoSession (0.00s)
=== RUN   TestVerifySessionName
--- PASS: TestVerifySessionName (0.00s)
=== RUN   TestVerifyCanonicalARN
--- PASS: TestVerifyCanonicalARN (0.00s)
PASS
coverage: 70.0% of statements
ok  	sigs.k8s.io/aws-iam-authenticator/pkg/token	1.152s	coverage: 70.0% of statements
go tool cover -html=coverage.out -o coverage.html
bash-5.0$

This is a great improvement! Mostly stylistic changes, and a reduction in the amount of logging

Thanks !!!
I somehow knew that there would be an idea of removing the loggings but given the fact for every instance the whole workflow prints the messages once except few which just roam around for cache lookup.

Thank you so much @mogren and @micahhausler for helping improve the Go code quality.
I have updated the code with comments incorporation.


text:
Bump client to 2.15.5

What type of PR is this?
/kind feature

What this PR does / why we need it:
Bumps client to 2.15.5-41-gc010524
Which issue(s) this PR fixes:

Fixes #
Requirements:

[ ] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] tested upgrade from previous version

Special notes for your reviewer:
Release note:
none

coverage: 82.552%. remained the same
when pulling a522498dde91f24a289fb35eac1bd48be9194dc8 on t-mialve:t-mialve/2.15.5-upgrade
into abb9837bf545561bc4c64dbe82fddb17eec3f31a on kubernetes-sigs:development.

source: gharchive/pull-request

authors: coveralls t-mialve

repo: kubernetes-sigs/azurelustre-csi-driver

license_type: permissive

license_source: github-api