text:
Pod failing to get IP address assigned

Version

Karpenter Version: 0.28.0-rc.2

Kubernetes Version: v1.26
Expected Behavior
Raised under this now closed issue: Karpenter is not aware of the Custom Networking VPC CNI pod limit per node
@bwagner5
https://github.com/aws/karpenter/issues/2273#issuecomment-1551853677

#3516 should address this and will be released in v0.28.0

Running 0.28.0-rc.2 with the option aws.reservedENIs set to 1 as described.
Pods should be assigned an IP address.
Actual Behavior
The pod fails to get assigned an IP address and does not start.
Steps to Reproduce the Problem
Scale a deployment beyond what current nodes can handle to ensure a new node is provisioned by Karpenter.
Resource Specs and Logs
❯ kubectl -n karpenter describe cm karpenter-global-settings | grep reserved -A 2
aws.reservedENIs:
----
1

❯ kubectl scale deployment ahoy-hello-world --replicas 250
deployment.apps/ahoy-hello-world scaled

2023-06-02T18:52:54.220Z        INFO    controller.provisioner  found provisionable pod(s)      {"commit": "2be6616-dirty", "pods": 45}
2023-06-02T18:52:54.221Z        INFO    controller.provisioner  computed new machine(s) to fit pod(s)   {"commit": "2be6616-dirty", "machines": 1, "pods": 45}
2023-06-02T18:52:54.234Z        INFO    controller.provisioner  created machine {"commit": "2be6616-dirty", "provisioner": "default", "requests": {"cpu":"155m","memory":"120Mi","pods":"48"}, "instance-types": "c3.large, c3.xlarge, c4.large, c4.xlarge, c5.2xlarge and 95 other(s)"}
2023-06-02T18:52:54.434Z        DEBUG   controller.machine_lifecycle    discovered subnets      {"commit": "2be6616-dirty", "machine": "default-h2rdv", "provisioner": "default", "subnets": ["subnet-034313c968a897f9a (eu-west-1a)", "subnet-0c4a2e5d88292ec83 (eu-west-1b)", "subnet-0de7acfef28444117 (eu-west-1c)"]}
2023-06-02T18:52:54.998Z        DEBUG   controller.machine_lifecycle    created launch template {"commit": "2be6616-dirty", "machine": "default-h2rdv", "provisioner": "default", "launch-template-name": "karpenter.k8s.aws/16237060489344945276", "id": "lt-00f8dc861d6b9fb24"}
2023-06-02T18:52:57.094Z        INFO    controller.machine_lifecycle    launched machine        {"commit": "2be6616-dirty", "machine": "default-h2rdv", "provisioner": "default", "provider-id": "aws:///eu-west-1c/i-$instance_id", "instance-type": "m3.medium", "zone": "eu-west-1c", "capacity-type": "spot", "allocatable": {"cpu":"940m","ephemeral-storage":"17Gi","memory":"1987Mi","pods":"110"}}
2023-06-02T18:53:24.421Z        DEBUG   controller.machine_lifecycle    registered machine      {"commit": "2be6616-dirty", "machine": "default-h2rdv", "provisioner": "default", "provider-id": "aws:///eu-west-1c/i-$instance_id", "node": "ip-10-3-9-89.eu-west-1.compute.internal"}
2023-06-02T18:53:47.958Z        DEBUG   controller.machine_lifecycle    initialized machine     {"commit": "2be6616-dirty", "machine": "default-h2rdv", "provisioner": "default", "provider-id": "aws:///eu-west-1c/i-$instance_id", "node": "ip-10-3-9-89.eu-west-1.compute.internal"}
2023-06-02T18:54:01.236Z        DEBUG   controller.deprovisioning       discovered instance types       {"commit": "2be6616-dirty", "count": 622}
2023-06-02T18:54:01.421Z        DEBUG   controller.deprovisioning       discovered subnets      {"commit": "2be6616-dirty", "subnets": ["subnet-034313c968a897f9a (eu-west-1a)", "subnet-0c4a2e5d88292ec83 (eu-west-1b)", "subnet-0de7acfef28444117 (eu-west-1c)"]}

❯ kubectl get pods | grep ContainerCreating | wc -l
     198

❯ kubectl describe pod ahoy-hello-world-79f545d68-zz5tx | grep Events -A 100
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               5m33s                default-scheduler  Successfully assigned test/ahoy-hello-world-79f545d68-zz5tx to ip-10-3-1-2.eu-west-1.compute.internal
  Warning  FailedCreatePodSandBox  5m31s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0500e5cfbe2f354d7ffbcf262057ed7c2316f6e9464a75e43ccf399526f631b1": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  5m16s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d05f4c6fc93373b69f86719028dab88974b45799f65730313728b214e45b3207": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  5m2s                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "87850f9ddf22ee77e24d2e26c5e98366b52f9f48b32e2bb1458a2a2174cd49a4": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  4m50s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6f07794c04ae201bcdf4469feecc53f4078dfa0f9df48edd1ff71366205f12e6": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  4m37s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bc60fed3c06f3455e8e915d9b45934d98a25305c62658c7a5c36ff1139bdc118": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  4m22s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "76fb8238fa20331638974695eac55c736aafb41a7564f32e91bbbcf17f3ffe4f": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  4m10s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8bf3110ea8ae3e28e0f649bdaa8e787b19666a88420c9c1084d1697c11c09ce7": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  3m58s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f0696e2b70f9ce38b9c80fe29f53955468d28591f8ee731bc5bc2e9bc6d8e33b": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  3m44s                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e3dec9ede940f9f4a14a1cfd2ca64f7b372f8e8b8914ea730970b2d5bef5ebae": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  0s (x11 over 2m50s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "956caa82c69220f2a2c9b84f998e72bf3a7d8d61db3699f4e7cb197f138c728a": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Hi @bwagner5,
I've misunderstood the intention of that option.
Can someone please have a look at the information below and point me in the right direction.
I deployed a new VPC and EKS cluster, there's nothing else running in either.
VPC configuration:
VPC CIDR | 10.1.0.0/19 |   |   |
-- | -- | -- | -- | --
AWS Region | eu-west-1 |   |   |
Addressable Hosts | 8178 |   |   |
Spare Capacity | 2046 |   |   |
  |   |   |   |
Subnet Type | Subnet address | Range of addresses | Useable IPs | Hosts
Public | 10.1.0.0/22 | 10.1.0.0 - 10.1.3.255 | 10.1.0.1 - 10.1.3.254 | 1022
Public | 10.1.4.0/22 | 10.1.4.0 - 10.1.7.255 | 10.1.4.1 - 10.1.7.254 | 1022
Public  | 10.1.8.0/22 | 10.1.8.0 - 10.1.11.255 | 10.1.8.1 - 10.1.11.254 | 1022
Private | 10.1.12.0/22 | 10.1.12.0 - 10.1.15.255 | 10.1.12.1 - 10.1.15.254 | 1022
Private |10.1.16.0/22 | 10.1.16.0 - 10.1.19.255 | 10.1.16.1 - 10.1.19.254 | 1022
Private |10.1.20.0/22 | 10.1.20.0 - 10.1.23.255 | 10.1.20.1 - 10.1.23.254 | 1022
Spare | 10.1.24.0/21 | 10.1.24.0 - 10.1.31.255 | 10.1.24.1 - 10.1.31.254 | 2046
Kubernetes nodes:

Name
Status
Version
IP

ip-10-1-6-216.eu-west-1.compute.internal
Ready
v1.26.4-eks-597964d
10.1.6.216

ip-10-1-8-251.eu-west-1.compute.internal
Ready
v1.26.4-eks-597964d
10.1.8.251

Subnet 10.1.0.0/22 with node 'ip-10-1-6-216.eu-west-1.compute.internal' has 998 available IPv4 addresses.
Subnet 10.1.8.0/22 with node 'ip-10-1-8-251.eu-west-1.compute.internal' has 999 available IPv4 addresses.
I'm using the Amazon VPC CNI plugin for Kubernetes Amazon EKS add-on.
I have added these configuration values:
{
  "env":{
    "ENABLE_PREFIX_DELEGATION":"true",
    "WARM_PREFIX_TARGET":"1"
  }
}

I can see the configuration values are applied:
kubectl describe node | grep Allocatable -A 10 | grep pods
  pods:               110
  pods:               110

kubectl -n kube-system describe pod aws-node-77sh5 | grep -i prefix
      ENABLE_PREFIX_DELEGATION:               true
      WARM_PREFIX_TARGET:                     1

kubectl -n kube-system describe pod aws-node-sv8qb | grep -i prefix
      ENABLE_PREFIX_DELEGATION:               true
      WARM_PREFIX_TARGET:                     1

I have one pod running in a test namespace:
kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
ahoy-hello-world-79f545d68-7gbvx   1/1     Running   0          18m

Please take a look at the AWSNodeTemplate and Provisioner in the description.
Karpenter looks happy.

Karpenter logs
2023-06-06T00:03:20.849Z        DEBUG   Successfully created the logger.
2023-06-06T00:03:20.849Z        DEBUG   Logging level set to: debug
2023-06-06T00:03:23.609Z        DEBUG   controller      discovered region       {"commit": "2be6616-dirty", "region": "eu-west-1"}
2023-06-06T00:03:23.609Z        DEBUG   controller      discovered cluster endpoint     {"commit": "2be6616-dirty", "cluster-endpoint": "https://blah.sk1.eu-west-1.eks.amazonaws.com"}
2023-06-06T00:03:23.613Z        DEBUG   controller      discovered kube dns     {"commit": "2be6616-dirty", "kube-dns-ip": "172.20.0.10"}
2023-06-06T00:03:23.613Z        DEBUG   controller      discovered version      {"commit": "2be6616-dirty", "version": "v0.28.0-rc.2"}
2023-06-06T00:03:23.613Z        DEBUG   controller      Registering 2 clients   {"commit": "2be6616-dirty"}
2023-06-06T00:03:23.614Z        DEBUG   controller      Registering 2 informer factories        {"commit": "2be6616-dirty"}
2023-06-06T00:03:23.614Z        DEBUG   controller      Registering 3 informers {"commit": "2be6616-dirty"}
2023-06-06T00:03:23.614Z        DEBUG   controller      Registering 5 controllers       {"commit": "2be6616-dirty"}
2023-06-06T00:03:23.614Z        INFO    controller      Starting server {"commit": "2be6616-dirty", "path": "/metrics", "kind": "metrics", "addr": "[::]:8000"}
2023-06-06T00:03:23.614Z        INFO    controller      Starting server {"commit": "2be6616-dirty", "kind": "health probe", "addr": "[::]:8081"}
2023-06-06T00:03:23.715Z        INFO    controller      attempting to acquire leader lease karpenter/karpenter-leader-election...
        {"commit": "2be6616-dirty"}
2023-06-06T00:03:23.736Z        INFO    controller      Starting informers...   {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.720Z        INFO    controller      successfully acquired lease karpenter/karpenter-leader-election
        {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.720Z        INFO    controller.provisioner  starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.720Z        INFO    controller.deprovisioning       starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.720Z        DEBUG   controller.deprovisioning       waiting on cluster sync {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.720Z        INFO    controller.metric_scraper       starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.720Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "source": "kind source: *v1.DaemonSet"}
2023-06-06T00:03:48.720Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet"}
2023-06-06T00:03:48.720Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "pod_state", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-06-06T00:03:48.720Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "pod_state", "controllerGroup": "", "controllerKind": "Pod"}
2023-06-06T00:03:48.720Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "provisioner_trigger", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-06-06T00:03:48.720Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "provisioner_trigger", "controllerGroup": "", "controllerKind": "Pod"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "node_state", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "node_state", "controllerGroup": "", "controllerKind": "Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "machine-state", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "machine-state", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "provisioner_state", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "provisioner_state", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "termination", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "termination", "controllerGroup": "", "controllerKind": "Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1alpha5.Provisioner"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "source": "kind source: *v1.Pod"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "provisioner_metrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "provisioner_metrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "pod_metrics", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "pod_metrics", "controllerGroup": "", "controllerKind": "Pod"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1alpha5.Provisioner"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "source": "kind source: *v1.Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner"}
2023-06-06T00:03:48.721Z        INFO    controller.machine_garbagecollection    starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "machine_lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "machine_lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Node"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "machine_lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "awsnodetemplate", "controllerGroup": "karpenter.k8s.aws", "controllerKind": "AWSNodeTemplate", "source": "kind source: *v1alpha1.AWSNodeTemplate"}
2023-06-06T00:03:48.721Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "awsnodetemplate", "controllerGroup": "karpenter.k8s.aws", "controllerKind": "AWSNodeTemplate"}
2023-06-06T00:03:48.722Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "machine_termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1alpha5.Machine"}
2023-06-06T00:03:48.722Z        INFO    controller      Starting EventSource    {"commit": "2be6616-dirty", "controller": "machine_termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "source": "kind source: *v1.Node"}
2023-06-06T00:03:48.722Z        INFO    controller      Starting Controller     {"commit": "2be6616-dirty", "controller": "machine_termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine"}
2023-06-06T00:03:48.722Z        INFO    controller.machine_garbagecollection    starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.722Z        INFO    controller.machine_link starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.722Z        INFO    controller.pricing      starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.722Z        INFO    controller.interruption starting controller     {"commit": "2be6616-dirty"}
2023-06-06T00:03:48.722Z        DEBUG   controller.interruption watching interruption queue     {"commit": "2be6616-dirty", "queue": "Karpenter-demo9"}
2023-06-06T00:03:48.778Z        DEBUG   controller      hydrated launch template cache  {"commit": "2be6616-dirty", "tag-key": "karpenter.k8s.aws/cluster", "tag-value": "demo9", "count": 0}
2023-06-06T00:03:48.821Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "provisioner_trigger", "controllerGroup": "", "controllerKind": "Pod", "worker count": 10}
2023-06-06T00:03:48.821Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "daemonset", "controllerGroup": "apps", "controllerKind": "DaemonSet", "worker count": 10}
2023-06-06T00:03:48.821Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "pod_state", "controllerGroup": "", "controllerKind": "Pod", "worker count": 10}
2023-06-06T00:03:48.821Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "node_state", "controllerGroup": "", "controllerKind": "Node", "worker count": 10}
2023-06-06T00:03:48.821Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "pod_metrics", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2023-06-06T00:03:48.822Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "machine_lifecycle", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 1000}
2023-06-06T00:03:48.822Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "machine-state", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 10}
2023-06-06T00:03:48.823Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "termination", "controllerGroup": "", "controllerKind": "Node", "worker count": 100}
2023-06-06T00:03:48.827Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "machine_termination", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 100}
2023-06-06T00:03:48.831Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "consistency", "controllerGroup": "karpenter.sh", "controllerKind": "Machine", "worker count": 10}
2023-06-06T00:03:48.926Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "provisioner_state", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10}
2023-06-06T00:03:48.927Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "awsnodetemplate", "controllerGroup": "karpenter.k8s.aws", "controllerKind": "AWSNodeTemplate", "worker count": 10}
2023-06-06T00:03:48.931Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "node", "controllerGroup": "", "controllerKind": "Node", "worker count": 10}
2023-06-06T00:03:48.932Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "counter", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 10}
2023-06-06T00:03:48.932Z        INFO    controller      Starting workers        {"commit": "2be6616-dirty", "controller": "provisioner_metrics", "controllerGroup": "karpenter.sh", "controllerKind": "Provisioner", "worker count": 1}
2023-06-06T00:03:49.038Z        DEBUG   controller.awsnodetemplate      discovered subnets      {"commit": "2be6616-dirty", "awsnodetemplate": "default", "subnets": ["subnet-0ab64e974252c4e3d (eu-west-1c)", "subnet-0ea7f03a13ffa5ded (eu-west-1a)", "subnet-045cf720eaf23a71a (eu-west-1b)"]}
2023-06-06T00:03:49.059Z        INFO    controller.pricing      updated spot pricing with instance types and offerings  {"commit": "2be6616-dirty", "instance-type-count": 642, "offering-count": 1805}
2023-06-06T00:03:49.133Z        ERROR   controller.node_state   PersistentVolume source 'AWSElasticBlockStore' uses an in-tree storage plugin which is unsupported by Karpenter and is deprecated by Kubernetes. Scale-ups may fail because Karpenter will not discover driver limits. Use a PersistentVolume that references the 'CSI' volume source for Karpenter auto-scaling support.       {"commit": "2be6616-dirty", "node": "ip-10-1-8-251.eu-west-1.compute.internal", "pod": "cost-analyzer-prometheus-server-56bdd9bc94-h6jzn", "volume": "config-volume", "volume": "storage-volume", "persistent-volume": "pvc-1ea59872-07f1-4a38-8a2b-defc50faa77e"}
2023-06-06T00:03:49.133Z        ERROR   controller.node_state   StorageClass .spec.provisioner uses an in-tree storage plugin which is unsupported by Karpenter and is deprecated by Kubernetes. Scale-ups may fail because Karpenter will not discover driver limits. Create a new StorageClass with a .spec.provisioner referencing the CSI driver plugin name 'ebs.csi.aws.com'.     {"commit": "2be6616-dirty", "node": "ip-10-1-8-251.eu-west-1.compute.internal", "pod": "cost-analyzer-prometheus-server-56bdd9bc94-h6jzn", "volume": "config-volume", "volume": "storage-volume", "storage-class": "gp2", "provisioner": "kubernetes.io/aws-ebs"}
2023-06-06T00:03:49.133Z        ERROR   controller.node_state   PersistentVolume source 'AWSElasticBlockStore' uses an in-tree storage plugin which is unsupported by Karpenter and is deprecated by Kubernetes. Scale-ups may fail because Karpenter will not discover driver limits. Use a PersistentVolume that references the 'CSI' volume source for Karpenter auto-scaling support.       {"commit": "2be6616-dirty", "node": "ip-10-1-8-251.eu-west-1.compute.internal", "pod": "cost-analyzer-8864bdff8-mwj4k", "volume": "tmp", "volume": "nginx-conf", "volume": "persistent-configs", "persistent-volume": "pvc-caba4b0c-d853-4415-99ae-517d19335527"}
2023-06-06T00:03:49.141Z        DEBUG   controller.awsnodetemplate      discovered security groups      {"commit": "2be6616-dirty", "awsnodetemplate": "default", "security-groups": ["sg-0ad21b0f5ad50723a"]}
2023-06-06T00:03:49.142Z        DEBUG   controller.awsnodetemplate      discovered kubernetes version   {"commit": "2be6616-dirty", "awsnodetemplate": "default", "version": "1.26"}
2023-06-06T00:03:49.530Z        DEBUG   controller.awsnodetemplate      discovered amis {"commit": "2be6616-dirty", "awsnodetemplate": "default", "ids": "ami-077ae7ace3bbba6fd, ami-05c44b4dc9f0ccf07, ami-05c44b4dc9f0ccf07, ami-0ae44cb1ba673fc31, ami-0ae44cb1ba673fc31, ami-0674532ce2b29c8e4", "count": 6}
2023-06-06T00:03:50.623Z        DEBUG   controller.deprovisioning       discovered instance types       {"commit": "2be6616-dirty", "count": 622}
2023-06-06T00:03:50.771Z        DEBUG   controller.deprovisioning       discovered offerings for instance types {"commit": "2be6616-dirty", "zones": ["eu-west-1a", "eu-west-1b", "eu-west-1c"], "instance-type-count": 624, "node-template": "default"}
2023-06-06T00:03:50.955Z        INFO    controller.pricing      updated on-demand pricing       {"commit": "2be6616-dirty", "instance-type-count": 625}

I scaled up a deployment to 250 pods.
Karpenter adds a new node successfully and pods get scheduled on it.

Karpenter logs - after scaling
2023-06-06T00:27:03.585Z        INFO    controller.provisioner  found provisionable pod(s)      {"commit": "2be6616-dirty", "pods": 45}
2023-06-06T00:27:03.585Z        INFO    controller.provisioner  computed new machine(s) to fit pod(s)   {"commit": "2be6616-dirty", "machines": 1, "pods": 45}
2023-06-06T00:27:03.607Z        INFO    controller.provisioner  created machine {"commit": "2be6616-dirty", "provisioner": "default", "requests": {"cpu":"155m","memory":"120Mi","pods":"48"}, "instance-types": "c3.large, c3.xlarge, c4.2xlarge, c4.large, c4.xlarge and 95 other(s)"}
2023-06-06T00:27:03.838Z        DEBUG   controller.machine_lifecycle    discovered subnets      {"commit": "2be6616-dirty", "machine": "default-zxhg9", "provisioner": "default", "subnets": ["subnet-0ab64e974252c4e3d (eu-west-1c)", "subnet-0ea7f03a13ffa5ded (eu-west-1a)", "subnet-045cf720eaf23a71a (eu-west-1b)"]}
2023-06-06T00:27:04.485Z        DEBUG   controller.machine_lifecycle    created launch template {"commit": "2be6616-dirty", "machine": "default-zxhg9", "provisioner": "default", "launch-template-name": "karpenter.k8s.aws/15287919082325933589", "id": "lt-0d0ed348f4f2b8b97"}
2023-06-06T00:27:05.981Z        DEBUG   controller.deprovisioning       waiting on cluster sync {"commit": "2be6616-dirty"}
2023-06-06T00:27:06.682Z        INFO    controller.machine_lifecycle    launched machine        {"commit": "2be6616-dirty", "machine": "default-zxhg9", "provisioner": "default", "provider-id": "aws:///eu-west-1c/i-0f00e5c3d69ecc5ed", "instance-type": "m3.medium", "zone": "eu-west-1c", "capacity-type": "spot", "allocatable": {"cpu":"940m","ephemeral-storage":"17Gi","memory":"1987Mi","pods":"110"}}
2023-06-06T00:27:32.880Z        DEBUG   controller.machine_lifecycle    registered machine      {"commit": "2be6616-dirty", "machine": "default-zxhg9", "provisioner": "default", "provider-id": "aws:///eu-west-1c/i-0f00e5c3d69ecc5ed", "node": "ip-10-1-11-1.eu-west-1.compute.internal"}
2023-06-06T00:27:57.971Z        DEBUG   controller.machine_lifecycle    initialized machine     {"commit": "2be6616-dirty", "machine": "default-zxhg9", "provisioner": "default", "provider-id": "aws:///eu-west-1c/i-0f00e5c3d69ecc5ed", "node": "ip-10-1-11-1.eu-west-1.compute.internal"}

New Kubernetes node:

Name
Status
Version
IP

ip-10-1-11-1.eu-west-1.compute.internal
Ready
v1.26.4-eks-597964d
10.1.11.1

The new node was deployed in subnet 10.1.8.0/22 which now has 977 IPv4 addresses available, a change of -22.
198 pods are stuck in the 'ContainerCreating' state.
kubectl get pods | grep ContainerCreating | wc -l
     198

52 pods are in the 'Running' state.
kubectl get pods | grep Running | wc -l
      52

kubectl get pods -owide
NAME                               READY   STATUS              RESTARTS   AGE   IP            NODE                                       NOMINATED NODE   READINESS GATES
ahoy-hello-world-79f545d68-229fl   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-22kg4   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-24tmj   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-28fjt   1/1     Running             0          13m   10.1.10.201   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-29xvg   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-2bzz5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-2qwgx   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-2r4bp   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-2s5ml   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-2t5x7   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-42wsk   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-4d7z6   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-4kf96   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-4kskg   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-4kznk   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-4nf6d   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-4sk9k   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-4xhnt   1/1     Running             0          13m   10.1.9.255    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-576jd   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-57zlg   1/1     Running             0          13m   10.1.8.238    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-58clt   1/1     Running             0          13m   10.1.6.248    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-58ksw   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5bzb2   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5ddnn   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5fpwg   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5ftsp   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5hbk9   1/1     Running             0          13m   10.1.6.36     ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5hppg   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5nz6x   1/1     Running             0          13m   10.1.5.185    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5pvmz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5pw8q   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-5rf2h   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5rmqb   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-5sc7g   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-5wg7p   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-64hzm   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-65565   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-67ljq   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-6dqx2   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-6mwhd   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-6slpm   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-6t9ll   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-6tvrf   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-6z5x9   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-724w7   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-7gbvx   1/1     Running             0          37m   10.1.6.143    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-7gtbc   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-7gzxx   1/1     Running             0          13m   10.1.11.13    ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-7kfpc   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-7qnhj   1/1     Running             0          13m   10.1.10.94    ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-7qstm   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-7tflt   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-85m7q   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-894ct   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-8bbmz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-8khrx   1/1     Running             0          13m   10.1.6.203    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-8m5wj   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-99bmj   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-99f9x   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-9c859   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-9p4zz   1/1     Running             0          13m   10.1.6.28     ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-9rcbp   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-9skhl   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-b9mm2   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-b9xww   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-bhdpc   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-bn776   1/1     Running             0          13m   10.1.5.53     ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-c4x68   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-c5gj9   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-chw76   1/1     Running             0          13m   10.1.10.212   ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-cj4mh   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-ck5n5   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-ckxn5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-cxx52   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-drnc2   1/1     Running             0          13m   10.1.10.40    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-ds2kw   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-dvbkl   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-dww84   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-dxrrg   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-dzzd4   1/1     Running             0          13m   10.1.11.207   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-f2d96   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-f2hzc   1/1     Running             0          13m   10.1.4.218    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-f4hjj   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-f4hrt   1/1     Running             0          13m   10.1.7.174    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-f54nb   1/1     Running             0          13m   10.1.8.225    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-f57s5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-f5crg   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-f8fh2   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-fcmc5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-fgqfz   1/1     Running             0          13m   10.1.11.115   ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-fjtlk   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-fwcl4   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-fxst5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-fzmjh   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-g577k   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-g5c4r   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-g7cr4   1/1     Running             0          13m   10.1.5.170    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-gk7qj   1/1     Running             0          13m   10.1.10.235   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-gv274   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-gxwsn   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-gzpss   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-h4szj   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-h5nkp   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-h92zc   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-h95m7   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-h95vw   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-h9ngz   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-h9s2f   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-hf844   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-hh5lq   1/1     Running             0          13m   10.1.5.115    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-hnnbz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-hsmjx   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-hvk9n   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-hxjzf   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-j24qb   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-j8d4n   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-jkdmg   1/1     Running             0          13m   10.1.8.80     ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-jnbhp   1/1     Running             0          13m   10.1.7.253    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-jq9sp   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-jrfs5   1/1     Running             0          13m   10.1.7.30     ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-jrn7s   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-jxlbd   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-jxp76   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-k2gmz   1/1     Running             0          13m   10.1.6.123    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-k55nz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-k5pnm   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-kcmt4   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-kcw4d   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-kfqhk   1/1     Running             0          13m   10.1.5.57     ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-khqcw   1/1     Running             0          13m   10.1.5.168    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-kmgbc   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-kqdfr   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-ktwhx   1/1     Running             0          13m   10.1.9.91     ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-kwwbk   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-kzrnw   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-l57vb   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-l6mmg   1/1     Running             0          13m   10.1.4.242    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-l7mt9   1/1     Running             0          13m   10.1.11.47    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-l7tzz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-l7x4w   1/1     Running             0          13m   10.1.11.241   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-l97sh   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-lgmhv   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-lpdlk   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-lrj4z   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-lwl69   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-lwrb5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-lxt7k   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-m4qbt   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-m7l9x   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-m8sn5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mdbm9   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mdkbf   1/1     Running             0          13m   10.1.8.177    ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-mpwnz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mr4m9   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-mvd6r   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mvhbk   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mvw4p   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mwp8x   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mwrbl   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-mxd7f   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-mxjh7   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-n2g9t   1/1     Running             0          13m   10.1.10.246   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-n58sx   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-n5v75   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-nbnmp   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-nft2z   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-ngsf4   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-nr6bn   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-nt9ln   1/1     Running             0          13m   10.1.10.42    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-nwqq9   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-p4djf   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-p7sz7   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-pbz9n   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-pfd6d   1/1     Running             0          13m   10.1.7.147    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-pgbnr   1/1     Running             0          13m   10.1.11.57    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-pgqsb   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-pkhtp   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-pqv2t   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-ps6b6   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-pwrbf   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-pxlxz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-q4brs   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-q4c98   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-qczsr   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-qdxwc   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-qgq7x   1/1     Running             0          13m   10.1.4.230    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-qhh9j   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-qlw6k   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-qmp9j   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-qq7qc   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-qr968   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-rd2hq   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-rnqwf   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-rs5pn   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-rsxt4   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-rxxd7   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-s9jr4   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-sbwg5   1/1     Running             0          13m   10.1.8.81     ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-smwdw   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-t4l8d   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-t4tn9   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-t6r2p   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-t72xf   1/1     Running             0          12m   10.1.8.149    ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-t9bcx   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-th8nk   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-thhvz   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-tm4pd   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-tn6gr   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-trxtc   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-tz79r   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-v6b5v   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-v844d   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-v87cq   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-vddsd   1/1     Running             0          13m   10.1.9.106    ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-vdx49   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-vfx5w   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-vgvdn   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-vh22l   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-vkc59   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-vkq69   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-vnl6h   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-w4j4c   1/1     Running             0          13m   10.1.9.12     ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-w4pcb   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-w6mmm   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-w8592   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-wc28g   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-wd76d   1/1     Running             0          13m   10.1.11.117   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-whvvk   1/1     Running             0          13m   10.1.11.204   ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-wm9kh   1/1     Running             0          13m   10.1.5.242    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-wqrxt   1/1     Running             0          13m   10.1.11.145   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-wvfl5   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-x5wqh   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-x72pt   1/1     Running             0          13m   10.1.4.145    ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-x8vvn   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-xc4j9   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-xf5vl   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-xgd58   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-xq4bk   0/1     ContainerCreating   0          13m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-xvgzv   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-xvpjt   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-xztp6   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-z4xtr   0/1     ContainerCreating   0          12m   <none>        ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-zbwcm   1/1     Running             0          13m   10.1.11.34    ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-zbzlh   1/1     Running             0          13m   10.1.10.126   ip-10-1-11-1.eu-west-1.compute.internal    <none>           <none>
ahoy-hello-world-79f545d68-zc2ws   1/1     Running             0          13m   10.1.9.67     ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-zgqv4   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-zn9kg   1/1     Running             0          13m   10.1.9.54     ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-zqc4q   1/1     Running             0          13m   10.1.10.179   ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-zszsh   0/1     ContainerCreating   0          13m   <none>        ip-10-1-8-251.eu-west-1.compute.internal   <none>           <none>
ahoy-hello-world-79f545d68-ztz4r   0/1     ContainerCreating   0          13m   <none>        ip-10-1-6-216.eu-west-1.compute.internal   <none>           <none>

Pod events:
Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               15m                default-scheduler  Successfully assigned test/ahoy-hello-world-79f545d68-ztz4r to ip-10-1-6-216.eu-west-1.compute.internal
  Warning  FailedCreatePodSandBox  15m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2585244756197a04982514cc97947f5b7f702492df9776871fa3d7d3b3cba718": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  15m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e47bc00bb26d8f5751147bd3a10c74ea0d9416b87c6e9411ed9dea9c513dca58": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  14m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8af56f7579803fd048d6a8b34f6aeae43d347d0437c72bf31381d7df72089799": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  14m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "24006b0da1058621d72b7ce106ede4dc60cc89335c2291fe48680678a70ecfe8": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  14m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "02517baf7153f532104dc880e8c3183d27e24cc01612c1821f32481869b3219e": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  14m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d86eb35de4cd63ae8157201cda023d2aec89587b235c6043770eb4d41fd59369": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  14m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "119d185bd793cf479f837ce232e6ead47f7b0da2aa7670f50b1ac7e328ee259a": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  13m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a78c4acdab7165b1ea22de703d477fd63977fd1021110f1d4a0444f269312aaf": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  13m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e8f1243c54d26a1a400c78a73ab309108e16fb1032c97e71227e483ac2419e09": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container
  Warning  FailedCreatePodSandBox  1s (x62 over 12m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "204bd57a988811b5df8952be22b24dc593b5c0ed07fe17ab8152dcac12039395": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

This seems like a VPC CNI issue, it would be useful to open an issue on https://github.com/aws/amazon-vpc-cni-k8s
This troubleshooting guide for the VPC CNI may also be useful:
https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/troubleshooting.md

Thanks for having a look @bwagner5, I've opened https://github.com/aws/amazon-vpc-cni-k8s/issues/2411.

@bwagner5 please check this article https://aws.amazon.com/blogs/containers/amazon-vpc-cni-increases-pods-per-node-limits/

authors: bwagner5 igorgolm kstevensonnv

repo: aws/karpenter


text:
Clarification in behaviour when MaxSpotInstanceCountExceeded exception is reached

Description
This is to clarify one thing in behaviour.
Here, https://aws.amazon.com/blogs/containers/using-amazon-ec2-spot-instances-with-karpenter/ under Capacity type it states:
If the Spot capacity is not available, then Karpenter falls back to On-Demand Instances to schedule the pods. However, if a Spot quota limit has been reached at the account level, you might get a MaxSpotInstanceCountExceeded exception. In this case, Karpenter won’t perform a fallback.

Is that still the case ? Or the blog post info is outdated and not relevant anymore ?
I'm wondering why Karpenter cannot fallback from reaching Spot quota limit back to On-Demand (if configured to use both) automatically given On-Demand quota is not reached yet ?

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

I don't think this is correct. I just spun up enough spot nodes to reach this, added in my on-demand requirement, and saw it provision on-demand nodes. Additionally, did the same, but with on-demand and spot both enabled, and saw it fallback to on-demand after I got the MaxSpotInstanceCountExceeded. Have you seen differently?

Thanks @njtran for confirmation.
Nope, I haven't seen differently but haven't found this behaviour documented anywhere and wanted to confirm.
I guess the above blog post os outdated.

Yeah, the blog post is about a year old so we assume that some of the information will be out-of-date. I'm going to close this since it looks like the fallback behavior works.

source: gharchive/issue

authors: cbugneac-nex jonathan-innis njtran

license_type: permissive

license_source: github-api


text:
fix(tabs): Make max tabs notification dismissible and allow external configuration

Problem
Currently the notification for max tabs isn't dismissible. Also, isn't possible the implementation overwrite the defaults (duration / tooltip text).

Solution
Add maxReachedOverlayDuration (to adjust the duration / fallbacks to default 5000) to use as a dismiss duration and noMoreTabsTooltip exposable for messaging configuration.

Note: I've tried to update the changelog with newChange, but doesn't seems to have the script.
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@ege0zcan do you have writes to merge this one? If don't, I can ask someone else in the team. Thanks

source: gharchive/pull-request

repo: aws/mynah-ui


text:
2023-01-14_00-09-33_master-next_aws-iot-device-sdk-python-v2

Automatically created.

/run

/run

authors: aws-iot-embedded-linux-ci thomas-roos

license: MIT


text:
No ChangeLog For amazon-eks-node-1.12-v20190614

I can't seem to find the change log for amazon-eks-node-1.12-v20190614 released on the 15th. Is this the correct place to find information on the changes?

@kody-abe
Sorry the late reply. I'll push a change log by TOD today.

repo: awslabs/amazon-eks-ami


text:
tar.gz support

Hello, I'd like to use this tool to extract certain folders in a .tar.gz file in s3. This is what I am facing
parallels@ubuntu-linux-22-04-02-desktop:~/amazon-s3-tar-tool/bin$ ./s3tar-linux-arm64 --region us-east-1 -xvf s3://test/test1/test1.tar.gz -C s3://anotherbucket/ folder/subfolder
2024/02/23 08:59:56 unable to parse CSV TOC from TAR

Is there any workaround I can do?

Hello @vicaraya it looks like your tar is using gz compression, based on the extension (tar.gz). The tool today doesn't support compression, as that would require to download the tar.gz and uncompress it. The purpose of s3tar is to extract the data without downloading it. Regular tarballs without compression are simply a concatenation of the files, this allows us to Copy the data to another Amazon S3 location.


text:
Documentation specifies the wrong version for Maven

Describe the issue
The documentation gives the example
<dependencies>
    <dependency>
        <groupId>software.amazon.jdbc</groupId>
        <artifactId>aws-advanced-jdbc-wrapper</artifactId>
        <version>2.2.3</version>
    </dependency>
</dependencies>

but version 2.2.3 has not been published on Maven yet (as of time of writing)
Is this automatically updated? It would be better if it is updated only once published to Maven
Links
https://github.com/awslabs/aws-advanced-jdbc-wrapper/blob/main/docs/GettingStarted.md#as-a-maven-dependency

Hi @larmitage-nex, thanks for raising this issue. We've updated the documentation to reflect the latest version. We'll be closing this ticket now, but please let us know if you notice any other discrepancies.

authors: crystall-bitquill larmitage-nex

repo: awslabs/aws-advanced-jdbc-wrapper


text:
Always run channel setup callback on pinned event loop if one exists

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (64c06af) 79.81% compared to head (ad7391a) 80.09%.

Files
Patch %
Lines

source/channel_bootstrap.c
93.10%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #618      +/-   ##
==========================================
+ Coverage   79.81%   80.09%   +0.28%
==========================================
  Files          28       28
  Lines        5915     5943      +28
==========================================
+ Hits         4721     4760      +39
+ Misses       1194     1183      -11

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: bretambrose codecov-commenter

repo: awslabs/aws-c-io


text:
CodeBuild: Missing EncyptionDisabled Artifact property

Note: for support questions, please first reference our documentation, then use Stackoverflow. This repository's issues are intended for feature requests and bug reports.

I'm submitting a ...

[ ] :beetle: bug report
[ ] :rocket: feature request
[x] :books: construct library gap
[ ] :phone: security issue or vulnerability => Please see policy
[ ] :question: support request => Please see note at the top of this template.

What is the current behavior?
If the current behavior is a :beetle:bug:beetle:: Please provide the steps to reproduce
There is currently no way to disable artifact encryption.

What is the expected behavior (or behavior of feature suggested)?
Include a boolean encryptionDisabled parameter to ArtifactsProps, matching CloudFormation's EncryptionDisabled

What is the motivation / use case for changing the behavior or adding this feature?
The motivation for this property is best explained in the CodeBuild Console:

Disable encryption if using the artifact to publish a static website or sharing content with others

Please tell us about your environment:

CDK CLI Version: 0.37.0
Module Version: 0.37.0
OS: all
Language: all

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc)

Hey @nmussy ,
thanks for opening the issue! Adding this should be fairly straightforward -- it's adding an encryptionDisabled property in the S3ArtifactProps interface, and a corresponding small change in the S3Artifacts class -- plus a unit test, of course!.
Would you be interested in opening us a Pull Request with this change?
Thanks,
Adam

Hey @skinny85,
Sure, I'll look into it. I should have a PR ready by the end of the day.

repo: awslabs/aws-cdk


text:
Timezone aware timestamp logging

What were you initially searching for in the docs?
How to log the timezone in a timezone aware format ("2019-02-04T12:23:34Z" ISO 8601 ) e.g. to guarantee UTC time is always logged, even when logging in JST, or to at least indicate the time zone.
Describe how we could make it clearer
I can see how with this library we can specify a format string for the datetime, and I think datetime is hard coded to come from https://docs.python.org/3/library/time.html#time.asctime asctime() if I understand correctly, which in turn means that we are logging in time-zone unaware local time.
The documentation is unclear (if it is even possible) on how to log in a timezone aware manner.
They say here that "Python datetime objects don't have time zone info by default, and without it, Python actually violates the ISO 8601 specification (if no time zone info is given, assumed to be local time). You can use the pytz package to get some default time zones, or directly subclass tzinfo yourself"
https://stackoverflow.com/questions/19654578/python-utc-datetime-objects-iso-format-doesnt-include-z-zulu-or-zero-offset
So if I wanted to say log in strict ISO 8601 specification, or ensure my logs are always UTC no matter where the code is run, I am unclear how to do this?

Hey @bml1g12 thanks for raising this.
It's spot on Standard lib comment and your research. It can be a new feature for Logger to have a flag to convert to UTC or a given offset.
On an extended weekend now but you can take a peek at the formatter.py as that's where it should go.
Would you mind creating a feature request with a proposed UX for this?
We'd be happy to implement it

Got my laptop and managed to test this to unblock you for now. You can either change the time converter after creating a Logger or monkeypatch Powertools Logging Formatter - I personally find the former cleaner and less error prone, so I'll leave at your discretion.
Setting GMT time converter at Logger instance
from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging.formatter import JsonFormatter

import time

logger = Logger(service="pt_issue_379")

logger.info("Local time")
logger._logger.handlers[0].formatter.converter = time.gmtime  # as per official Python docs
logger.info("GMT time")

Setting GMT time via monkeypatching
from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging.formatter import JsonFormatter

import time

JsonFormatter.converter = time.gmtime  # monkeypatch `converter` Formatter method before creating a logger instance

logger = Logger(service="pt_issue_379")

logger.info("Local time")
logger.info("GMT time")

Results
{"level": "INFO", "location": "<module>:10", "message": "Local time", "timestamp": "2021-04-03 10:49:26,302", "service": "pt_issue_379", "sampling_rate": 0.0}

{"level": "INFO", "location": "<module>:14", "message": "GMT time", "timestamp": "2021-04-03 08:49:26,303", "service": "pt_issue_379", "sampling_rate": 0.0}

That sounds perfect! Thanks for the swift solution.
As you suggested, I'll open up a feature request for a flag for this; given I imagine it would be a common desired use case.

repo: awslabs/aws-lambda-powertools-python


text:
[BUG] - Policy limit exceeded

Currently occurring in the nightly deploy env
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     deploy_env(env_name=env_name, manifest_dir=manifest_dir)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_codeseeder/codeseeder.py", line 199, in wrapper
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     return func(*args, **kwargs)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_orbit/remote_files/deploy.py", line 371, in deploy_env
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     changeset=changeset,
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_orbit/remote_files/eksctl.py", line 494, in deploy_env
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     f"{context.eks_oidc_provider}:sub": f"system:serviceaccount:orbit-system:orbit-{context.name}-admin"
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/aws_orbit/services/iam.py", line 110, in add_assume_role_statement
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     iam_client.update_assume_role_policy(RoleName=role_name, PolicyDocument=policy_body)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/botocore/client.py", line 388, in _api_call
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     return self._make_api_call(operation_name, kwargs)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]   File "/root/.venv/lib/python3.7/site-packages/botocore/client.py", line 708, in _make_api_call
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD]     raise error_class(parsed_response, operation_name)
[2021-12-28 03:40:42,188][_remote.py   : 30] [CODEBUILD] botocore.errorfactory.LimitExceededException: An error occurred (LimitExceeded) when calling the UpdateAssumeRolePolicy operation: Cannot exceed quota for ACLSizePerRole: 2048```

When I look at this role (orbit-nightly-us-west-2-admin) I see that the trust policy has 4 entries for OIDC to have Assume RoleWithWebIdentity.  I believe there should only be one....for the active cluster.

Looking at the code, the command orbit toolkit creates the IAM admin role.  Once created, the orbit deploy env updates the role's trust policy with OIDC of the cluster.
When we delete the env (the cluster) and create a new one, the trust policy is updated with the new cluster OIDC, but the references to the deleted cluster are NOT removed from the policy.
: recycling the env multiple times fills up the space in the trust policy

repo: awslabs/aws-orbit-workbench


text:
ELB Health Check Issue

When I run the Master Template, it fails and does a rollback at the point of the Web Template. As such,  I ran the Templates individually successfully until I run the Web Template where it fails and does a rollback (of just the web template).
The issue is new instances fail the Health Check, and as a result, the ASG launches another EC2 instance, another fails, it launches a new instance, etc... hence, causing the Web Template to fail and rollback.
As a result, I changed the Web Template from "HealthCheckType: ELB" to "HealthCheckType: EC2" and the Web Template runs successfully to completion and the ASG does not continually launch instances. As such, it appears the Heath Check is not visible from the ELB as it should be.
Aside from the above modification, the only other modification I made to the Templates was changing the subnet IP ranges from 10.0.x.x/xx to 10.10.x.x/xx. in the VPC Template.
Is there some other configuration I need to make as a result of adjusting the CIDR ranges? or are there any other configuration I need to make in the AWS Console for these Templates to execute in totality?
Would appreciate any/all guidance and thoughts re how to get the Health Check from the ELB working.
Thank you, Mike

Since the Web Template only completed with "HealthCheckType: EC2" I decided to test/change the ASG configuration in the console to "HealthCheckType: ELB"... Interesting, I have no issues now--I can adjust the ASG instance settings up/down and instances are launching/terminating as expected.

I'll take a look at this over the next few days and see if a change to resolve this will be pushing during my next push in the next few days.

Darryl, Know I'm currently working with AWS Tech Support as it appears the issue I was having with "HealthCheckType: ELB" are timeout issues due to latency within the site. Not sure why, yet, but the site is slow and that was causing HealthChecks to timeout/fail. If the root cause is something related to the CloudFormation Template (versus our wordpress site) I'll post it here.

Michael. Have you been able to resolve your ELB healthcheck issue?

Darryl,
Not yet... we have AWS techs reviewing ALB log files and Web Server packet captures as we speak--should have an answer soon.
-Mike

Darryl,
FYI, AWS Techs informed me this morning they are going to now involve a CloudFormation engineer in my case--current thinking is still high latency is causing Health Checks to Fail, and they are trying to determine root cause.
Has anyone reported to you any latency issues with this setup that I could bring to the attention of the techs working the case? Particularly, once a site is migrated over or a site is heavily built out?
-Mike

Mike,
The only issue we're aware of is related to EFS file systems running out of burst credits which drops the permitted throughput to the baseline throughput of 50 MiB/s per TiB of storage (or 50 KiB/s per GiB). If the WordPress site has a small amount of file system storage on EFS, it doesn't earn enough credits to sustain the throughput it needs and eventually the burst credit balance drops to zero. File systems earn credits at the baseline throughput rate and consume credits at the throughput driven by the site. If you're using more than your earning, eventually it will drop to zero. Performance may be great initially but when the burst credit balance drops to zero and the permitted throughput changes to the baseline throughput, the site experiences performance issues when the file system is unable to achieve the necessary throughput it demands. That's one of the reasons I built the burst credit balance alarms and dashboard, so customers can monitor and get notified on this condition.

authors: darrylsosborne michael-newman

repo: awslabs/aws-refarch-wordpress


text:
[WIP/Discussion] Add hot reload for python runtimes

Issue #, if available:
n/a
Description of changes:
Adds a "hot loader" for local development when using python runtimes.
Since python runtimes require a cp src_dir/* src_dir/build/ on each change (due to the flat dir constraint of the lambda framework), I initially made a wrapper around sam local start-api to watch and copy src files to the build dir on changes.
I started integrating the loader into the cli to make the wrapper unnecessary. If there's an interest in this feature,  I'll keep working to productionize this. Otherwise, I don't want to waste the cycles.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mijdavis2  This feature is super cool and incredibly useful for Python devs
I think we can generalize this further into a separate sam build command and add a --watch flag to watch files and run build actions. You could have sam build --watch running in a separate terminal that continually rebuilds in background while you use sam local start-api in another terminal to test the function. I think this will be more powerful. Thoughts?
This PR makes three assumptions:

Users have a build/ directory where Lambda function code files reside
Py users want to always hot reload
The only "build" action for Py users is copying files to the 'build/' directory.

By providing a separate sam build command, we can let user customize these assumptions. In this command, we can provide default implementation for all Lambda runtimes and allow users override default implementation by providing a makefile or bash script or an executable. This extension mechanism could also be CodeBuild Local (#402).
What do you think about this? If you like it, I am happy to collaborate with you on an RFC for the sam build command so we can get a clear set of requirements before building it out.

Similar in spirit to #195

There hasn't been any activity here in a while, so going to close the PR. Happy to reopen in the future though.

For those who get here from google search.
Here is alternative solution to enable hot reload on your python sam app:
Use watchmedo to watch any changes in your function.
For example:
watchmedo shell-command -p ".\Lambda*\*.py" -i ".\Lambda*\*.py.*" -R -W --command='sam build' .

This will rebuild your apps if there is any changes.
Then on another terminal tab, start the server:
sam local start-api

authors: jfuss mijdavis2 pyk sanathkr

repo: awslabs/aws-sam-cli


text:
Removes all instances of ARPA

Signed-off-by: Felipe R. Monteiro felisous@amazon.com
Issue #, if available:
N/A.
Description of changes:
Removes all instances of ARPA.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Could we update the run instructions in the readme?
Follow these tool installation instructions to install cbmc and cbmc-viewer.
Run "make"
Open "report/html/index.html"

Could we update the run instructions in the readme?
Follow these tool installation instructions to install cbmc and cbmc-viewer.
Run "make"
Open "report/html/index.html"

@markrtuttle done.

authors: feliperodri markrtuttle

repo: awslabs/aws-templates-for-cbmc-proofs


text:
Leading Whitespace in GetAtt syntax of !Sub not ignored

cfn-lint version: cfn-lint 0.4.1
Description of issue.
When using the ${} syntax in a sub it doesn't ignore whitespace and throws an error, e.g.
"Resource": {
    "Fn::Sub": "${ AppendNotificationWatchLambdaLogGroup.Arn}*"
}

Will throw the following error:
E1019 String parameter  AppendNotificationWatchLambdaLogGroup.Arn not found in string for Resources/AppendNotificationWatchLambdaRolePolicy/Properties/PolicyDocument/Statement/0/Resource/Fn::Sub

When the whitespace is removed this works perfectly fine in cfn-lint (and always works in CloudFormation).

Good point.  I will work on a fix for this.  Thank you for the issue should have a fix shortly.

Great! Let me know if there is anything I can help with. At a quick glance I wasn't sure where to implement this in the code, so didn't send a PR. If you let me know which part this is implemented in I'm happy to send a PR

I think I got it and made a few improvements along the way.  I think the same issue exists for spaces at the end of the parameter so we do a complete strip (left and right side empty spaces removed).

authors: flomotlik kddejong

repo: awslabs/cfn-python-lint


text:
Support CloudFormation Templates

Most, if not all RESTful APIs require some sort of data source.
I'm not sure if this is out of scope for this project, hopefully not, but having the ability to deploy CloudFormation templates to buildout a DynamoDB table, or RDS instance etc would be beneficial.
It also makes cleanup, removing API Gateways, Lambda functions and roles much easier.

I agree that most APIs require a data source.
This is something I've thought about, but I'm inclined to say that it's out of scope, at least initially.
One of things I want to make sure this project doesn't lose is the minimalistic nature of the framework.  Similar to how django has manage.py migrate, but in flask you have to manage this yourself, I think a similar parallel can be drawn here.

I've drafted a proposal of integrating with cloudformation/SAM: https://github.com/awslabs/chalice/issues/183
Let me know if that's similar to what you had in mind.

authors: JamieCressey jamesls

repo: awslabs/chalice

license_source: bigquery