text:
CRAYSAT-1839-sat-bootsys-waiter-subclass-addition

Updating the sat bootsys kubelet start function to address chronjob creation failure
IM:CRAYSAT-1839
Reviewer:Ryan
Summary and Scope
modified the do_kubelet_start function to add the wait after kubelet gets started.
using the Waiter class that is already used in this module. Created a new Waiter subclass that implements the has_completed method to query the Kubernetes API to get the nodes in the Cluster.
Once the query returns successfully, would be proceeding with the next step.
Issues and Related PRs
List and characterize relationship to Jira/Github issues and other pull requests. Be sure to list dependencies.

Resolves CRAYSAT-1839

Testing
Yet to be tested .
Tested on:
YTT
Test description:
will power off the nodes and power on the node to validate the functionality inclusion during the power on.
Risks and Mitigations
Minimal
Pull Request Checklist

[x] Version number(s) incremented, if applicable
[x] Copyrights updated
[x] License file intact
[x] Target branch correct
[x] CHANGELOG.md updated
[ ] Testing is appropriate and complete, if applicable
[ ] HPC Product Announcement prepared, if applicable

Test output:
ncn-m001:/mnt/shiva # sat bootsys boot --stage platform-services --ceph-timeout 120
The following Non-compute Nodes (NCNs) will be included in this operation:
managers:

ncn-m001
ncn-m002
ncn-m003
storage:
ncn-s001
ncn-s002
ncn-s003
ncn-s004
workers:
ncn-w001
ncn-w002
ncn-w003
ncn-w004

Are the above NCN groupings correct? [yes,no] yes
INFO: Executing step: Ensure containerd is running and enabled on all Kubernetes NCNs.
INFO: Executing step: Ensure etcd is running and enabled on all Kubernetes manager NCNs.
INFO: Executing step: Start and enable kubelet on all Kubernetes NCNs.
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c987f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9ca7f70>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c87790>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8a9a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8abe0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9ca6a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d250>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8eeb0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8ee20>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87adc0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87ad00>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87a0a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c9f040>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d400>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d190>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99c40>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99a90>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99fd0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881be0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881220>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885250>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881130>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881d30>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881820>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99ac0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99c40>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aabe8850>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d4c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c9f6a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9cad1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8853d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885280>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885a00>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87deb0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d6a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99fa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99fd0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aabe44c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885ca0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885610>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8857c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aaa731c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c767f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881af0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3ad1a3bb0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99f70>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c99fd0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d4c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d8e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885640>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aaa73370>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885670>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8859a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885790>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d400>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87df10>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8a9a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3ad1a3b50>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aabe44f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881820>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aaa73490>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aaa73400>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aaa73910>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d2b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87deb0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885ca0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885f10>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885e50>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885ee0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87aca0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87aaf0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c9fb80>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98490>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c988e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c9f6a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885a60>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885340>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8856a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885b80>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885640>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885820>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d3a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881820>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c982b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9cae880>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98940>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c982b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aab4fe80>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8854c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885520>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8852b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8853a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885df0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87ab50>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c9fbb0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c9f6a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885970>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8852b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885f10>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d3a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d3d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d190>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98070>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98940>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9caefa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d430>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d550>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9cae580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881820>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881b80>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885220>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885790>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885640>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885be0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d730>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8858b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885730>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885b50>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885910>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aab4f760>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9ca1a00>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9ca6dc0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98070>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d9d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d820>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8df40>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881070>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d430>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d400>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d8b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885490>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8855b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885df0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885fa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885610>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8aee0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aab4fe80>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885bb0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885b80>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8857c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8854c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8811c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881820>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c983d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98040>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8af70>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98040>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98940>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c8d8b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881d60>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d4c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d1f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa8854c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885940>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885a30>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c87790>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c879d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c87cd0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885b50>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa885d00>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa87d3a0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3aa881190>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9ca1fa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9ca14f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c98040>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c87580>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c87a00>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c995b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c87f40>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe3a9c87d30>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/nodes
WARNING:root:MaxRetryError occurred while trying to connect to Kubernetes API.
ERROR:/sat/venv/lib/python3.9/site-packages/sat/waiting.py:Waiting for condition "Kubernetes API availability check" timed out after 300 seconds
INFO: Executing step: Recreate cron jobs that have become stuck
INFO:sat.cli.bootsys.platform:Executing step: Recreate cron jobs that have become stuck
DEBUG:csm_api_client.k8s:Couldn't load the in-cluster config: Service host/port is not set. (proceeding under the assumption that the config should be loaded from the kubeconfig file)
the warning seems to be lot many, which may lead to confusion, even though its on the expected lines

Updated output, post addressing the comments:
ncn-m001:/mnt/shiva # sat bootsys boot --stage platform-services --ceph-timeout 120
The following Non-compute Nodes (NCNs) will be included in this operation:
managers:

ncn-m001
ncn-m002
ncn-m003
storage:
ncn-s001
ncn-s002
ncn-s003
ncn-s004
workers:
ncn-w001
ncn-w002
ncn-w003
ncn-w004

Are the above NCN groupings correct? [yes,no] yes
INFO: Executing step: Ensure containerd is running and enabled on all Kubernetes NCNs.
INFO: Executing step: Ensure etcd is running and enabled on all Kubernetes manager NCNs.
INFO: Executing step: Start and enable kubelet on all Kubernetes NCNs.
INFO: Waiting for Kubernetes API to become reachable...
Waiting for condition "Kubernetes API available" timed out after 300 seconds
INFO: Executing step: Recreate cron jobs that have become stuck
WARNING: Jobs for cronjob "cray-dns-unbound-manager" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: Jobs for cronjob "hms-discovery" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
ncn-m001:/mnt/shiva # systemctl is-enabled kubelet
enabled
ncn-m001:/mnt/shiva # systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
/etc/systemd/system/kubelet.service.d
└─10-kubelet.conf
Active: active (running) since Thu 2024-04-25 06:46:32 UTC; 20min ago
Docs: https://kubernetes.io/docs/
Main PID: 2977924 (kubelet)
Tasks: 79
CGroup: /system.slice/kubelet.service
└─ 2977924 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kube>
Apr 25 06:52:37 ncn-m001 kubelet[2977924]: E0425 06:52:37.079689 2977924 pod_workers.go:951] "Error syncing pod, skipping" er>
Apr 25 06:52:51 ncn-m001 kubelet[2977924]: E0425 06:52:51.079839 2977924 pod_workers.go:951] "Error syncing pod, skipping" er>
Apr 25 06:53:04 ncn-m001 kubelet[2977924]: E0425 06:53:04.080191 2977924 pod_workers.go:951] "Error syncing pod, skipping" er>
Apr 25 06:53:17 ncn-m001 kubelet[2977924]: E0425 06:53:17.079126 2977924 pod_workers.go:951] "Error syncing pod, skipping" er>
Apr 25 06:53:45 ncn-m001 kubelet[2977924]: I0425 06:53:45.778928 2977924 scope.go:110] "RemoveContainer" containerID="f58d126>
Apr 25 06:53:45 ncn-m001 kubelet[2977924]: I0425 06:53:45.779894 2977924 scope.go:110] "RemoveContainer" containerID="f58d126>
Apr 25 06:53:45 ncn-m001 kubelet[2977924]: E0425 06:53:45.781904 2977924 remote_runtime.go:296] "RemoveContainer from runtime>
Apr 25 06:53:45 ncn-m001 kubelet[2977924]: E0425 06:53:45.781995 2977924 kuberuntime_container.go:798] failed to remove pod i>
Apr 25 06:53:47 ncn-m001 kubelet[2977924]: I0425 06:53:47.796994 2977924 scope.go:110] "RemoveContainer" containerID="07bfd90>
Apr 25 06:53:47 ncn-m001 kubelet[2977924]: I0425 06:53:47.797035 2977924 scope.go:110] "RemoveContainer" containerID="a90ee4d>
ncn-m001:/mnt/shiva # kubectl cluster-info
Kubernetes control plane is running at https://10.252.1.2:6442
CoreDNS is running at https://10.252.1.2:6442/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

@shivaprasad-metimath, from your testing, I can see that this code does not behave correctly. Specifically, see these lines:
INFO: Waiting for Kubernetes API to become reachable...
Waiting for condition "Kubernetes API available" timed out after 300 seconds
INFO: Executing step: Recreate cron jobs that have become stuck
WARNING: Jobs for cronjob "cray-dns-unbound-manager" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: Jobs for cronjob "hms-discovery" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.

This shows that it timed out waiting for the Kubernetes API to become available. This means your has_completed method in your new KubernetesAPIWaiter class must never be returning True and allowing the waiter to complete. I looked into it, and I noticed that you are never loading the Kubernetes configuration, so it's not able to connect to the Kubernetes cluster. To fix this, please use the function load_kube_api from csm_api_client.k8s. This will properly ensure the Kubernetes configuration is loaded and then instantiate a CoreV1Api and return it to you.
Note that when I tested using load_kube_api in a sat bash shell, I found another bug with calling list_node on the resulting CoreV1Api object. I addressed that in my pull request for CRAYSAT-1848 here: https://github.com/Cray-HPE/sat/pull/216 Please take a look at that. Once that's fixed in the main branch, you'll want to merge main into your feature/CRAYSAT-1740 branch and then rebase your CRAYSAT-1839-bootsys-waiter-subclass-addition branch on the feature branch to pull in the fix before you test again.

While I was looking at other usages of the {{wait_for_completion}} method of various {{Waiters}}, I noticed that there is actually already a KubernetesAPIAvailableWatier in sat.cli.bootsys.k8s. It uses get_api_resources instead of list_node, and that's fine. All we're really trying to do is query the API, and that's probably the simplest API call you could make. As an added bonus, it doesn't look like get_api_resources has the same problem with the Kubernetes version mismatch as described in CRAYSAT-1848.
The only thing missing from that class that you have in yours is an info log message the first time the waiter finds the Kubernetes API to be unreachable. I think it's fine to add such a log message. Please modify and use the existing KubernetesAPIAvailableWaiter instead of implementing another one.

Latest output:
ncn-m001:/mnt/shiva # sat bootsys boot --stage platform-services --ceph-timeout 120
The following Non-compute Nodes (NCNs) will be included in this operation:
managers:

ncn-m001
ncn-m002
ncn-m003
storage:
ncn-s001
ncn-s002
ncn-s003
ncn-s004
workers:
ncn-w001
ncn-w002
ncn-w003
ncn-w004

Are the above NCN groupings correct? [yes,no] yes
INFO: Executing step: Ensure containerd is running and enabled on all Kubernetes NCNs.
INFO: Executing step: Ensure etcd is running and enabled on all Kubernetes manager NCNs.
INFO: Executing step: Start and enable kubelet on all Kubernetes NCNs.
INFO: The Kubernetes API is currently unreachable.
INFO: Executing step: Recreate cron jobs that have become stuck
WARNING: Jobs for cronjob "cray-bos-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-bos-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '57bb6e09-ec2b-4dcb-a2bf-86e0a855d65b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:15 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-bss-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-bss-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'be931aa2-c6ab-4f5d-9325-23c0d231e2c4', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:20 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-dns-unbound-manager" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-dns-unbound-manager" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '8c4d8af1-911d-4b90-a9ea-23881c3bef50', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:23 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-fas-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-fas-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '52dd4fe9-1419-4733-9527-0c6349b25ea5', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:26 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-hbtd-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-hbtd-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '61bc1a16-c0da-43b8-9386-4980c4f14642', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:29 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-hmnfd-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-hmnfd-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '2784415f-090b-4c01-acd9-89eed145a24b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:29 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-power-control-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-power-control-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '6842288e-8752-4b75-ae21-902cbf0fb2f7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:38 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-uas-mgr-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-uas-mgr-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '47650866-831a-4e66-bd36-6473a3f5f6cd', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:41 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "etcd-backup-pvc-snapshots-to-s3" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "etcd-backup-pvc-snapshots-to-s3" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '030149ce-48bf-4e2e-9103-84eede7bc86f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:44 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "hms-discovery" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "hms-discovery" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '60fd74fb-e410-4c41-8fd8-46dc7748b734', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:47 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "sonar-sync" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "sonar-sync" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ffd241fa-b628-4624-9d83-72816c5da21d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '4ea23580-d341-47ea-a184-d76c9126b813', 'X-Kubernetes-Pf-Prioritylevel-Uid': '03d335b2-1a9f-4bed-979b-b187c7e83a43', 'Date': 'Tue, 30 Apr 2024 07:26:47 GMT', 'Content-Length': '545'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.17.72.31:443: connect: no route to host"}]},"code":500}
WARNING:
ncn-m001:/mnt/shiva #

Waiting for condition "Kubernetes API available" timed out after 300 seconds
ERROR: Fatal error in step "Start and enable kubelet on all Kubernetes NCNs." of platform services start: Failed to start kubelet: Kubernetes API not available

Why is this error popped up even after I see kubelet is active and enabled

Kubernetes api availability check seems to be working fine with expected output, during the chron job creation there is a delay observed leading to warnings. It could be observed only on this system.
Complete log has been attached for reference.
sat1.log
1st attempt:
ncn-m001:/mnt/shiva # sat bootsys boot --stage platform-services --ceph-timeout 120
The following Non-compute Nodes (NCNs) will be included in this operation:
managers:

ncn-m001
ncn-m002
ncn-m003
storage:
ncn-s001
ncn-s002
ncn-s003
ncn-s004
workers:
ncn-w001
ncn-w002
ncn-w003
ncn-w004

Are the above NCN groupings correct? [yes,no] yes
INFO: Executing step: Ensure containerd is running and enabled on all Kubernetes NCNs.
INFO: Executing step: Ensure etcd is running and enabled on all Kubernetes manager NCNs.
INFO: Executing step: Start and enable kubelet on all Kubernetes NCNs.
INFO: Waiting up to 300 seconds for the Kubernetes API to become available
INFO: The Kubernetes API is currently unreachable.
INFO: Kubernetes API is available
INFO: Executing step: Recreate cron jobs that have become stuck
WARNING: Jobs for cronjob "cray-bos-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-bos-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'cc070e0a-fce8-45a1-8902-7e5d4ab8ff49', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:29 GMT', 'Content-Length': '493'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": context deadline exceeded","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": context deadline exceeded"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-bss-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-bss-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '1065b4ab-5747-4449-9ae6-cd000f659d59', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:32 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-dns-unbound-manager" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-dns-unbound-manager" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a383ca98-5a4a-4bb0-962e-3ad778cac7a4', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:35 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-fas-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-fas-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'df5dc7f7-9a90-4f96-8de1-7daaf829166c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:38 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-hbtd-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-hbtd-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '6eceeac2-152a-4ace-8c9b-d436c8f60c4f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:41 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-hmnfd-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-hmnfd-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'da31639f-0f55-4b1e-8729-dc21aa9c4298', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:42 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-power-control-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-power-control-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '0a6b87f1-ed1b-461a-bb94-f535e0ec7738', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:45 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "cray-uas-mgr-bitnami-etcd-snapshotter" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "cray-uas-mgr-bitnami-etcd-snapshotter" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c06ba9bd-cbbe-4e12-b24f-6be1ee537104', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:48 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "etcd-backup-pvc-snapshots-to-s3" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "etcd-backup-pvc-snapshots-to-s3" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ef533cf7-2517-4304-bc21-7c78023f270c', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:51 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "hms-discovery" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "hms-discovery" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '1785c39e-98ed-4a44-88ae-d435d046ca78', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:54 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
WARNING: Jobs for cronjob "sonar-sync" in namespace "services" do not appear to be scheduled on time according to the cron job's schedule; recreating cron job.
WARNING: An error occurred while re-creating cronjob "sonar-sync" in namespace "services": (500)
WARNING: Reason: Internal Server Error
WARNING: HTTP response headers: HTTPHeaderDict({'Audit-Id': '4cd734ff-ba4c-493c-bb7b-b6de10682b69', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '206250a1-21b4-4b9d-9608-833e6a1583cd', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b5beaacd-765d-46f9-988c-cf2c75636da4', 'Date': 'Wed, 15 May 2024 10:23:57 GMT', 'Content-Length': '549'})
WARNING: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook "validate.kyverno.svc-fail": Post "https://cray-kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s\": dial tcp 10.23.152.167:443: connect: no route to host"}]},"code":500}
WARNING:
2nd attempt:
ncn-m001:/mnt/shiva # sat bootsys boot --stage platform-services --ceph-timeout 120
The following Non-compute Nodes (NCNs) will be included in this operation:
managers:

ncn-m001
ncn-m002
ncn-m003
storage:
ncn-s001
ncn-s002
ncn-s003
ncn-s004
workers:
ncn-w001
ncn-w002
ncn-w003
ncn-w004

Are the above NCN groupings correct? [yes,no] yes
INFO: Executing step: Ensure containerd is running and enabled on all Kubernetes NCNs.
INFO: Executing step: Ensure etcd is running and enabled on all Kubernetes manager NCNs.
INFO: Executing step: Start and enable kubelet on all Kubernetes NCNs.
INFO: Waiting up to 300 seconds for the Kubernetes API to become available
INFO: Kubernetes API is available
INFO: Executing step: Recreate cron jobs that have become stuck

repo: Cray-HPE/sat


text:
Translations update from Hosted Weblate

Translations update from Hosted Weblate for Metadata Remover/Metadata.
It also includes following components:

Metadata Remover/Metadata Remover

Current translation status:

Thank you very much for your support! 😘

source: gharchive/pull-request

authors: CrazyMarvin weblate

repo: Crazy-Marvin/MetadataRemover

license_type: permissive

license_source: github-api


text:
🛑 sneak.page is down

In 4b6a6a9, sneak.page (https://sneak.page) was down:

HTTP code: 0
Response time: 0 ms

Resolved: sneak.page is back up in 4cc7895 after 8 minutes.

source: gharchive/issue

repo: Crazy-Marvin/upptime

license: MIT


text:
🛑 sneak.page is down

In 52a693a, sneak.page (https://sneak.page) was down:

HTTP code: 0
Response time: 0 ms

Resolved: sneak.page is back up in 83127fe after 11 minutes.