text:
Rancher 2.2 rc2 - prometheus metrics unavailable

What kind of request is this (question/bug/enhancement/feature request):
bug
Steps to reproduce (least amount of steps as possible):
Enable cluster level monitoring with default option
Click on the etcd metrics of the cluster
Result:
you get a popup with:
query series failed, query range failed, Get http://prometheus-operated.cattle-prometheus:9090/api/v1/query_range?end=2019-02-27T08%3A47%3A46.442214178Z&query=sum%28rate%28node_network_transmit_packets_total%7Bdevice%21~%22lo%7Cveth.%2A%7Cdocker.%2A%7Cflannel.%2A%7Ccali.%2A%7Ccbr.%2A%22%7D%5B5m%5D%29%29+by+%28instance%29&start=2019-02-27T07%3A47%3A46.442214178Z&step=60.000: dial tcp 10.42.5.6:9090: i/o timeout, expression: sum(rate(node_network_transmit_packets_total{device!~"lo|veth.*|docker.*|flannel.*|cali.*|cbr.*"}[5m])) by (instance)

Pod: prometheus-cluster-monitoring-nginx
Container name : prometheus-proxy
Container image : rancher/nginx:1.15.8-alpine
The container is started (green state) but contains the following error message :
/var/run/nginx-start.sh: line 7: ![: not found
2019/02/27 08:40:24 [warn] 11#11: duplicate MIME type "text/html" in /var/run/nginx.conf:50
nginx: [warn] duplicate MIME type "text/html" in /var/run/nginx.conf:50

Pod: prometheus-cluster-monitoring-nginx
Container name : prometheus-agent
Container image : rancher/prometheus-auth:v0.2.0
The container is started (green state) but contains the following error message :
INFO[2019-02-27T08:40:24Z] Start listening for connections on 10.42.5.6:9090

2019/02/27 08:40:37 http: proxy error: context canceled

2019/02/27 08:40:41 http: proxy error: context canceled

2019/02/27 08:40:47 http: proxy error: context canceled

2019/02/27 08:40:51 http: proxy error: context canceled

ERRO[2019-02-27T08:40:54Z] github.com/rancher/prometheus-auth/pkg/kube/namespaces.go:151: Failed to list *v1.Secret: Get https://10.43.0.1:443/api/v1/secrets?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: i/o timeout

ERRO[2019-02-27T08:40:54Z] github.com/rancher/prometheus-auth/pkg/kube/namespaces.go:152: Failed to list *v1.Namespace: Get https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: i/o timeout

2019/02/27 08:40:57 http: proxy error: context canceled

2019/02/27 08:41:01 http: proxy error: context canceled

Other details that may be helpful:
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.2.0-alpha6
Installation option (single install/HA): single install
No persistent storage

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Hosted

Machine type (cloud/VM/metal) and specifications (CPU/memory): VMware : 3 masters, 3 nodes, 2CPU/4GB/20GB HDD each

Kubernetes version (use kubectl version): 1.13.1-Rancher

Centos version : CentOS Linux release 7.6.1810 (Core)

Kernel version :

uname -r
3.10.0-957.5.1.el7.x86_64

Docker version (use docker version):

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        6247962
 Built:             Sun Feb 10 04:13:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:47:25 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Thanks for your help

@jbroq, Can you retry with rancher/rancher:2.2-rc3 ? On the other hand, enabling Monitoring and have been through for a long time, if you still get the below tip on the main dashboard, you should validate the Kubernetes DNS service work or not.

query series failed, query range failed, Get http://prometheus-operated.cattle-prometheus:9090/api/v1/query_range

@thxCode , thanks for your reply.
I am still having the issue with 2.2.0-rc6.
After many tests, my only clue is the prometheus-agent error.
> kubectl get endpoints -n cattle-prometheus
NAME                            ENDPOINTS                                                                  AGE
access-grafana                  10.42.5.3:80                                                               105m
access-prometheus               10.42.3.4:80                                                               105m
expose-grafana-metrics          10.42.5.3:3000                                                             105m
expose-kube-cm-metrics          MASTER1_IP:10252,MASTER2_IP:10252,MASTER3_IP:10252                105m
expose-kube-etcd-metrics        MASTER1_IP:2379,MASTER2_IP:2379,MASTER3_IP:2379                   105m
expose-kube-scheduler-metrics   MASTER1_IP:10251,MASTER2_IP:10251,MASTER3_IP:10251                105m
expose-kubelets-metrics         MASTER1_IP:10255,MASTER2_IP:10255,MASTER3_IP:10255 + 15 more...   102m
expose-kubernetes-metrics       10.42.3.3:8081,10.42.3.3:8080                                              105m
expose-node-metrics             MASTER1_IP:9796,MASTER2_IP:9796,MASTER3_IP:9796 + 3 more...       105m
expose-operator-metrics         10.42.5.2:8080                                                             105m
expose-prometheus-metrics       10.42.3.4:9090                                                             105m
prometheus-operated             10.42.3.4:9090                                                             105m

The pod prometheus-operated is accessible for the worker node on which it is deployed :
+1:  Out 17:20:17.278650 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 188: WORKER_IP.57894 > 10.42.3.4.websm: Flags [P.], seq 1:121, ack 1, win 229, options [nop,nop,TS val 17214350 ecr 17214350], length 120
E....   @.@.[?..!-
*...&#.Z.[................
........GET /-/healthy HTTP/1.1
Host: 10.42.3.4:9090
User-Agent: kube-probe/1.13
Accept-Encoding: gzip
Connection: close

................
17:20:17.278662  In fe:25:77:b9:12:f0 ethertype IPv4 (0x0800), length 68: 10.42.3.4.websm > WORKER_IP.57894: Flags [.], ack 121, win 227, options [nop,nop,TS val 17214350 ecr 17214350], length 0
E..418@.@...
*....!-#..&....Z.\......).....
........................
17:20:17.279318  In fe:25:77:b9:12:f0 ethertype IPv4 (0x0800), length 227: 10.42.3.4.websm > WORKER_IP.57894: Flags [P.], seq 1:160, ack 121, win 227, options [nop,nop,TS val 17214351 ecr 17214350], length 159
E...19@.@...
*....!-#..&....Z.\............
........HTTP/1.1 200 OK
Content-Length: 23
Content-Type: text/plain; charset=utf-8
Date: Mon, 18 Mar 2019 16:20:17 GMT
Connection: close

Prometheus is Healthy.

DNS seems to work.
That is confirmed by TCPDUMP :
17:57:25.781217 IP 10.42.4.3.52746 > 10.43.0.10.53: 8954+ AAAA? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:25.781255 IP 10.42.4.3.52746 > 10.42.3.2.53: 8954+ AAAA? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:25.781371 IP 10.42.4.3.53976 > 10.43.0.10.53: 311+ A? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:25.781386 IP 10.42.4.3.53976 > 10.42.3.2.53: 311+ A? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:30.781358 IP 10.42.4.3.42390 > 10.43.0.10.53: 62201+ AAAA? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:30.781391 IP 10.42.4.3.42390 > 10.42.3.2.53: 62201+ AAAA? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:30.781572 IP 10.42.4.3.41967 > 10.43.0.10.53: 31825+ A? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:30.781595 IP 10.42.4.3.41967 > 10.42.4.2.53: 31825+ A? prometheus-operated.cattle-prometheus.cattle-system.svc.cluster.local. (87)
17:57:30.781922 IP 10.42.4.2.53 > 10.42.4.3.41967: 31825 NXDomain 0/1/0 (180)
17:57:30.781935 IP 10.43.0.10.53 > 10.42.4.3.41967: 31825 NXDomain 0/1/0 (180)
17:57:35.781524 IP 10.42.4.3.36032 > 10.43.0.10.53: 37510+ AAAA? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:35.781564 IP 10.42.4.3.36032 > 10.42.4.2.53: 37510+ AAAA? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:35.781639 IP 10.42.4.3.55573 > 10.43.0.10.53: 23069+ A? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:35.781662 IP 10.42.4.3.55573 > 10.42.3.2.53: 23069+ A? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:35.781758 IP 10.42.4.2.53 > 10.42.4.3.36032: 37510 0/0/0 (73)
17:57:35.781773 IP 10.43.0.10.53 > 10.42.4.3.36032: 37510 0/0/0 (73)
17:57:35.781891 IP 10.42.4.3.55726 > 10.43.0.10.53: 21227+ AAAA? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:35.781912 IP 10.42.4.3.55726 > 10.42.3.2.53: 21227+ AAAA? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:40.782390 IP 10.42.4.3.48471 > 10.43.0.10.53: 63351+ A? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:40.782429 IP 10.42.4.3.48471 > 10.42.4.2.53: 63351+ A? prometheus-operated.cattle-prometheus.svc.cluster.local. (73)
17:57:40.782824 IP 10.42.4.2.53 > 10.42.4.3.48471: 63351* 1/0/0 A 10.42.3.4 (89)
17:57:40.782839 IP 10.43.0.10.53 > 10.42.4.3.48471: 63351* 1/0/0 A 10.42.3.4 (89)

What I see, is that the prometheus-agent is not answering to requests
18:02:23.982458 IP 10.42.4.3.45860 > 10.42.3.4.9090: Flags [S], seq 2417066922, win 29200, options [mss 1460,sackOK,TS val 19741050 ecr 0,nop,wscale 7], length 0
18:02:23.982514 IP 10.42.4.3.45860 > 10.42.3.4.9090: Flags [S], seq 2417066922, win 29200, options [mss 1460,sackOK,TS val 19741050 ecr 0,nop,wscale 7], length 0
18:02:23.982595 IP 10.42.4.3.45862 > 10.42.3.4.9090: Flags [S], seq 3062027549, win 29200, options [mss 1460,sackOK,TS val 19741050 ecr 0,nop,wscale 7], length 0
18:02:23.982610 IP 10.42.4.3.45862 > 10.42.3.4.9090: Flags [S], seq 3062027549, win 29200, options [mss 1460,sackOK,TS val 19741050 ecr 0,nop,wscale 7], length 0
18:02:23.982663 IP 10.42.4.3.45864 > 10.42.3.4.9090: Flags [S], seq 2793781803, win 29200, options [mss 1460,sackOK,TS val 19741050 ecr 0,nop,wscale 7], length 0
18:02:23.982680 IP 10.42.4.3.45864 > 10.42.3.4.9090: Flags [S], seq 2793781803, win 29200, options [mss 1460,sackOK,TS val 19741050 ecr 0,nop,wscale 7], length 0
[...]
18:02:24.984278 IP 10.42.4.3.45862 > 10.42.3.4.9090: Flags [S], seq 3062027549, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0
18:02:24.984173 IP 10.42.4.3.45864 > 10.42.3.4.9090: Flags [S], seq 2793781803, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0
18:02:24.984291 IP 10.42.4.3.45864 > 10.42.3.4.9090: Flags [S], seq 2793781803, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0
18:02:24.984176 IP 10.42.4.3.45866 > 10.42.3.4.9090: Flags [S], seq 2282303013, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0
18:02:24.984302 IP 10.42.4.3.45866 > 10.42.3.4.9090: Flags [S], seq 2282303013, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0
18:02:24.984180 IP 10.42.4.3.45868 > 10.42.3.4.9090: Flags [S], seq 173986768, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0
18:02:24.984313 IP 10.42.4.3.45868 > 10.42.3.4.9090: Flags [S], seq 173986768, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0
18:02:24.984183 IP 10.42.4.3.45870 > 10.42.3.4.9090: Flags [S], seq 1030721675, win 29200, options [mss 1460,sackOK,TS val 19742052 ecr 0,nop,wscale 7], length 0

shell in prometheus-agent proves that prometheus container is able to collect and provide metrics
curl "http://localhost:9090/api/v1/query_range?end=2019-03-18T17%3A13%3A24.061195063Z&query=sum%28rate%28container_network_transmit_packets_total%7Bnamespace%3D~%22cattle-prometheus%22%2Cpod_name%3D~%22prometheus-cluster-monitoring-0%22%2C+container_name%21%3D%22%22%7D%5B5m%5D%29%29&start=2019-03-18T16%3A13%3A24.061195063Z&step=60.000"
{"status":"success","data":{"resultType":"matrix","result":[{"metric":{},"values":[[1552925604.061,"9.492592592592592"],[1552925664.061,"9.670370370370371"],[1552925724.061,"9.55925925925926"],[1552925784.061,"8.644444444444446"],[1552925844.061,"9.385185185185186"],[1552925904.061,"9.137037037037038"],[1552925964.061,"9.1"],[1552926024.061,"9.496296296296297"],[1552926084.061,"9.014814814814814"],[1552926144.061,"9.511111111111111"],[1552926204.061,"9.518518518518519"],[1552926264.061,"9.837037037037037"],[1552926324.061,"10.203703703703704"],[1552926384.061,"10.222222222222223"],[1552926444.061,"10.314814814814817"],[1552926504.061,"10.003703703703705"],[1552926564.061,"10.522222222222224"],[1552926624.061,"10.155555555555557"],[1552926684.061,"10.655555555555557"],[1552926744.061,"10.281481481481483"],[1552926804.061,"9.462962962962964"],[1552926864.061,"8.82962962962963"],[1552926924.061,"8.551851851851852"],[1552926984.061,"8.207407407407407"],[1552927044.061,"9.48888888888889"],[1552927104.061,"9.333333333333334"],[1552927164.061,"9.511111111111111"],[1552927224.061,"9.35925925925926"],[1552927284.061,"9.574074074074074"],[1552927344.061,"9.570370370370371"],[1552927404.061,"9.811111111111112"],[1552927464.061,"10.52962962962963"],[1552927524.061,"9.825925925925926"],[1552927584.061,"10.77037037037037"],[1552927644.061,"10.077777777777778"],[1552927704.061,"10.318518518518518"],[1552927764.061,"10.15925925925926"],[1552927824.061,"9.685185185185185"],[1552927884.061,"8.87037037037037"],[1552927944.061,"9.096296296296297"],[1552928004.061,"9.403703703703705"],[1552928064.061,"9.437037037037038"],[1552928124.061,"9.466666666666667"],[1552928184.061,"9.203703703703704"],[1552928244.061,"8.58148148148148"],[1552928304.061,"8.855555555555556"],[1552928364.061,"8.877777777777778"],[1552928424.061,"9.148148148148147"],[1552928484.061,"8.955555555555556"],[1552928544.061,"8.322222222222223"],[1552928604.061,"8.677777777777779"],[1552928664.061,"7.862962962962964"],[1552928724.061,"8.222222222222223"],[1552928784.061,"9.011144485720319"],[1552928844.061,"8.496296296296297"],[1552928904.061,"9.066666666666666"],[1552928964.061,"8.866666666666667"],[1552929024.061,"8.977777777777778"],[1552929084.061,"9.196296296296298"],[1552929144.061,"8.866666666666667"],[1552929204.061,"9.337037037037037"]]}]}}/

so we are back to the log in for the container prometheus-agent, saying that the proxy is not working
INFO[2019-03-18T17:28:40Z] listening on 10.42.3.5:9090, proxying to http://localhost:9090 with ignoring 'remote reader' labels [prometheus,prometheus_replica], only allow maximum 512 connections with 5m0s read timeout .
INFO[2019-03-18T17:28:40Z] Start listening for connections on 10.42.3.5:9090
2019/03/18 17:28:56 http: proxy error: context canceled
2019/03/18 17:28:57 http: proxy error: context canceled
2019/03/18 17:28:59 http: proxy error: context canceled
2019/03/18 17:29:06 http: proxy error: context canceled
2019/03/18 17:29:07 http: proxy error: context canceled
2019/03/18 17:29:16 http: proxy error: context canceled
2019/03/18 17:29:17 http: proxy error: context canceled

@jbroq , thank you for all the detail. There is a related issue from go: https://github.com/golang/go/issues/20071, but I need to reproduce this myself and try to fix.

thanks @thxCode
I am really happy to read that.

@jbroq we can't reproduce this issue on 2.2.2-rc12. Could you give this build a try?

2019/03/18 17:29:16 http: proxy error: context canceled is a no harmful log
Not sure if you hit https://github.com/rancher/rancher/issues/19243

Please let us know in a comment if you can still reproduce the issue, and we'll reopen it

authors: alena1108 jbroq loganhz thxCode


text:
Upgrade from Rancher 2.5+

Upgrade and rollback (downgrading using a lower version helm chart, data should stay the same)

No changes needed. Should follow the standard upgrading process.

source: gharchive/issue

repo: rancher/rancher

license_type: permissive

license_source: github-api


text:
cattle-cluster-agent behind proxy: can't add Chart Repository

What kind of request is this (question/bug/enhancement/feature request):
enhancement
Steps to reproduce (least amount of steps as possible):
create cluster behind proxy and add any public Chart Repository, for example, https://charts.jetstack.io
Result:
Chart Repository hangs with the "Downloading" status, connect: no route to host in cattle-cluster-agent logs
Other details that may be helpful:
as I understand cattle-cluster-agent created with template systemtemplate/template.go and it has no proxy environment variables
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5.5
Installation option (single install/HA): HA

Possibly related: https://github.com/rancher/rancher/issues/29961

Possibly related: https://github.com/rancher/rancher/issues/29961

Thank you for opening the issue, tracking it in #29961

Thank you for opening the issue, tracking it in #29961

authors: kinarashah kinolaev


text:
Given prefixPath of wins-upgrader is not considered

What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible):

On 2.5.7 - deployed a windows cluster (2004 SAC) with win_prefix_path set.
deployed the wins-upgrader app with the following set

masquerade:
  as: 'c:\\host\opt\etc\wmi-exporter\wmi-exporter.exe'
  enabled: false
prefixPath: 'c:\\host\opt'

The app failed to deploy. The wins-upgrader pod logs show:

FATA[2021-03-31T13:51:18Z] rpc error: code = NotFound desc = could not found binary: could not get checksum: exec: "c:\\etc\\rancher\\wins\\wins-upgrade.exe": file does not exist
Returned exit 1

Expected Result:
Given prefixPath of wins-upgrader should be considered
Other details that may be helpful:
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5.8-rc1/2.5.7
Installation option (single install/HA): single install

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): custom windows
Kubernetes version (use kubectl version):

1.20(Any)

@sowmyav27 I was able to successfully deploy in a cluster using win_prefix_path.
I tried this in a 2 node cluster (1 Ubuntu, 1 Windows 1909 Server Core) with the following override to the cluster.yaml:
rancher_kubernetes_engine_config:
  win_prefix_path: 'c:\host\opt'

And the following override on deploy:
masquerade:
  as: 'c:\host\opt\etc\wmi-exporter\wmi-exporter.exe'
  enabled: true
prefixPath: 'c:\host\opt\'

Moving it back to test to see if it can be replicated.

On 2.5-head commit id: 41f173ae93

monitoring chart version 14.5.100-rc07, deployed wins upgrader using

masquerade:
  as: 'c:\host\opt\etc\wmi-exporter\wmi-exporter.exe'
  enabled: true
prefixPath: 'c:\host\opt\'

and no error was seen.


text:
Rancher docker stopped working and doesn't start anymore

Rancher Server Setup

Rancher version: latest tag on docker hub
Installation option (Docker install/Helm Chart): Docker on standalone server

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):

Proxy/Cert Details:

Information about the Cluster

Kubernetes version: v1.21.7+k3s1
Cluster Type (Local/Downstream): Downstream imported K3S. Cluster of raspberry pi on the same network

If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider):

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)

If custom, define the set of permissions:

Admin
Describe the bug
After about a week of importing an existing cluster, doing operations, monitoring the cluster. The docker process of rancher just stopped and can no longer be restarted.
To Reproduce

Import an existing cluster into rancher
Wait about a week and the rancher docker process stops and doesn't restart.

Result
2022/01/26 05:49:46 [ERROR] error syncing 'cattle-fleet-system/helm-operation-vfh5c': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-vfh5c": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-fleet-system/helm-operation-vfh5c, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: a2061277-9147-465f-8c01-79619cb01147, UID in object meta: , requeuing

2022/01/26 05:49:55 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-gqgxh failed, watch closed

2022/01/26 05:50:55 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-wv89b failed, watch closed

2022/01/26 05:51:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:51:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:51:24 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 05:51:46 [ERROR] error syncing 'cattle-system/helm-operation-56vp6': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-56vp6": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-system/helm-operation-56vp6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 12603b25-b6a7-4a43-be8d-c625dece2f80, UID in object meta: , requeuing

2022/01/26 05:51:55 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-j4bgj failed, watch closed

2022/01/26 05:52:55 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-2skrr failed, watch closed

2022/01/26 05:53:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:53:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:53:47 [ERROR] error syncing 'cattle-fleet-system/helm-operation-pzzpf': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-pzzpf": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-fleet-system/helm-operation-pzzpf, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 19515e47-ee13-4438-8b68-218a9a9ff518, UID in object meta: , requeuing

2022/01/26 05:53:55 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-ch76l failed, watch closed

2022/01/26 05:54:11 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 05:54:55 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-p96gd failed, watch closed

2022/01/26 05:55:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:55:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:55:55 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-gtgkm failed, watch closed

2022/01/26 05:56:55 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-dvzvl failed, watch closed

2022/01/26 05:57:00 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 05:57:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:57:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:57:56 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-nxprp failed, watch closed

2022/01/26 05:58:56 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-8vd5s failed, watch closed

2022/01/26 05:59:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:59:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 05:59:45 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 05:59:56 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-5x888 failed, watch closed

2022/01/26 06:00:56 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-gmlhg failed, watch closed

2022/01/26 06:01:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:01:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:01:56 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-2nj82 failed, watch closed

2022/01/26 06:02:31 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:02:56 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-ml445 failed, watch closed

2022/01/26 06:03:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:03:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:03:56 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-bwpxv failed, watch closed

2022/01/26 06:04:57 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-xbtbt failed, watch closed

2022/01/26 06:05:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:05:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:05:20 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:05:57 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-8s9d9 failed, watch closed

2022/01/26 06:06:57 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-rslp9 failed, watch closed

2022/01/26 06:07:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:07:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:07:57 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-9hcfj failed, watch closed

2022/01/26 06:08:05 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:08:48 [ERROR] error syncing 'cattle-fleet-system/helm-operation-6tdjm': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-6tdjm": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-fleet-system/helm-operation-6tdjm, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 0ab1ad6e-0d37-4e76-ad98-9df26dab2d15, UID in object meta: , requeuing

2022/01/26 06:08:57 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-2tsn7 failed, watch closed

2022/01/26 06:09:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:09:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:09:49 [ERROR] error syncing 'cattle-system/helm-operation-hl2vv': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-hl2vv": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-system/helm-operation-hl2vv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 13599241-50f0-41ea-9b99-4156a2b7edef, UID in object meta: , requeuing

2022/01/26 06:09:57 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-cv6pq failed, watch closed

2022/01/26 06:10:46 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:10:49 [ERROR] error syncing 'cattle-fleet-system/helm-operation-jg4d7': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-jg4d7": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-fleet-system/helm-operation-jg4d7, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 36616e73-de28-4a04-af95-8f955a3b7a76, UID in object meta: , requeuing

2022/01/26 06:10:57 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-mtznl failed, watch closed

2022/01/26 06:11:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:11:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:11:57 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-djvtt failed, watch closed

2022/01/26 06:12:58 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-ll69h failed, watch closed

2022/01/26 06:13:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:13:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:13:35 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:13:58 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-jkl6g failed, watch closed

2022/01/26 06:14:58 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-bwbkj failed, watch closed

2022/01/26 06:15:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:15:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:15:58 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-fznrz failed, watch closed

2022/01/26 06:16:18 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:16:58 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-zs4s7 failed, watch closed

2022/01/26 06:17:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:17:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:17:58 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-qsl6c failed, watch closed

2022/01/26 06:18:58 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-654sq failed, watch closed

2022/01/26 06:19:04 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:19:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:19:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:19:59 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-jlndk failed, watch closed

2022/01/26 06:20:59 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-qs579 failed, watch closed

2022/01/26 06:21:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:21:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:21:44 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:21:59 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-rl8qt failed, watch closed

2022/01/26 06:22:59 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-2gqdx failed, watch closed

2022/01/26 06:23:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:23:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:23:59 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-crlcw failed, watch closed

2022/01/26 06:24:32 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:24:59 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-swrw5 failed, watch closed

2022/01/26 06:25:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:25:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:25:59 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-rmltg failed, watch closed

2022/01/26 06:26:59 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-8xbvj failed, watch closed

2022/01/26 06:27:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:27:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:27:17 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:28:00 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-zp95p failed, watch closed

2022/01/26 06:29:00 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-vd7hp failed, watch closed

2022/01/26 06:29:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:29:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:30:00 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-tdpz8 failed, watch closed

2022/01/26 06:30:04 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:31:00 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-5l6gx failed, watch closed

2022/01/26 06:31:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:31:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:32:00 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-v22gq failed, watch closed

2022/01/26 06:32:56 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:33:00 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-r9s78 failed, watch closed

2022/01/26 06:33:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:33:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:34:00 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-h6kh5 failed, watch closed

2022/01/26 06:35:00 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-pmj8n failed, watch closed

2022/01/26 06:35:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:35:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:35:42 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:36:01 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-ppcvg failed, watch closed

2022/01/26 06:37:01 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-8lc95 failed, watch closed

2022/01/26 06:37:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:37:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:38:01 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-fbb29 failed, watch closed

2022/01/26 06:38:25 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:38:53 [ERROR] error syncing 'cattle-fleet-system/helm-operation-p78nx': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-p78nx": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-fleet-system/helm-operation-p78nx, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: b9c9432b-dc13-4f51-970c-981b9dcee8f4, UID in object meta: , requeuing

2022/01/26 06:39:01 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-sk684 failed, watch closed

2022/01/26 06:39:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:39:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:40:01 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-lr42p failed, watch closed

2022/01/26 06:41:01 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-7kpr6 failed, watch closed

2022/01/26 06:41:11 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:41:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:41:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:42:01 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-jnd4k failed, watch closed

2022/01/26 06:43:01 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-xgw8g failed, watch closed

2022/01/26 06:43:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:43:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:43:56 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:44:02 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-qq92x failed, watch closed

2022/01/26 06:45:02 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-ft9hr failed, watch closed

2022/01/26 06:45:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:45:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:45:54 [ERROR] error syncing 'cattle-system/helm-operation-d7fqn': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-d7fqn": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-system/helm-operation-d7fqn, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 3e410350-9ee4-4aed-ae84-0b8410fd5c93, UID in object meta: , requeuing

2022/01/26 06:46:02 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-98l2g failed, watch closed

2022/01/26 06:46:37 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:47:02 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-56ngg failed, watch closed

2022/01/26 06:47:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:47:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:48:02 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-nmh2q failed, watch closed

2022/01/26 06:49:02 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-zr8zp failed, watch closed

2022/01/26 06:49:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:49:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:49:19 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:50:02 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-f4jtp failed, watch closed

2022/01/26 06:51:03 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-8tzvc failed, watch closed

2022/01/26 06:51:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:51:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:51:59 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:52:03 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-kn6lv failed, watch closed

2022/01/26 06:53:03 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-cpf4f failed, watch closed

2022/01/26 06:53:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:53:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:54:03 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-8s8zz failed, watch closed

2022/01/26 06:54:47 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:55:03 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-bn7cc failed, watch closed

2022/01/26 06:55:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:55:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:56:03 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-9c86x failed, watch closed

2022/01/26 06:57:03 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-bjxwp failed, watch closed

2022/01/26 06:57:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:57:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:57:32 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 06:57:56 [ERROR] error syncing 'cattle-system/helm-operation-8vd5s': handler helm-operation: Operation cannot be fulfilled on operations.catalog.cattle.io "helm-operation-8vd5s": StorageError: invalid object, Code: 4, Key: /registry/catalog.cattle.io/operations/cattle-system/helm-operation-8vd5s, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 7ccdc3a8-036a-4899-b53b-918de9576d32, UID in object meta: , requeuing

2022/01/26 06:58:03 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-d7gjw failed, watch closed

2022/01/26 06:59:04 [ERROR] Failed to install system chart rancher-webhook: pod cattle-system/helm-operation-58jm6 failed, watch closed

2022/01/26 06:59:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 06:59:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 07:00:04 [ERROR] Failed to install system chart fleet-crd: pod cattle-system/helm-operation-765kz failed, watch closed

2022/01/26 07:00:22 [ERROR] error syncing 'c-m-7ssl7cbl': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": cluster agent disconnected, requeuing

2022/01/26 07:01:04 [ERROR] Failed to install system chart fleet: pod cattle-system/helm-operation-fj6t6 failed, watch closed

2022/01/26 07:01:11 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 07:01:11 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing

2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterAlert workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterLogging workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=Notifier workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=Preference workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterRegistrationToken workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=MultiClusterAppRevision workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ConfigMap workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=DynamicSchema workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Namespace workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=PodSecurityPolicyTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ResourceQuota workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Secret workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=User workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Node workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=LimitRange workers
2022/01/26 07:01:20 [INFO] Shutting down project.cattle.io/v3, Kind=PipelineExecution workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=MultiClusterApp workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=NodeTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Namespace workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ProjectLogging workers
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=RKEBootstrap workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ConfigMap workers
2022/01/26 07:01:20 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ConfigMap workers
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=CustomMachine workers
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=RKEBootstrapTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine-config.cattle.io/v1, Kind=VmwarevsphereConfig workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=KontainerDriver workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine-config.cattle.io/v1, Kind=Amazonec2Config workers
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=RKEBootstrap workers
2022/01/26 07:01:20 [INFO] Shutting down provisioning.cattle.io/v1, Kind=Cluster workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine-config.cattle.io/v1, Kind=AzureConfig workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=VmwarevsphereMachineTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=RKEControlPlane workers
2022/01/26 07:01:20 [INFO] Shutting down cluster.x-k8s.io/v1alpha3, Kind=Cluster workers
2022/01/26 07:01:20 [INFO] Shutting down cluster.x-k8s.io/v1alpha3, Kind=Machine workers
2022/01/26 07:01:20 [INFO] Shutting down monitoring.coreos.com/v1, Kind=Prometheus workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down monitoring.coreos.com/v1, Kind=ServiceMonitor workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Secret workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=AzureMachine workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=SamlToken workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=HarvesterMachine workers
2022/01/26 07:01:20 [INFO] Shutting down networking.k8s.io/v1, Kind=Ingress workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Endpoints workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down cluster.x-k8s.io/v1alpha3, Kind=MachineSet workers
I0126 07:01:20.889706 33 trace.go:205] Trace[1961689565]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:11.382) (total time: 185649507ms):
Trace[1961689565]: [51h34m9.50746478s] [51h34m9.50746478s] END
I0126 07:01:20.889824 33 trace.go:205] Trace[277571127]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:11.313) (total time: 185649576ms):
Trace[277571127]: [51h34m9.576707688s] [51h34m9.576707688s] END
2022/01/26 07:01:20 [INFO] Shutting down admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration workers
I0126 07:01:20.889920 33 trace.go:205] Trace[960432200]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:11.138) (total time: 185649751ms):
Trace[960432200]: [51h34m9.751852197s] [51h34m9.751852197s] END
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=RKEControlPlane workers
I0126 07:01:20.891025 33 trace.go:205] Trace[2001662301]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:11.066) (total time: 185649824ms):
Trace[2001662301]: [51h34m9.824643275s] [51h34m9.824643275s] END
2022/01/26 07:01:20 [INFO] Shutting down apps/v1, Kind=ReplicaSet workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=UserAttribute workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ComposeConfig workers
I0126 07:01:20.892001 33 trace.go:205] Trace[506514743]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:11.016) (total time: 185649874ms):
Trace[506514743]: [51h34m9.8748787s] [51h34m9.8748787s] END
I0126 07:01:20.892117 33 trace.go:205] Trace[1352124239]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:10.997) (total time: 185649894ms):
Trace[1352124239]: [51h34m9.894211882s] [51h34m9.894211882s] END
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Secret workers
2022/01/26 07:01:20 [INFO] Shutting down catalog.cattle.io/v1, Kind=App workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2022/01/26 07:01:20 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Node workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2022/01/26 07:01:20 [INFO] Shutting down batch/v1beta1, Kind=CronJob workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ResourceQuota workers
2022/01/26 07:01:20 [INFO] Shutting down batch/v1, Kind=Job workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=LinodeMachine workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterAlertRule workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=LimitRange workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Namespace workers
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=CustomMachine workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Service workers
I0126 07:01:20.893532 33 trace.go:205] Trace[516470414]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:10.986) (total time: 185649906ms):
Trace[516470414]: [51h34m9.906785154s] [51h34m9.906785154s] END
I0126 07:01:20.893648 33 trace.go:205] Trace[1678026043]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:10.806) (total time: 185650086ms):
Trace[1678026043]: [51h34m10.086911309s] [51h34m10.086911309s] END
I0126 07:01:20.893746 33 trace.go:205] Trace[1055613205]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:10.736) (total time: 185650157ms):
Trace[1055613205]: [51h34m10.157532056s] [51h34m10.157532056s] END
I0126 07:01:20.893836 33 trace.go:205] Trace[495245640]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:27:10.644) (total time: 185650249ms):
Trace[495245640]: [51h34m10.249594926s] [51h34m10.249594926s] END
I0126 07:01:20.893932 33 trace.go:205] Trace[1377920201]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:26:14.866) (total time: 185706027ms):
Trace[1377920201]: [51h35m6.027517879s] [51h35m6.027517879s] END
2022/01/26 07:01:20 [INFO] Shutting down cluster.x-k8s.io/v1alpha4, Kind=MachineDeployment workers
2022/01/26 07:01:20 [INFO] Shutting down rke.cattle.io/v1, Kind=RKECluster workers
2022/01/26 07:01:20 [INFO] Shutting down catalog.cattle.io/v1, Kind=Operation workers
2022/01/26 07:01:20 [INFO] Shutting down admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration workers
I0126 07:01:20.894953 33 trace.go:205] Trace[2020229790]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:26:14.797) (total time: 185706096ms):
Trace[2020229790]: [51h35m6.096561493s] [51h35m6.096561493s] END
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterAlertGroup workers
2022/01/26 07:01:20 [INFO] Shutting down fleet.cattle.io/v1alpha1, Kind=Bundle workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ConfigMap workers
2022/01/26 07:01:20 [INFO] Shutting down apps/v1, Kind=Deployment workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ConfigMap workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ReplicationController workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=GroupMember workers
2022/01/26 07:01:20 [INFO] Shutting down apps/v1, Kind=ReplicaSet workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Secret workers
2022/01/26 07:01:20 [INFO] Shutting down batch/v1, Kind=Job workers
2022/01/26 07:01:20 [INFO] Shutting down fleet.cattle.io/v1alpha1, Kind=Cluster workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=UserAttribute workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=Token workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=User workers
2022/01/26 07:01:20 [INFO] Shutting down batch/v1beta1, Kind=CronJob workers
2022/01/26 07:01:20 [INFO] Shutting down apps/v1, Kind=Deployment workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ReplicationController workers
I0126 07:01:20.898236 33 trace.go:205] Trace[724862757]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.22.3-rancher.1/tools/cache/reflector.go:168 (24-Jan-2022 03:26:14.403) (total time: 185706491ms):
2022/01/26 07:01:20 [INFO] Shutting down apps/v1, Kind=DaemonSet workers
2022/01/26 07:01:20 [INFO] Shutting down apps/v1, Kind=StatefulSet workers
2022/01/26 07:01:20 [INFO] Shutting down apps/v1, Kind=StatefulSet workers
2022/01/26 07:01:20 [INFO] Shutting down cluster.x-k8s.io/v1alpha4, Kind=Cluster workers
Trace[724862757]: [51h35m6.491151769s] [51h35m6.491151769s] END
2022/01/26 07:01:20 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Node workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Namespace workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=LimitRange workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Node workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ResourceQuota workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterScan workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=Secret workers
2022/01/26 07:01:20 [INFO] Shutting down cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck workers
2022/01/26 07:01:20 [INFO] Shutting down cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=HarvesterMachineTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ConfigMap workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine.cattle.io/v1, Kind=DigitaloceanMachine workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ConfigMap workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ManagedChart workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterCatalog workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=Node workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ProjectCatalog workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=Feature workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=GroupMember workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=Group workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterRoleTemplateBinding workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=AuthConfig workers
2022/01/26 07:01:20 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2022/01/26 07:01:20 [INFO] Shutting down apiextensions.k8s.io/v1, Kind=CustomResourceDefinition workers
2022/01/26 07:01:20 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=CatalogTemplate workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=CisBenchmarkVersion workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=NodePool workers
2022/01/26 07:01:20 [INFO] Shutting down rke-machine-config.cattle.io/v1, Kind=HarvesterConfig workers
2022/01/26 07:01:20 [INFO] Shutting down catalog.cattle.io/v1, Kind=ClusterRepo workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=RkeK8sServiceOption workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=GlobalRole workers
2022/01/26 07:01:20 [INFO] Shutting down management.cattle.io/v3, Kind=Project workers
2022/01/26 07:01:20 [FATAL] context canceled

Expected Result
Rancher container starts up normally. Showing information about cluster even if the remote cluster is not available.

Did you ever figure anything out? I'm having the same issue on a fresh install about 2 weeks old

No I have not, it keeps crashing even after a new fresh install. I just stopped using rancher and installed kubernetes-dashboard to have some sort of control over the cluster with a web UI.


text:
Swarm CLI fails to launch through UI

**Rancher Version:**1.2.0-pre4-rc11
**Docker Version:**1.12.2
**OS and where are the hosts located? (cloud, bare metal, etc):**Ubuntu, Cloud
**Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)**Single node
**Environment Type: (Cattle/Kubernetes/Swarm/Mesos)**Cattle
Steps to Reproduce:

Create a Swarm setup by launching the swarmkit catalog
After the swarm stack comes up, click on CLI from "Swarm" tab

Results:
CLI does not launch. Error screen below:

This is probably because swarm mode (current) != swarm (old).
With swarm mode, we will need the Swarm CLI link in the UI to point to a swarmkit container on a host with label swarm=manager. If there are no hosts, return No manager available at this time error response.

Version - v1.2.0-rc2
Verified fixed

authors: LLParse soumyalj tfiduccia


text:
Add Validator for RoleTemplate

Problem:
Builtin RoleTemplates can be updated through the API
Solution:
Add a Validator to ensure the only field that can be changed on a
builtin RoleTemplate is 'locked'
Issue: https://github.com/rancher/rancher/issues/13595

LGTM

source: gharchive/pull-request

authors: cjellick dramich


text:
Update golang version to fix golang TLS issue for AD

https://github.com/rancher/rancher/issues/14502
After updating to go1.10.3, the CGO_ENABLED=0 part of build command
in build-server script threw this error "loadinternal: cannot find runtime/cgo"
and then that would eventually lead to a segmentation fault. The build
would just freeze without running any tests. This commit removes CGO_ENABLED=0,
since cgo is disabled by default when cross compiling, as per https://golang.org/cmd/cgo/
f164240

LGTM

authors: ibuildthecloud mrajashree


text:
Audit log enhancements for Kubectl access path

https://github.com/rancher/rancher/issues/33126
This fix is to support audit logging from kubectl access which use kubeconfig tokens. In case of kubeconfig, the userPrincipal is not found and is missing information. Hence the audit log data is not seen.
As a fix, instead of reading extraInfo from user's login token, this is now read from userattributes where the extra info is saved after user logs in. This information can be found using the userId extracted from any token.
The userAttributes.Extra is also saved to ClusterUserAttributes.Extra field.

Discussed with @dramich and @aiyengar2 and following changes/alternatives were identified:

This solution is saving extraInfo from user's login token into userAttribute CR and reading it back by using the token's userId. But we need to additionally lookup principals of the user from the token or from userId if extraInfo is not found in userAttribute  - this is necessary to take care of the upgrade case where old tokens/old userAttribute will not have the extraInfo
We also should ideally fix kubeconfig token to carry the real authProvider and not set "local"; this will let us use the correct principalId of that provider instead of us adding a list of all principalIds to audit log

@prachidamle , even if this is approved, please do not merge it to master yet as it was decided this should not get into 2.6.0  release. So, until branches are created for release/v2.6 (and likely release/v2.6) we should't put it in master.

@snasovich I have reopened this against release/v2.6, added you as well fro review since you had reviewed and approved it earlier.
Closing this one.

authors: prachidamle snasovich


text:
Graduate Wrangler to stable and move to v1.x.y versioning

As part of a breaking change https://github.com/rancher/wrangler/pull/194 that is required for kubernetes 1.24 support, I believe we should graduate this project from beta to stable.

IMO due to the breaking change introduced to the signal handler it should already be on 1.x and this should make it go to 2.x

Wrangler [BETA - Lacks full documentation]

That's actually the title of wrangler's README. So it's not beta in the sense that it has bugs, but because it's not a well maintained project with docs, a release cycle and a logo.
Since it's mostly an internal dependency that is okay and I understand that having a major version might help with automatically bumping it and such.
Still, I would suggest to add some documentation, at least so people know what kind of stableness they can expect. What is the public API of wrangler, the modules below pkg, the example from the README?  The code generator invocation? The generated controllers? The supported k8s versions?
Note: Change is not a bad thing, obviously a change is needed to work with newer k8s versions. Supporting new versions is good, right? Wrangler depends on apimachinery and client-go, two big libraries. So it will have to change when those upstream dependencies change.

authors: ekristen manno rosskirkpat

repo: rancher/wrangler


text:
When container is restored and started, it is not part of LB configuration any more

Server Version - V0.14.2
Steps to reproduce  the problem:
Create LB with LB config that has 1 listener.
Add 1 host and 2 targets.
Remove one of the containers.
Notice that this target (container) is already removed from the LB's list of targets.
Restore this container and start it.
This container is not reachable using the LB Ip address.
Expected Behavior:
Container should be reachable using the LB Ip address after it was deleted , restored and started.

I should change the trigger point for lb-target map removal from instance.destroy to instance.purge

@sangeethah: @alena1108 says this was fixed a long time ago. i've just labeled it for testing.

Closing as won't fix as the support for standalone LB is disabled.

authors: alena1108 deniseschannon sangeethah

repo: rancherio/rancher

license_source: bigquery


text:
Can't change component state in Angular 9.0.0

Augury version (required): 1.25.2
Angular version (required): 9.0.0
Date: 2020/2/8
OS: Windows 10 (Build 10.0.18363.592)
Demo test application:
-- Git repository for demo app showing the issue (optional but very helpful for difficult issues).
-- If a code snippet will completely show the issue, please include it.
ng new demo1 --routing --style css
ng serve --open

Description of issue:
-- Include (clipped) screenshot images if possible.
I can't change component state.

Steps to reproduce:

ng new demo1 --routing --style css
ng serve --open
Hit F12
Click Augury tab
Click AppComponent
Modify component state for title property from demo1 to demo2.

Additional details:

set the rendering mode to components only. (for now)

Setting the rendering mode to components only didn't solve the issue for me

authors: Venipa doggy8088 lastrafda

repo: rangle/augury

license: mit


text:
add react native segment target

Checklist

[x] I have added tests that prove my fix is effective or that my feature works.
[x] I have added all necessary documentation (if appropriate)

What was done
Added Segment using the React Native library, for native analytics lib.
Basically, I copied the segment target and modified it to use the native library.
Associated Issues
Thank you.

we have tested it, yes, and we are actually using it from our fork. let me know if you need something else for that.

@pvinis Awesome! This is now available in npm under @redux-beacon/react-native-segment 🙂

authors: pvinis ttmarek

repo: rangle/redux-beacon

license: MIT


text:
Cant place enchanted book in soldener

I cant place enchanted book in soldener
Cant make Silk touch upgrade
refinedstorage-1.2.11
forge-1.10.2-12.18.3.2185-universal

Can't reproduce. Some other mod is causing a conflict with RS.

I made a fortune 3 book from combining fortune 2 books, and i cant place that in a solderer
Does the solderer check enchanted books for nbt? Because if so, minecraft adds the repaircost tag when you combine things in an anvil, and if the solderer expects no nbt but it has that, it wouldnt allow the book
Or am i overthinking this?

Aha, that might be it. Didn't test with repairing.

authors: DuskFalls Kruberoid raoulvdberge

repo: raoulvdberge/refinedstorage


text:
Footnotes exports incorrect semantic html

Description
The generated html for footnotes has the <sup> tag outside of the <p> tag of the footnote. Phrasing Content such as <pre> tags should be inside a paragraph. Because the <sup> doesn't belong to the <p>, it ends up on its own line and requires additional css to make it display correctly.
I'm happy to tackle this in a Pull Request, but wanted to make sure there's agreement that this is a valid bug report first.
Expected Behavior
Markdown with footnotes generates html with the correct semantic behavior. The <sup> tag for the footnote number should be inside the <p> tag for the footnote content.
jsfiddle
Actual Behavior
Markdown with footnotes generates html with a <sup> as a direct child of a <div>
jsfiddle
Note: This html was copied directly out of the ./specs/footnotes.txt

I took an initial look into this yesterday, but I think it might end up being a somewhat complex change.
Hope to take another stab at it this weekend, but I'm not sure it'll be possible without either modifying the parser to change the order it pushes footnote items, or turning the run iterator into a peek-able iterator.

Hmm yes, it seems those are our options. Perhaps easiest would be to never generate a paragraph node for reference definitions, since they will always contain exactly one anyway. That's a larger breaking change than doing some peaking-ahead wizardry inside the renderer, though that seems less attractive from a technical perspective.

Perhaps easiest would be to never generate a paragraph node for reference definitions, since they will always contain exactly one anyway.

But that's not true. Footnotes are allowed to contain more than one paragraph. You can see that they do both in the spec file, and in "markdown guide".
What the multimarkdown seems to do, based on babelmark results, is generate an <ol>.

TIL. That's interesting. Perhaps the solution is to simply generate an ordered list always? That'd be good from a semantic standpoint, right? And if I'm not mistaken, it should be trivial to implement.

TIL. That's interesting. Perhaps the solution is to simply generate an ordered list always? That'd be good from a semantic standpoint, right? And if I'm not mistaken, it should be trivial to implement.

I'm not sure I get what your proposing: if you have footnotes definitions scattered in the document (which I think Commonmark allows), would you generate an ol for each footnote? As far as I can tell there is currently no concept of "the single footnote block of this documents, where all footnotes definitions go".

The strategy of generating a list seems common. I'm trying to get littlefoot.css working with zola, which relies on pulldown-cmark, and what I'm noticing is that the non-standard output here causes littlefoot to fail to identify the footnotes as such.
Would be in favor of standardization here.

authors: diminishedprime krtab marcusklaas notriddle ragona

repo: raphlinus/pulldown-cmark


text:
Fix #113 - Define Stack#fetch which fails fast when a value is unset

Fix #113.

@erran Is there a test that validates this new behavior?

@erran Is there a test that validates this new behavior?

@athompson-r7 Now there are (since https://github.com/rapid7/convection/pull/192/commits/e0b7517936c3c5f9207f5e15bec35f251908f459). 😜


text:
Update run script for local builds

Update run script to download GDS from public package (cufile.h is needed to build it locally)

Update CMakeLists.txt accordingly
GDS is still disabled (need to use GDS Conda package and verify it before enabling GDS)

Update CUDA EULA (which include libcufile)
Add dependency to yasm which is needed for building libjpeg-turbo
Clear CMakeCathe.txt and existing .so files to build with the updated options
Minor code change (unnecessarily std::move which degrade performance).
Add changelog for v0.18.3 and update the release date of v0.19.0 to 4/19/2021

Verified with a fresh system that https://github.com/rapidsai/cucim/blob/branch-0.20/CONTRIBUTING.md#script-to-build-cucim-from-source is working. (Previously it didn't work due to yasm dependency).

Hi @raydouglass
Since we didn't tag v0.19.0 yet and VERSION files in branch-0.20 are still referring to 0.19.0, would it make sense to fast-forward branch-0.19 branch to branch-0.20 instead of creating a PR with the same commits to branch-0.19?
If branch-0.19 is not available for fast-forwarding, I can create a PR with the duplicate commits in branch-0.20 for branch-0.19.

I see some build error because python build is using the existing libcucim conda package but libcucim code in C++ is also changed in the PR.
: https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci-v0.20/job/cucim/job/prb/job/cucim-cpu-python-build/CUDA=11.0,PYTHON=3.7/17/console line 704
Wonder I need to push PRs for libcucim(core lib code) and cucim (pybind11&python code) separately.

It seems that it always using libcucim from nightly build
https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci-v0.20/job/cucim/job/prb/job/cucim-cpu-python-build/CUDA=11.0,PYTHON=3.7/18/console
18:08:35     libcucim:         0.19.0a210416-cuda11.0_g6cfdbfb_9 rapidsai-nightly

@ajschmidt8 Do you have any idea on that? How could we handle the situation?

I found that libcucim's built package is available at ${WORKSPACE}/ci/artifacts/cucim/cpu/.conda-bld when cucim is built so I updated build.sh for cpu cucim build through https://github.com/rapidsai/cucim/pull/12/commits/cd3ae09578407c401515febff2cca0316f57211a, to use cpu libcucim build.
This approach was already made in GPU build (https://github.com/rapidsai/cucim/blob/branch-0.20/ci/gpu/build.sh#L61).
This PR is passing now.
@quasiben, please help review the change.
This change also requires an approval from @jakirkham (rapidsai/cucim-cpp-codeowners) so I don't know how to merge it without @jakirkham's help.

repo: rapidsai/cucim


text:
[FEA] Rust API

Rust API has been highly requested by users. Ideally this would be a wrapper around the C-API

Closed with #34

repo: rapidsai/cuvs


text:
Fix remaining unsafe string path creations

https://github.com/rapidsai/tpcx-bb/pull/80 resolved most path issues, but we still can hit this when we circumvent the scheduler and mainline API and use delayed cudf reads.

This is the root cause of #128

Closed by https://github.com/rapidsai/tpcx-bb/pull/130

authors: VibhuJawa beckernick

repo: rapidsai/tpcx-bb


text:
unable to access admin part

access/users url redirecting to dashboard. i used laravel-5-boilerplate and vault package, i didn't found any errors at time of installation but am unable access admin part, all Role and Permissions of Controllers and Middleware files are at Vendor only

I think you see this because you are not logged in as administrator.
Try this:

go to yourtestarea.app:8000/auth/login
login as admin (if you "migrate and seed table" this is the account "admin@admin.com" and password i don't know. i generate new password for my testcase).
after this go to yourtestarea.app:8000/access/users

With this method, i can enter in the Route correctly

Yes it will obviously only allow access if you are logged in as the default admin. Make sure you run the seeder, after doing so you can change the admin password from the backend.

authors: NeDetto rameshreddyvemula rappasoft

repo: rappasoft/vault


text:
accessing nested widgets with gaps

There seem to be some issues in accessing nested widgets when there are gaps between widgets. In this case we have some parentWidget. And then we create children widgets (similar issues can be replicated when using custom tags rather than invoked renderer).
Say that we have some child templates (foo, bar, and baz). Only baz is an actual marko widget, while foo and bar are just template.marko files.
data.renderers = [
    require('./foo'),
    require('./bar'),
    require('./baz')
];

Then we assign nested widgets as follows (although these are not all "true" widgets):
<for each="renderer in data.renderers">
    <invoke function="renderer(myInputObject, out)" w-id="childWidget[]" />
</for>

This will result in markup as follows:

foo

bar

baz

The childWidget in this case starts at index 2, because foo and bar were correctly skipped. However, when trying to access the nested widgets, this gap seems to cause problems.
parentWidget.getWidgets('childWidget') // comes up empty

Should everything be created as widget to begin with, in order to avoid issues like this? Wouldn't that create unnecessary overhead? Ideally we would only use w-id when we know it's a widget we want to access, but I imagine in some cases that may not be possible when using both widgets and non-widgets similarly.

I think this is fair to say that what you are doing is very much an edge case. When a w-id attribute is assigned, we surround the nested widget with code that passes along a generated widget ID to the nested widget by storing the assigned widget ID in the out context. When the nested widget renders it will pick up its assigned widget ID out of the same out context variable. After the widget renders we clear out that variable in the out context. It is actually considered user error if a widget ID is assigned but no widget renders to pick up that ID. We can try to handle your use case, but it will put some more complexity into the runtime... I'll take a closer look and see what we can do.

I agree it is somewhat of an edge case. I guess it seems like either the index shouldn't get incremented with non-widgets, or the parent should be able to access child widgets regardless of index gaps. I have had the gap issue come up in the past, and it can usually be resolved by either 1) making everything widgets, or 2) being very strict about when you try to assign w-id(s).

I'm looking into a solution that would have the nested widget increment the repeated widget ID counter instead of the parent providing the pre-incremented widget ID counter. I'll provide an update soon.

Changing the logic such that the nested widget increments the repeated ID seems to solve your edge case without introducing any other problems. I will be publishing a new version soon.

New version published: marko-widgets@4.2.1

Awesome, thanks!

authors: patrick-steele-idem yomed

repo: raptorjs/marko-widgets


text:
Laminar is a Scala.js reactive library [SEO]

It's not always ethical but seeing as you are a "reactive toolkit" but you don't appear on the first page searching for scala.js reactive library or scala.js reactive toolkit, a bit of SEO might help.
How about:
Native Scala.js library for building reactive user interfaces ?

Ok I've updated the issue title so that our search gods know that Laminar is a scala.js reactive library. In fact, you could say it's not just a Scala.js reactive library, but subjectively one of the finest scala.js reactive libraries. Scala.js surprisingly has more than its fair number of reactive libraries, but some of them are abandoned and unmaintained. Laminar is a popular reactive library in the Scala.js world.
Goddamit I can't do this anymore, that will have to be enough. :D
On a serious note, I think Laminar does not come up for those search terms because people don't often talk about this kind of library as "reactive library", but rather more often they call it a "frontend library", "frontend framework". As you see most other frontend UI libraries don't come up either for your search query. In fact if you google "javascript reactive library", Laminar's most prominent Javascript alternatives such as React don't come up either. Actually, I spent a couple minutes and couldn't come up with a short name-agnostic query that would give me React's website as the first result. Closest I came was with "javascript frontend library" which had React's website at #7. Everything else is just listicle spam from various websites I've never heard of. I think this says more about the current state of google search than anything else, really.
Fortunately Laminar users tend to be very vocal about our library, various people have been mentioning it in basically every discussion of Scala frontend tools that I've seen. I'm grateful for the enthusiasm – word of mouth is what really counts, long term, I think, especially now that all our "words" are recorded online for any competent search engine to pick up on.

repo: raquo/Laminar


text:
v0.2.1

Version 0.2.1dev
Changes

The amino3to1 method of biopandas.pdb.PandasPDB objects now returns a pandas DataFrame instead of a pandas Series object. The returned data frame has two columns, 'chain_id' and 'residue_name', where the former contains the chain ID of the amino acid and the latter contains the 1-letter amino acid code, respectively.
Significant speed improvements of the distance method of both PandasPdb and PandasMol2 (now about 300 percent faster than previously).

Bug Fixes

The amino3to1 method of biopandas.pdb.PandasPDB objects now handles multi-chain proteins correctly.
The amino3to1 method of biopandas.pdb.PandasPDB objects now also works as expected if the 'ATOM' entry DataFrame contains disordered DataFrame indices or duplicate DataFrame index values.

Coverage remained the same at 94.157% when pulling 10fa12e9c94892456d25539fb5d826ef81e57fae on 2.1 into 802f1a6b8f0594ab5dfac9dd35d4afaf7bf0146e on master.

authors: coveralls rasbt

repo: rasbt/biopandas


text:
Configure HA master node topology

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
In this approach, we want to use a stacked etcd topology.
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/

@omarst9

Some examples of bootstrapping a K8s HA master topology https://github.com/IBM/ansible-kubernetes-ha-cluster/blob/master/playbooks/kubernetes-masters.yaml

Once HAProxy has been configured on all masters, nodes should connect to masters using the VIP.

The following are some of the errors I've received when configuring the slave masters
error execution phase control-plane-join/update-status: error uploading configuration: etcdserver: leader changed

error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://192.168.1.123:2379 with maintenance client: context deadline exceeded