text:
Peer already exists error in MetalLb

The metallb of our Kubernetes cluster throws that error on startup:
{"branch":"HEAD","caller":"level.go:63","commit":"v0.12.1","goversion":"gc / go1.16.14 / amd64","level":"info","msg":"MetalLB controller starting version 0.12.1 (commit v0.12.1, branch HEAD)","ts":"2023-07-17T07:11:16.64900186Z","version":"0.12.1"}
{"caller":"level.go:63","configmap":"metallb-system/config","error":"peer #25 already exists","event":"configStale","level":"error","msg":"config (re)load failed, config marked stale","ts":"2023-07-17T07:11:24.348221116Z"}
{"caller":"level.go:63","event":"stateSynced","level":"info","msg":"controller synced, can allocate IPs now","ts":"2023-07-17T07:11:24.448924397Z"}

the respective peer config is:
    - hold-time: ""
      my-asn: xxxx
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: shoot--codesphere--fra-worker-67fdf-m76pk
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: default
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: default
      password: ""
      peer-address: 169.254.255.1
      peer-asn: xxxx
      peer-port: 0
      router-id: ""
      source-address: ""

In the equinix metalconsole I also see that we 2 nodes without a learned bgp route (might be related).
Environment:
K8s: 1.22.17
Equinix CCM: v3.6.2
MetalLb: 0.12.1

Hi @schrodit!  Is this causing your cluster and/or the applications running in it to malfunction in some way, or is the log message the only observable outcome?  Did this issue happen after upgrading CPEM in an existing cluster, or is this happening in a new cluster that was set up from the beginning with the software versions listed in the issue description?

Hey @ctreatma ,

Did this issue happen after upgrading CPEM in an existing cluster, or is this happening in a new cluster that was set up from the beginning with the software versions listed in the issue description?

the cpem and metallb were updated from metallb v9.5 and equinix ccm 3.3.0.

Is this causing your cluster and/or the applications running in it to malfunction in some way, or is the log message the only observable outcome?

An issue is that the bgp routes of some nodes are not correctly updated.
So only half of the nodes serve the IP according to the Metal Console. And less and less machines are serving the domain.
I think all nodes should serve the domain.

Could you provide the config for the other peers as well?  Are there 2 identical peer configs in metallb-system/config?

I think its the best if provide the whole configmap:

MetalLb Configmap

apiVersion: v1
data:
  config: |
    address-pools:
    - addresses:
      - 1.1.1.1/32
      auto-assign: false
      avoid-buggy-ips: false
      bgp-advertisements: []
      name: my-ns/prerequisites-ingress-nginx-controller
      protocol: bgp
    - addresses:
      - 1.1.1.2/32
      auto-assign: false
      avoid-buggy-ips: false
      bgp-advertisements: []
      name: my-ns/workspace-reverse-proxy-ingress-nginx-controller
      protocol: bgp
    bgp-communities: {}
    peers:
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: []
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.1
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
    - hold-time: ""
      my-asn: 65000
      node-selectors:
      - match-expressions: []
        match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      - match-expressions: null
        match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
      password: ""
      peer-address: 169.254.255.2
      peer-asn: 65530
      peer-port: 0
      router-id: ""
      source-address: ""
kind: ConfigMap
metadata:
  name: config
  namespace: metallb-system

We now had to completly roll back to equinix ccm v3.3.0 and metallb v0.9.5 as all bgp routes were deleted.
After the rollback everything was fine again

Do you have the config map for CPEM that you tried using?

Hey @cprivitere ,

Do you have the config map for CPEM that you tried using?

we do not use a configmap but rather the env var approach.
Our env vars are:
        env:
        - name: METAL_API_KEY
          valueFrom:
            secretKeyRef:
              key: apiToken
              name: cloudprovider
        - name: METAL_PROJECT_ID
          valueFrom:
            secretKeyRef:
              key: projectID
              name: cloudprovider
        - name: METAL_METRO_NAME
          value: fr
        - name: METAL_LOAD_BALANCER
          value: metallb:///

Do you have what the metallb config map looked like before the upgrade?

yes like it currently does (becasue we had to roll back):

peers:
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    bgp-communities: {}
    address-pools:
    - protocol: bgp
      name: my-ns/prerequisites-ingress-nginx-controller
      addresses:
      - 1.1.1.1/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []
    - protocol: bgp
      name: my-ns/workspace-reverse-proxy-ingress-nginx-controller
      addresses:
      - 1.1.1.2/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []peers:
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: prerequisites-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      - match-labels:
          nomatch.metal.equinix.com/service-name: workspace-reverse-proxy-ingress-nginx-controller
          nomatch.metal.equinix.com/service-namespace: my-ns
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-qz2k8
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-vzm6r
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrl7d
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-xrrdq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-66xwp
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-7692t
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m4rct
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-nlf7g
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-dp7dq
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-hc8bd
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-67fdf-m76pk
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.1
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    - my-asn: 65000
      peer-asn: 65530
      peer-address: 169.254.255.2
      peer-port: 0
      hold-time: ""
      router-id: ""
      node-selectors:
      - match-labels:
          kubernetes.io/hostname: my-worker-large-788c6-dkjbv
        match-expressions: []
      password: ""
    bgp-communities: {}
    address-pools:
    - protocol: bgp
      name: my-ns/prerequisites-ingress-nginx-controller
      addresses:
      - 1.1.1.1/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []
    - protocol: bgp
      name: my-ns/workspace-reverse-proxy-ingress-nginx-controller
      addresses:
      - 1.1.1.2/32
      avoid-buggy-ips: false
      auto-assign: false
      bgp-advertisements: []

Have you tried letting metallb rebuild the config map for metallb (obviously you'd want to do this in a testing environment to see how hard it is to kick the cluster into rebuilding that config map for you).

I think it did that when the new ccm was started. I think this was because first we used metallb 0.9.5 with ccm 3.6.2 which resulted in an error in the metallb (source-address was unknown).

Do you already have a planned timeframe for getting to MetalLB 0.13 or later? If so, we may be able to side-step troubleshooting this given the 0.13 versions use a CRD based setup and the work of getting to 0.12.1 won't help with that.

We are not tight to version 0.12.1. But I guess the CRD approach needs more testing. And the question is if it will work better.
Because I tried to reproduce the issue in a fresh environment, but at least there it worked.
Maybe another sidenote to our setup:
we have two services of type Loadbalancer configured whereas one of them has externalTrafficPolicy: Local.
Not sure if that makes a difference.
DO you have an Idea how we ended up there and how we can upgrade.
If you say that the CRD approach is the safest way we could do that (But currently as I understood the configmap way is the default. so the stbale one?).
Are there any things we need to consider while updating or is it mostly updating metallb and the configuring the ccm as described here

Because I tried to reproduce the issue in a fresh environment, but at least there it worked.
Are you saying that this works properly in a fresh environment so the issue here has to do with upgrading? We don't have a lot of testing or feedback about the upgrade process.
I'm wondering if you need to upgrade in steps of metallb minor versions until you reach an issue and then cpem minor versions until you hit an issue and then back to metallb.

Is it possible to instead plan to a fresh cluster and migrate the applications? I feel like a fresh cluster with 3.6.2 and MetalLB 0.13.X would be an easier route.

Are you saying that this works properly in a fresh environment so the issue here has to do with upgrading?

It works in my simple test. But I'm not sure how to reproduce the situation in our prod cluster.

Is it possible to instead plan to build a fresh cluster and migrate the applications?

This is unfortunately not possible. We have customers running on that cluster and can not simple migrate them.
@cprivitere do you have an idea why the ccm generates a invalid metallb config?
You're suggest way foward is to upgrade metallb. then upgrade the equinix ccm. What if there are still issues?

@schrodit
Still working on reproducing this and figuring out how to best guide you.  One thing that did come up in my testing. Between CPEM 3.3.0 and 3.6.2 we changed it from running as a deployment to running as a daemonset. Did you possibly have both the old and new version running at the same time? How did you go about stopping the old deployment and starting the new daemonset?

Ok we did some attempts to reproduce this and while we can't get the same exact issue you ran into, we did see several issues come up depending on how one tries to upgrade. Here's what we've found to be the best way to upgrade from MetalLB 0.9.5 and CPEM 3.3.0 to MetalLB 0.12.1 and CPEM 3.6.2.
This assumes your metallb lives in the metallb-system namespace and the metallb configmap is named "config".
# Remove current Metal LB
kubectl delete -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# Remove current CPEM
kubectl delete -f https://github.com/equinix/cloud-provider-equinix-metal/releases/download/v3.3.0/deployment.yaml
# Remove current MetalLB configmap (CPEM will regenerate it)
kubectl -n metallb-system delete cm config
# Install current Metal LB
kubectl  apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml
# Install current CPEM
kubectl apply -f https://github.com/equinix/cloud-provider-equinix-metal/releases/download/v3.6.2/deployment.yaml

Are you able to test this out and see if the load balancers come up following your upgrade?

Hi @schrodit, complementing @cprivitere ' comment and for other people seeking this information, during the tests, we found that version Metallb 0.9.5 doesn't work with CPEM 3.6.2, so upgrading both is necessary to ensure it works well. We also noticed that it is possible to upgrade them without deleting the configmap. However, we still recommend doing so when possible, as we encountered duplicated fields as in the config you shared when we perform multiple upgrades/rollbacks between different versions where changes were made to some data types and fields, and they were interpreted as different

@cprivitere @ocobles thanks for the input.
Cleaning up the config.yaml seems to be the best option for us.
We still try that out and let you know.

/reopen
/remove-lifecycle rotten
/triage accepted

authors: cprivitere ctreatma ocobles schrodit

repo: equinix/cloud-provider-equinix-metal