text:
troubleshooting connection refused error on linux

I have successfuly built source code on ubuntu (32 bit). I am trying to run neverball with ga-server-event-driven, game starts fine but client is not able to connect to the server. The behavior is the same when I use the pre-compiled binaries for Linux. Is there a way to troubleshoot this problem to see the error in more detail?
here is my server's output:
galinux@galinux-VirtualBox:~/myprog/gaminganywhere/bin$ ./ga-server-event-driven config/server.neverball.conf
# [11271] 1536687690.454406 # include: config/common/server-common.conf
# [11271] 1536687690.454468 # include: config/common/controller.conf
# [11271] 1536687690.454483 # include: config/common/video-x264.conf
# [11271] 1536687690.454495 # include: config/common/video-x264-param.conf
# [11271] 1536687690.454525 # include: config/common/audio-lame.conf
# [11271] 1536687690.454575 Game exe: /usr/games/neverball
Configuration file: /home/galinux/myprog/gaminganywhere/bin/./config/server.neverball.conf
# [11271] 1536687690.454596 Game dir: switched to /usr/games
# [11272] 1536687690.455236 CMD: LD_LIBRARY_PATH=/home/galinux/myprog/gaminganywhere/bin/. LD_PRELOAD="/home/galinux/myprog/gaminganywhere/bin/./ga-hook-sdl.so" GA_ROOT=/home/galinux/myprog/gaminganywhere/bin/. GA_CONFIG=/home/galinux/myprog/gaminganywhere/bin/./config/server.neverball.conf /usr/games/neverball
===================================================

galinux@galinux-VirtualBox:~/myprog/gaminganywhere/bin$ # [11274] 1536687690.558132 ga-hook-sdl loaded!
# [11274] 1536687690.559997 # include: /home/galinux/myprog/gaminganywhere/bin/./config/common/server-common.conf
# [11274] 1536687690.560063 # include: /home/galinux/myprog/gaminganywhere/bin/./config/common/controller.conf
# [11274] 1536687690.560084 # include: /home/galinux/myprog/gaminganywhere/bin/./config/common/video-x264.conf
# [11274] 1536687690.560102 # include: /home/galinux/myprog/gaminganywhere/bin/./config/common/video-x264-param.conf
# [11274] 1536687690.560139 # include: /home/galinux/myprog/gaminganywhere/bin/./config/common/audio-lame.conf
# [11274] 1536687690.560197 # RTSP[config]: using 'tcp' for RTP flows.
# [11274] 1536687690.560207 # RTSP[config]: controller port = 8555
# [11274] 1536687690.560214 # RTSP[config]: controller via 'udp' protocol.
# [11274] 1536687690.560274 # RTSP[config]: video-encoder = libx264 (libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
# [11274] 1536687690.560303 # RTSP[config]: audio-encoder = libmp3lame (libmp3lame MP3 (MPEG audio layer 3))
# [11274] 1536687690.560337 # RTSP[config]: video specific option: b = 3000000
# [11274] 1536687690.560348 # RTSP[config]: video specific option: g = 48
# [11274] 1536687690.560354 # RTSP[config]: video specific option: intra-refresh = 1
# [11274] 1536687690.560360 # RTSP[config]: video specific option: me_method = dia
# [11274] 1536687690.560366 # RTSP[config]: video specific option: me_range = 16
# [11274] 1536687690.560372 # RTSP[config]: video specific option: preset = faster
# [11274] 1536687690.560378 # RTSP[config]: video specific option: profile = main
# [11274] 1536687690.560383 # RTSP[config]: video specific option: refs = 1
# [11274] 1536687690.560390 # RTSP[config]: video specific option: slices = 4
# [11274] 1536687690.560395 # RTSP[config]: video specific option: threads = 4
# [11274] 1536687690.560401 # RTSP[config]: video specific option: tune = zerolatency
# [11274] 1536687690.564065 key-blocking initialized: 0+0 keys blocked.
# [11274] 1536687690.564136 SDL hook: initialized.
# [11274] 1536687690.564151 SDL hook: BlitSurface == UpperBlit.
Creating a window (800x600, windowed)
Created a window (2, 800x600, windowed)

and this is client's output:
galinux@galinux-VirtualBox:~/myprog/gaminganywhere-0.8.0/bin$ ./ga-client config/client.abs.conf rtsp://localhost:8554/desktop
# [11121] 1536686981.765890 # include: config/common/controller.conf
# [11121] 1536686981.766282 # include: config/common/video-x264.conf
# [11121] 1536686981.766318 # include: config/common/audio-lame.conf
# [11121] 1536686981.766402 # RTSP[config]: using 'udp' for RTP flows.
# [11121] 1536686981.766418 # RTSP[config]: controller port = 8555
# [11121] 1536686981.766421 # RTSP[config]: controller via 'udp' protocol.
# [11121] 1536686981.766462 # RTSP[config]: video-encoder = libx264 (libx264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
# [11121] 1536686981.766482 # RTSP[config]: audio-encoder = libmp3lame (libmp3lame MP3 (MPEG audio layer 3))
# [11121] 1536686981.766505 # RTSP[config]: video specific option: threads = auto
Remote server @ localhost[127.0.0.1]:8554
# [11121] 1536686982.006081 SDL: prefer opengl hardware renderer.
# [11121] 1536686982.006109 controller queue: initialized size=32708 (481 units)
watchdog: launched, waiting for audio/video frames ...
# [11121] 1536686982.006246 rtspclient: max tolerable video delay disabled.
*** SAVEFILE: YUV image saved to 'NULL'; timestamp saved to 'NULL'.
RTP reordering threshold = 300000
# [11121] 1536686982.006279 qos-measurement: initialized.
# [11121] 1536686982.006326 controller socket: socket address [127.0.0.1:8555]
# [11121] 1536686982.006337 controller client-thread started: tid=11127.
Opening connection to 127.0.0.1, port 8554...
...Connection to server failed: Connection refused
[URL:"rtsp://localhost:8554/desktop"]: Failed to get a SDP description: Connection to server failed: Connection refused
Connect to rtsp://localhost:8554/desktop failed.
# [11121] 1536686982.006570 qos-measurement: deinitialized.
[URL:"rtsp://localhost:8554/desktop"]: Closing the stream.
rtsp thread: no more rtsp clients
rtsp thread: terminated.

I am facing the same issue with Assaultcube. Were you able to connect the server and client successfully? Please share what you did to resolve this connection issue.

authors: kadirayk sab-sv

repo: chunying/gaminganywhere


text:
函数跳转之后怎么返回？

+u之后跳转函数成功，但是怎么返回呢？我看配置文件里没有相应的按键呢

@Tesseractrr

source: gharchive/issue

authors: Tesseractrr chxuan

repo: chxuan/cpp-mode

license: MIT

license_type: permissive

license_source: github-api


text:
进到vim里面头部颜色就变成这样，怎么回事？

@kirito6876 每次都是这样吗？可能是vim没有刷新的原因，正常情况下那个vimplus的logo图案不会显示在左边目录树上的。

@kirito6876

repo: chxuan/vimplus


text:
求助:YouCompleteme安装失败解压的问题   谢谢

YouCompleteMe插件开始没有安装成功，然后采取了下载压缩包重新安装的方法。但是解压的时候显示找不到压缩包，提示没有那个目录。一开始以为是压缩包没有下好，后来确定压缩包也没有问题。压缩包的路径是home下的下载。希望给予帮助，谢谢。

你好，安装包我确认过没有问题，你看这篇博客：http://www.cnblogs.com/highway-9/p/5984285.html，应该可以帮到你

非常感谢 正在尝试中

@chxuan 按照你的博客然后我找出了自己的问题时因为浏览器下载压缩包的路径不对 所以会出现刚刚的那种情况，然后现在能进入了，但是安装的时候会一直卡住不走，按道理已经将压缩包下载下来后安装不没道理那么慢啊。不知道是不是正常情况。
实际安装情况
Your C++ compiler supports C++11, compiling in that mode.
-- Found PythonLibs: /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.6")
Downloading Clang 3.9.0
-- [download 100% complete]
-- [download 0% complete]
-- [download 1% complete]
重新安装YouCompleteMe-- [download 2% complete]

我刚看了你的博客，提示如果手动安装YouCompleteMe的话需要把vimplus下的vimrc文件中该项注释掉，不然会重新下载。我一开始时直接安装的，后来YouCompleteMe插件没有安装成功现在手动安装这个插件把它注释掉然后就是上面的安装过程，请问一下这样有问题么？

authors: chxuan tianxieao


text:
Error when compiling with IVY enabled

When compiling my app with IVY enabled, it throws below error:
  0% compiling
Compiling ng2-ckeditor : main as commonjs
chunk {0} runtime-es2015.10a1f01eef199006286d.js (runtime) 1.41 kB [entry] [rendered]
chunk {1} main-es2015.601938459e47ad4810a2.js (main) 128 bytes [initial] [rendered]
chunk {2} polyfills-es2015.57a4297f1bb8472f293f.js (polyfills) 130 bytes [initial] [rendered]
chunk {3} styles.09e193d08f365ca57de0.css (styles) 197 kB [initial] [rendered]
Date: 2019-08-04T23:20:28.090Z - Hash: 8e9af5da311b34546887 - Time: 6779ms

ERROR in SourceMapGenerator.prototype.applySourceMap requires either an explicit source file, or the source map's "file" property. Both were omitted.

Environment:
Angular CLI: 8.3.0-next.0
Node: 10.16.0
Angular: 9.0.0-next.0
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... platform-server, router, service-worker

Package                                    Version
--------------------------------------------------------------------
@angular-devkit/architect                  0.803.0-next.0
@angular-devkit/build-angular              0.803.0-next.0
@angular-devkit/build-optimizer            0.803.0-next.0
@angular-devkit/build-webpack              0.803.0-next.0
@angular-devkit/core                       8.3.0-next.0
@angular-devkit/schematics                 8.3.0-next.0
@angular/cdk                               8.1.2
@angular/cli                               8.3.0-next.0
@angular/http                              8.0.0-beta.10
@ngtools/webpack                           8.3.0-next.0
@schematics/angular                        8.3.0-next.0
@schematics/update                         0.803.0-next.0
rxjs                                       6.5.2
typescript                                 3.4.5
webpack                                    4.38.0

Test with version 1.2.6. It should works now.

license: mit

license_source: bigquery


text:
CI: v1.11: can't test program: BPF_PROG_TEST_RUN not supported

The build datapath test is broken on the v1.11 branch since at least v1.11.0. We probably didn't notice because it only runs when there are BPF changes.
--- FAIL: TestCt (0.02s)
    --- FAIL: TestCt/Nop_test (0.00s)
        prog_test.go:368: test failed: test run failed: can't test program: BPF_PROG_TEST_RUN not supported (requires >= v4.12)
    --- FAIL: TestCt/Map_test (0.00s)
        prog_test.go:375: test failed: test run failed: can't test program: BPF_PROG_TEST_RUN not supported (requires >= v4.12)
    --- FAIL: TestCt/RST_handling (0.00s)
        prog_test.go:382: test failed: test run failed: can't test program: BPF_PROG_TEST_RUN not supported (requires >= v4.12)
FAIL
FAIL	github.com/cilium/cilium/bpf/tests/prog_test	0.041s
FAIL

Example: https://github.com/cilium/cilium/pull/23667.
First noticed in: https://github.com/cilium/cilium/pull/23627.

Should be good now on all maintained releases (v1.13 requires a 4.19 kernel).

authors: julianwiedmann pchaigno

repo: cilium/cilium


text:
Cilium hard coded service map size

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
Cilium code base explicitly sets max service id as 65536 ( https://github.com/cilium/cilium/blob/main/pkg/service/const.go#L12 )
If cilium runs full mode ( kube-proxy replacement) then it creates couple of service entry in the map per kubernetes service. ( based on the number of machines, service type etc)
This limitation decreases total number of kubernetes service to around ~10-15k.. Even smaller if there are more machines.
This limit should be configurable as cilium should able to handle huge number of services.
Cilium Version
1.11.x
Kernel Version
4.9.0-14-amd64
Kubernetes Version
1.25.x
Sysdump
No response
Relevant log output
No response
Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

@youngnick @julianwiedmann any estimation for this issue?
seems it should be easy to make it configurable.

Hi @buggyhunter
What's the max number of services you are looking to deploy in your cluster? Do you know the approximate proportion of service types (e.g., clusterIP, node port)?
The code you've linked is the max number of service ID allocator. Have you tries bumping up the lb map size - https://github.com/cilium/cilium/blob/main/daemon/cmd/daemon_main.go#L1021?

@aditighag we want to have ~40-50K node port service count.
We need to extend cilium_lb4_services_v2
That map's size is fixed to ~65K which is not enough. Because for each nodeport, ~3-4 map entry is created so it ends up around ~10-15K node port size.
After limit we see following log in cilium:
level=error msg="Error while inserting service in LB map"
 error="Unable to allocate service ID 0 for {{10.8.48.102 {TCP 32894} 0} 0}: no service ID available"...

We are using cilium in IPAM ENI mode.
When we  execute cilium service list:
                                              6 => 10.9.47.37:32261
23104   100.67.101.97:30568    ClusterIP      1 => 10.9.40.60:30568
                                              2 => 10.9.37.123:30568
                                              3 => 10.9.56.38:30568
                                              4 => 10.9.45.20:30568
                                              5 => 10.9.46.96:30568
                                              6 => 10.9.47.37:30568
23105   0.0.0.0:30568          NodePort       1 => 10.9.40.60:30568
                                              2 => 10.9.37.123:30568
                                              3 => 10.9.56.38:30568
                                              4 => 10.9.45.20:30568
                                              5 => 10.9.46.96:30568
                                              6 => 10.9.47.37:30568
23106   10.9.52.130:30568      NodePort       1 => 10.9.40.60:30568

as you see for each port there are 3-4 entries in map.

level=error msg="Error while inserting service in LB map"
error="Unable to allocate service ID 0 for {{10.8.48.102 {TCP 32894} 0} 0}: no service ID available"...

You seem to be hitting the service ID allocation limit, it's a uint16 field at the moment (65k max). If you want to extend that limit, you would have to make similar changes as previously done for backend ID - https://github.com/cilium/cilium/pull/17235.

@aditighag Is changing limit here enough?
or does it require more changes?
If it requires more changes then we very appreciate if cilium team prioritize this issue and bring this capability.

@buggyhunter , based on @aditighag comments, the change to increase the 65K limits is vastly more complex than just changing a few constants.  An example of a similar type of change is the change she linked here: https://github.com/cilium/cilium/pull/17235/files
The other potential angle is to keep the service limit at 65K, but eliminate the multiple service-ids allocated per NodePort service.  There might be some quick-and-dirty changes that you could apply in a private branch to try and reduce the number of service-ids allocated (e.g., if you only care about NodePort connectivity and not clusterIP connectivity for that same service).   I think we're setting up a chat with your team, so we can talk more then.

Like @buggyhunter, we are also wanting to have a large number of nodePorts. In our use-case each nodePort we add allocates 8 Cilium serviceIDs limiting the cluster to only supporting about 8k services.
@danwent Did anything materialize from the off issue chat?
65521   23.111.147.42:17724     NodePort        1 => 240.2.9.24:11418 (active)
                                                2 => 240.2.9.47:11418 (active)
65522   23.111.147.42:17724/i   NodePort        1 => 240.2.2.102:11418 (active)
                                                2 => 240.2.9.24:11418 (active)
                                                3 => 240.2.9.47:11418 (active)
65523   10.10.0.47:17724        NodePort        1 => 240.2.9.24:11418 (active)
                                                2 => 240.2.9.47:11418 (active)
65524   10.10.0.47:17724/i      NodePort        1 => 240.2.2.102:11418 (active)
                                                2 => 240.2.9.24:11418 (active)
                                                3 => 240.2.9.47:11418 (active)
65525   240.0.3.72:17724        NodePort        1 => 240.2.9.24:11418 (active)
                                                2 => 240.2.9.47:11418 (active)
65526   240.0.3.72:17724/i      NodePort        1 => 240.2.2.102:11418 (active)
                                                2 => 240.2.9.24:11418 (active)
                                                3 => 240.2.9.47:11418 (active)
65527   0.0.0.0:17724           NodePort        1 => 240.2.9.24:11418 (active)
                                                2 => 240.2.9.47:11418 (active)
65528   0.0.0.0:17724/i         NodePort        1 => 240.2.2.102:11418 (active)
                                                2 => 240.2.9.24:11418 (active)
                                                3 => 240.2.9.47:11418 (active)

authors: aditighag buggyhunter danwent jhead-slg


text:
Coredns fails connecting to kube-api via kubernetes service

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
As initially reported here I'm unable to get dns working due to coredns failing to connect to kubernetes api - I think it might be a regression since upgrading to $latest.
Whole post follows as copy-paste:
Cluster information:
Kubernetes version: v1.28.1
Cloud being used: bare-metal
Installation method: kubeadm
Host OS: Ubuntu 22.04.3 LTS
CNI and version: cilium 1.14.1
CRI and version: containerd 1.6.22
Today, after upgrading to 1.28.1 I realized that my test cluster is unable to get coredns ready:
$ k get po -A | grep core
kube-system   coredns-5dd5756b68-hchqq            0/1     Running   0             57m
kube-system   coredns-5dd5756b68-r768b            0/1     Running   0             57m

Upon inspecting the logs there seem to be some connectivity issue between coredns and kube-api:
$ k -n kube-system logs coredns-5dd5756b68-hchqq | tail -5 | tail -2
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"

cilium connectivity test seem to run into the same issue:
$ cilium connectivity test
ℹ  Monitor aggregation detected, will skip some flow validation steps
⌛ [kubernetes] Waiting for deployment cilium-test/client to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/client2 to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/echo-same-node to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/echo-other-node to become ready...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/client-78f9dffc84-g5z5l to appear...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/client2-59b578d4bb-jttvw to appear...
⌛ [kubernetes] Waiting for pod cilium-test/client-78f9dffc84-g5z5l to reach DNS server on cilium-test/echo-same-node-54cc4f75b8-xt4cf pod...
⌛ [kubernetes] Waiting for pod cilium-test/client2-59b578d4bb-jttvw to reach DNS server on cilium-test/echo-same-node-54cc4f75b8-xt4cf pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-78f9dffc84-g5z5l to reach DNS server on cilium-test/echo-other-node-5b87f6f4f4-cdmtl pod...
⌛ [kubernetes] Waiting for pod cilium-test/client2-59b578d4bb-jttvw to reach DNS server on cilium-test/echo-other-node-5b87f6f4f4-cdmtl pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-78f9dffc84-g5z5l to reach default/kubernetes service...
connectivity test failed: timeout reached waiting for lookup for kubernetes.default from pod cilium-test/client-78f9dffc84-g5z5l to succeed (last error: context deadline exceeded)

Accessing the kube-api from outside the cluster works fine - as is demonstrated by kubectl working. :wink:
Cilium status seem ok.
$ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       OK
    \__/       ClusterMesh:        disabled

Deployment             hubble-relay       Desired: 1, Ready: 1/1, Available: 1/1
Deployment             hubble-ui          Desired: 1, Ready: 1/1, Available: 1/1
Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
DaemonSet              cilium             Desired: 4, Ready: 4/4, Available: 4/4
Containers:            cilium             Running: 4
                       hubble-relay       Running: 1
                       hubble-ui          Running: 1
                       cilium-operator    Running: 2
Cluster Pods:          8/8 managed by Cilium
Helm chart version:    1.14.1
Image versions         cilium             quay.io/cilium/cilium:v1.14.1@sha256:edc1d05ea1365c4a8f6ac6982247d5c145181704894bb698619c3827b6963a72: 4
                       hubble-relay       quay.io/cilium/hubble-relay:v1.13.2: 1
                       hubble-ui          quay.io/cilium/hubble-ui:v0.11.0@sha256:bcb369c47cada2d4257d63d3749f7f87c91dde32e010b223597306de95d1ecc8: 1
                       hubble-ui          quay.io/cilium/hubble-ui-backend:v0.11.0@sha256:14c04d11f78da5c363f88592abae8d2ecee3cbe009f443ef11df6ac5f692d839: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.14.1@sha256:e061de0a930534c7e3f8feda8330976367971238ccafff42659f104effd4b5f7: 2

There are no network policies I can find to blame.
$ k get ciliumnetworkpolicies.cilium.io -A
No resources found
$ k get networkpolicies.networking.k8s.io -A
No resources found

There are endpoints which I believe should be implicitly targeted by the service:
$ k get endpointslices.discovery.k8s.io
NAME         ADDRESSTYPE   PORTS   ENDPOINTS                                      AGE
kubernetes   IPv4          6443    192.168.100.10,192.168.100.11,192.168.100.12   140d
$ k get svc -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: "2023-09-01T08:00:11Z"
    labels:
      component: apiserver
      provider: kubernetes
    name: kubernetes
    namespace: default
    resourceVersion: "2726902"
    uid: 5e7c32c9-ab89-47e4-8940-db010c2ffc4d
  spec:
    clusterIP: 10.96.0.1
    clusterIPs:
    - 10.96.0.1
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: https
      port: 443
      protocol: TCP
      targetPort: 6443
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""

I don’t believe I have any funny business in the coredns config:
$ k get all -A -l k8s-app=kube-dns
NAMESPACE     NAME                           READY   STATUS    RESTARTS        AGE
kube-system   pod/coredns-5dd5756b68-hchqq   0/1     Running   1 (6m39s ago)   4h46m
kube-system   pod/coredns-5dd5756b68-r768b   0/1     Running   1 (6m38s ago)   4h46m

NAMESPACE     NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-system   service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4h46m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/2     2            0           4h46m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5dd5756b68   2         2         0       4h46m

$ k -n kube-system describe cm coredns
Name:         coredns
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
Corefile:
----
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

BinaryData
====

Events:  <none>

There is a service running in the container - but it does not seem to hold any data, probably due to not being able to connect to the api:
$ kubectl -n kube-system debug -it pod/coredns-5dd5756b68-hchqq --image=nicolaka/netshoot --target=coredns
coredns-5dd5756b68-hchqq  ~  ss -lnp | grep :53
udp   UNCONN 0      0                  *:53               *:*    users:(("coredns",pid=1,fd=12))
tcp   LISTEN 0      4096               *:53               *:*    users:(("coredns",pid=1,fd=11))
coredns-5dd5756b68-hchqq  ~  dig @localhost kubernetes.default

; <<>> DiG 9.18.13 <<>> @localhost kubernetes.default
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 29162
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 7fdf8d625c0b48eb (echoed)
;; QUESTION SECTION:
;kubernetes.default.            IN      A

;; Query time: 0 msec
;; SERVER: ::1#53(localhost) (UDP)
;; WHEN: Fri Sep 01 13:23:08 UTC 2023
;; MSG SIZE  rcvd: 59

I can access the api from the pod on the external ip, but not the service ip:
 coredns-5dd5756b68-hchqq  ~  ping kubernetes
ping: kubernetes: Try again

 coredns-5dd5756b68-hchqq  ~  ping k8s
PING k8s.kubenet (192.168.100.5) 56(84) bytes of data.
64 bytes from k8s.kubenet (192.168.100.5): icmp_seq=1 ttl=62 time=0.139 ms
64 bytes from k8s.kubenet (192.168.100.5): icmp_seq=2 ttl=62 time=0.147 ms
^C
--- k8s.kubenet ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1023ms
rtt min/avg/max/mdev = 0.139/0.143/0.147/0.004 ms

 coredns-5dd5756b68-hchqq  ~  curl -k https://k8s:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}#

 coredns-5dd5756b68-hchqq  ~  curl -k https://10.96.0.1:443
curl: (28) Failed to connect to 10.96.0.1 port 443 after 130812 ms: Couldn't connect to server

What am I missing?
Cilium Version
cilium-cli: v0.15.7 compiled with go1.21.0 on linux/amd64
cilium image (default): v1.14.1
cilium image (stable): v1.14.1
cilium image (running): 1.14.1
Kernel Version
6.4.11-200.fc38.x86_64
Kubernetes Version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1
Sysdump
cilium-sysdump-20230903-172137.zip
Relevant log output
$ k get po -A | grep core
kube-system   coredns-5dd5756b68-hchqq            0/1     Running   0             57m
kube-system   coredns-5dd5756b68-r768b            0/1     Running   0             57m

$ k -n kube-system logs coredns-5dd5756b68-hchqq | tail -5 | tail -2
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"

Anything else?
https://discuss.kubernetes.io/t/coredns-fails-connecting-to-kube-api-via-kubernetes-service/
Code of Conduct

[x] I agree to follow this project's Code of Conduct

Thanks for this detailed issue @azzid.
Unfortunately Cilium 1.14 only supports Kubernetes 1.27 - it looks like there might be something in the 1.28 upgrade that's not working properly.
Could you try either doing Cilium 1.14 on Kubernetes 1.28, or Cilium main (you could also use the 1.15.0-pre.0 tag) and Kubernetes 1.28, and seeing if you get the same issue?

Could you try either doing Cilium 1.14 on Kubernetes 1.28

Is there a miss in the spelling there? 1.14 on 1.28 is what I'm already failing at, right? ;-)

🤦 Yes, thanks, Cilium 1.14 on Kubernetes 1.27 is what I meant. Thanks!

Forced a downgrade (basically apt install -y kubeadm=1.27.5-00 kubelet=1.27.5-00 kubectl=1.27.5-00 && kubeadm upgrade apply v1.27.5 - might contain traces of 1.28) and rebooted the nodes:
$ k get no
NAME        STATUS   ROLES           AGE    VERSION
k8scp       Ready    control-plane   143d   v1.27.5
k8scp2      Ready    control-plane   143d   v1.27.5
k8scp3      Ready    control-plane   143d   v1.27.5
k8sworker   Ready    <none>          143d   v1.27.5

Thereafter coredns is running!
$ k get po -A | grep core
kube-system   coredns-5dd5756b68-k6bjt           1/1     Running   0                3m57s
kube-system   coredns-5dd5756b68-lsqcd           1/1     Running   1 (4m43s ago)    8m33s

Okay, so it's probably something about coredns and Cilium on 1.14 and 1.28, thanks!

Oh, to be clear, were you using Kubernetes 1.28 before version 1.14, or is this problem only with Cilium 1.14 and Kubernetes 1.28. Do you know what happens if it's 1.13 and Kube 1.28? (Trying to think about if this is only the Kubernetes version or if we changed something in 1.14 that causes it).

were you using Kubernetes 1.28 before version 1.14

That's unfortunately a bit unclear. I was using something older before doing the kubernetes upgrade - so after the nodes where 1.28 I upgraded Cilium to 1.14 - but I don't know:
A. if coredns worked before the cilium upgrade
B. which cilium version I upgraded from

Tried upgrading to 1.15 on 1.27:
$ helm upgrade cilium cilium/cilium --version 1.15.0-pre.0 --namespace=kube-system --set k8sServiceHost=k8s --set k8sServicePort=6443
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Mon Sep  4 10:46:09 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.15.0-pre.0.

For any further help, visit https://docs.cilium.io/en/v1.15/gettinghelp

$ kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '<none>'
 | awk '{print "-n "$1" "$2}' | xargs -L 1 -r kubectl delete pod
pod "client-6b4b857d98-5xmlw" deleted
pod "client2-646b88fb9b-tmc7w" deleted
pod "echo-other-node-8b4df78df-vhxbw" deleted
pod "echo-same-node-775456cfcf-9f6ll" deleted
pod "cowboy" deleted
pod "coredns-5dd5756b68-k6bjt" deleted
pod "coredns-5dd5756b68-lsqcd" deleted
pod "hubble-relay-585bb7f68-ggqfg" deleted
pod "hubble-ui-694cf76f4c-t8kx6" deleted

$ k delete ns cilium-test
namespace "cilium-test" deleted

$ cilium connectivity test
ℹ  Monitor aggregation detected, will skip some flow validation steps
✨ [kubernetes] Creating namespace cilium-test for connectivity check...
✨ [kubernetes] Deploying echo-same-node service...
✨ [kubernetes] Deploying DNS test server configmap...
✨ [kubernetes] Deploying same-node deployment...
✨ [kubernetes] Deploying client deployment...
✨ [kubernetes] Deploying client2 deployment...
✨ [kubernetes] Deploying echo-other-node service...
✨ [kubernetes] Deploying other-node deployment...
✨ [host-netns] Deploying kubernetes daemonset...
✨ [host-netns-non-cilium] Deploying kubernetes daemonset...
✨ [kubernetes] Deploying echo-external-node deployment...
⌛ [kubernetes] Waiting for deployment cilium-test/client to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/client2 to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/echo-same-node to become ready...
⌛ [kubernetes] Waiting for deployment cilium-test/echo-other-node to become ready...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/client-6b4b857d98-8wfxx to appear...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/client2-646b88fb9b-8v579 to appear...
⌛ [kubernetes] Waiting for pod cilium-test/client-6b4b857d98-8wfxx to reach DNS server on cilium-test/echo-same-node-695c975475-9ls6m pod...
⌛ [kubernetes] Waiting for pod cilium-test/client2-646b88fb9b-8v579 to reach DNS server on cilium-test/echo-same-node-695c975475-9ls6m pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-6b4b857d98-8wfxx to reach DNS server on cilium-test/echo-other-node-6f988f8697-p8hlz pod...
⌛ [kubernetes] Waiting for pod cilium-test/client2-646b88fb9b-8v579 to reach DNS server on cilium-test/echo-other-node-6f988f8697-p8hlz pod...
⌛ [kubernetes] Waiting for pod cilium-test/client-6b4b857d98-8wfxx to reach default/kubernetes service...
⌛ [kubernetes] Waiting for pod cilium-test/client2-646b88fb9b-8v579 to reach default/kubernetes service...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/echo-other-node-6f988f8697-p8hlz to appear...
⌛ [kubernetes] Waiting for CiliumEndpoint for pod cilium-test/echo-same-node-695c975475-9ls6m to appear...
⌛ [kubernetes] Waiting for Service cilium-test/echo-other-node to become ready...
⌛ [kubernetes] Waiting for Service cilium-test/echo-other-node to be synchronized by Cilium pod kube-system/cilium-qtqch
⌛ [kubernetes] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [kubernetes] Waiting for Service cilium-test/echo-same-node to be synchronized by Cilium pod kube-system/cilium-qtqch
⌛ [kubernetes] Waiting for NodePort 192.168.100.10:30370 (cilium-test/echo-other-node) to become ready...
connectivity test failed: timeout reached waiting for NodePort 192.168.100.10:30370 (cilium-test/echo-other-node) (last error: command terminated with exit code 1)

I've installed minikube with k8s 1.28.1 and Cilium 1.14.1 and the connectivity tests have passed:
$ minikube version
minikube version: v1.31.2
commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396e
$ minikube start --kubernetes-version=v1.28.1
$ cilium version
cilium-cli: v0.15.7 compiled with go1.21.0 on linux/amd64
cilium image (default): v1.14.1
cilium image (stable): v1.14.1
cilium image (running): 1.14.1
$ cilium install
🔮 Auto-detected Kubernetes kind: minikube
✨ Running "minikube" validation checks
✅ Detected minikube version "1.31.2"
ℹ  Using Cilium version 1.14.1
🔮 Auto-detected cluster name: minikube
🔮 Auto-detected kube-proxy has been installed
$ kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1

$ cilium connectivity test
ℹ  Single-node environment detected, enabling single-node connectivity test
ℹ  Monitor aggregation detected, will skip some flow validation steps
✨ [minikube] Creating namespace cilium-test for connectivity check...
✨ [minikube] Deploying echo-same-node service...
✨ [minikube] Deploying DNS test server configmap...
✨ [minikube] Deploying same-node deployment...
✨ [minikube] Deploying client deployment...
✨ [minikube] Deploying client2 deployment...
⌛ [minikube] Waiting for deployment cilium-test/client to become ready...
⌛ [minikube] Waiting for deployment cilium-test/client2 to become ready...
⌛ [minikube] Waiting for deployment cilium-test/echo-same-node to become ready...
⌛ [minikube] Waiting for CiliumEndpoint for pod cilium-test/client-78f9dffc84-887bg to appear...
⌛ [minikube] Waiting for CiliumEndpoint for pod cilium-test/client2-59b578d4bb-8kld9 to appear...
⌛ [minikube] Waiting for pod cilium-test/client-78f9dffc84-887bg to reach DNS server on cilium-test/echo-same-node-79d996cb79-k7p6p pod...
⌛ [minikube] Waiting for pod cilium-test/client2-59b578d4bb-8kld9 to reach DNS server on cilium-test/echo-same-node-79d996cb79-k7p6p pod...
⌛ [minikube] Waiting for pod cilium-test/client-78f9dffc84-887bg to reach default/kubernetes service...
⌛ [minikube] Waiting for pod cilium-test/client2-59b578d4bb-8kld9 to reach default/kubernetes service...
⌛ [minikube] Waiting for CiliumEndpoint for pod cilium-test/echo-same-node-79d996cb79-k7p6p to appear...
⌛ [minikube] Waiting for Service cilium-test/echo-same-node to become ready...
⌛ [minikube] Waiting for Service cilium-test/echo-same-node to be synchronized by Cilium pod kube-system/cilium-sgtdt
⌛ [minikube] Waiting for NodePort 192.168.49.2:30363 (cilium-test/echo-same-node) to become ready...
ℹ  Skipping IPCache check
🔭 Enabling Hubble telescope...
⚠  Unable to contact Hubble Relay, disabling Hubble telescope and flow validation: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [::1]:4245: connect: connection refused"
ℹ  Expose Relay locally with:
   cilium hubble enable
   cilium hubble port-forward&
ℹ  Cilium version: 1.14.1
🏃 Running tests...
[=] Test [no-policies]
..................
[=] Test [no-policies-extra]
..
[=] Test [allow-all-except-world]
........
[=] Test [client-ingress]
..
[=] Test [client-ingress-knp]
..
[=] Test [allow-all-with-metrics-check]
..
[=] Test [all-ingress-deny]
......
[=] Test [all-ingress-deny-knp]
......
[=] Test [all-egress-deny]
........
[=] Test [all-egress-deny-knp]
........
[=] Test [all-entities-deny]
......
[=] Test [cluster-entity]
..
[=] Test [host-entity]
..
[=] Test [echo-ingress]
..
[=] Test [echo-ingress-knp]
..
[=] Test [client-ingress-icmp]
..
[=] Test [client-egress]
..
[=] Test [client-egress-knp]
..
[=] Test [client-egress-expression]
..
[=] Test [client-egress-expression-knp]
..
[=] Test [client-with-service-account-egress-to-echo]
..
[=] Test [client-egress-to-echo-service-account]
..
[=] Test [to-entities-world]
......
[=] Test [to-cidr-external]
....
[=] Test [to-cidr-external-knp]
....
[=] Test [echo-ingress-from-other-client-deny]
....
[=] Test [client-ingress-from-other-client-icmp-deny]
....
[=] Test [client-egress-to-echo-deny]
....
[=] Test [client-ingress-to-echo-named-port-deny]
..
[=] Test [client-egress-to-echo-expression-deny]
..
[=] Test [client-with-service-account-egress-to-echo-deny]
..
[=] Test [client-egress-to-echo-service-account-deny]
.
[=] Test [client-egress-to-cidr-deny]
....
[=] Test [client-egress-to-cidr-deny-default]
....
[=] Test [health]
.
[=] Skipping Test [north-south-loadbalancing] (feature node-without-cilium is disabled)
[=] Skipping Test [pod-to-pod-encryption] (feature encryption-pod is disabled)
[=] Skipping Test [node-to-node-encryption] (feature encryption-pod is disabled)
[=] Skipping Test [egress-gateway-excluded-cidrs] (feature enable-ipv4-egress-gateway is disabled)
[=] Skipping Test [north-south-loadbalancing-with-l7-policy] (feature node-without-cilium is disabled)
[=] Test [echo-ingress-l7]
......
[=] Test [echo-ingress-l7-named-port]
......
[=] Test [client-egress-l7-method]
......
[=] Test [client-egress-l7]
........
[=] Test [client-egress-l7-named-port]
........
[=] Skipping Test [client-egress-l7-tls-deny-without-headers] (feature secret-backend-k8s is disabled)
[=] Skipping Test [client-egress-l7-tls-headers] (feature secret-backend-k8s is disabled)
[=] Skipping Test [client-egress-l7-set-header] (feature secret-backend-k8s is disabled)
[=] Skipping Test [echo-ingress-auth-always-fail] (feature mutual-auth-spiffe is disabled)
[=] Skipping Test [echo-ingress-mutual-auth-spiffe] (feature mutual-auth-spiffe is disabled)
[=] Skipping Test [pod-to-ingress-service] (feature ingress-controller is disabled)
[=] Skipping Test [pod-to-ingress-service-deny-all] (feature ingress-controller is disabled)
[=] Skipping Test [pod-to-ingress-service-allow-ingress-identity] (feature ingress-controller is disabled)
[=] Test [dns-only]
........
[=] Test [to-fqdns]
........

✅ All 42 tests (182 actions) successful, 13 tests skipped, 1 scenarios skipped.

Since yesterday my version soup looks like:
$ cilium version
cilium-cli: v0.15.7 compiled with go1.21.0 on linux/amd64
cilium image (default): v1.14.1
cilium image (stable): v1.14.1
cilium image (running): 1.15.0-pre.0
$ ssh k8scp kubectl version --short
Client Version: v1.27.5
Kustomize Version: v5.0.1
Server Version: v1.27.5

Everything looks a bit broken
$ k get po -A | grep -e cilium -e core
kube-system   cilium-4kb4q                       0/1     Init:Error              57 (6m13s ago)   22h
kube-system   cilium-operator-6f99cbd84f-56l9s   0/1     CrashLoopBackOff        69 (2m4s ago)    22h
kube-system   cilium-operator-6f99cbd84f-nclll   0/1     CrashLoopBackOff        72 (55s ago)     22h
kube-system   cilium-t2b8z                       0/1     Init:0/6                56 (5m52s ago)   22h
kube-system   cilium-tcjbc                       0/1     Init:Error              57 (6m21s ago)   22h
kube-system   cilium-wtw2k                       0/1     Init:CrashLoopBackOff   57 (19s ago)     22h
kube-system   coredns-5dd5756b68-52czp           0/1     Unknown                 0                26h
kube-system   coredns-5dd5756b68-pl5bd           0/1     Unknown                 1                26h

not only coredns, but cilium as well is having trouble with the kube-api service
$ k -n kube-system logs cilium-operator-6f99cbd84f-56l9s | grep -e error -e fatal
level=error msg="Unable to contact k8s api-server" error="Get \"https://10.96.0.1:443/api/v1/namespaces/kube-system\": dial tcp 10.96.0.1:443: i/o timeout" ipAddr="https://10.96.0.1:443" subsys=k8s-client
level=error msg="Start hook failed" error="Get \"https://10.96.0.1:443/api/v1/namespaces/kube-system\": dial tcp 10.96.0.1:443: i/o timeout" function="client.(*compositeClientset).onStart" subsys=hive
level=fatal msg="failed to start: Get \"https://10.96.0.1:443/api/v1/namespaces/kube-system\": dial tcp 10.96.0.1:443: i/o timeout" subsys=cilium-operator-generic

Since it works in minikube I guess something has gone awry in my configuration during up/downgrades.
How does one go about flushing whatever configuration might be left behind since the earlier attempts? I'd prefer to get a grasp on what's going on rather than restarting with kubeadm reset.
I've found cilium cleanup in the docs, but I stumble on using it.
$ cilium cleanup
unknown command "cleanup" for "cilium"

I tried running it in a cilium-agent pod yesterday, but while the cilium command there was aware of it it refused to do anything as the cilium service was running.

I have exactly the same problem. Pods are failing to connect to apiserver - 10.255.100.1 service is not working. Direct connection to apiserver ip's is working. The problem occured after upgrdaing from kubernetes 1.27.4 to 1.28.1. Using latest cilium git code (1.15 branch) doesn't help either.

Starting to investigate the problem :) So I have cluster with 4x nodes. First 3 nodes are control plane. I've created test daemonset and trying to launch: wget https://10.225.100.1 . And it works with no problem on node 4. It seems that only control plane nodes are affected....

I managed to get v1.14.1 working again. Seems the --set global.kubeProxyReplacement=true option to helm works poorly.
I did a helm pull and tweaked the values.yaml like so:
$ diff -u cilium/values.yaml{.distOrig,}
--- cilium/values.yaml.distOrig 2023-08-15 13:00:00.000000000 +0200
+++ cilium/values.yaml  2023-09-06 09:37:47.091867691 +0200
@@ -36,9 +36,9 @@
 # @default -- `"~/.kube/config"`
 kubeConfigPath: ""
 # -- (string) Kubernetes service host
-k8sServiceHost: ""
+k8sServiceHost: "k8s"
 # -- (string) Kubernetes service port
-k8sServicePort: ""
+k8sServicePort: "6443"

 # -- Configure the client side rate limit for the agent and operator
 #
@@ -137,7 +137,7 @@
 name: cilium

 # -- Roll out cilium agent pods automatically when configmap is updated.
-rollOutCiliumPods: false
+rollOutCiliumPods: true

 # -- Agent container image.
 image:
@@ -526,7 +526,7 @@
   # if you're removing Cilium from the cluster. Disable this to prevent the CNI
   # configuration file from being removed during agent upgrade, which can cause
   # nodes to go unmanageable.
-  uninstall: false
+  uninstall: true

   # -- Configure chaining on top of other CNI plugins. Possible values:
   #  - none
@@ -1575,7 +1575,7 @@
 # -- Configure the kube-proxy replacement in Cilium BPF datapath
 # Valid options are "true", "false", "disabled" (deprecated), "partial" (deprecated), "strict" (deprecated).
 # ref: https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/
-#kubeProxyReplacement: "false"
+kubeProxyReplacement: "true"

 # -- healthz server bind address for the kube-proxy replacement.
 # To enable set the value to '0.0.0.0:10256' for all ipv4

which got the configmap right from the start:
$ k -n kube-system describe cm cilium-config | grep -A2 kube-proxy-replacement:
kube-proxy-replacement:
----
true

also got all the pods in working shape
$ k get po -A | grep -e cilium -e core
kube-system   cilium-dtxqg                       1/1     Running   0             3m47s
kube-system   cilium-lh9t4                       1/1     Running   0             3m47s
kube-system   cilium-operator-8487fbb8d7-dxz9p   1/1     Running   0             3m47s
kube-system   cilium-operator-8487fbb8d7-wvwt5   1/1     Running   0             3m47s
kube-system   cilium-wxvzm                       1/1     Running   0             3m47s
kube-system   cilium-zkn9k                       1/1     Running   0             3m47s
kube-system   coredns-5dd5756b68-mt8wr           1/1     Running   0             3m9s
kube-system   coredns-5dd5756b68-rlxvg           1/1     Running   0             3m9s

@azzid can you clarify which steps did you use for the working scenario and steps did you use for the non-working scenario? Thank you

@azzid can you clarify which steps did you use for the working scenario and steps did you use for the non-working scenario?

I'll try to reproduce again. What should trigger the problem is upgrading to 1.28.
I have however come to realize that I'm using the now deprecated google repos - after switching to pkgs.k8s.io I see that there are new versions not only for kubeadm, kubectl and kubelet, but also kubernets-cni and cri-tools. Perhaps the kubernetes-cni version impacts this.
cri-tools/unknown 1.28.0-1.1 amd64 [upgradable from: 1.26.0-00]
kubeadm/unknown 1.28.1-1.1 amd64 [upgradable from: 1.27.5-00]
kubectl/unknown 1.28.1-1.1 amd64 [upgradable from: 1.27.5-00]
kubelet/unknown 1.28.1-1.1 amd64 [upgradable from: 1.27.5-00]
kubernetes-cni/unknown 1.2.0-2.1 amd64 [upgradable from: 1.2.0-00]

I think I have a reproducer using a development Kind cluster.
But here's a wrinkle, I can make it happen with both K8s 1.27.3 and 1.28.0 and Cilium 1.14.1 by just restarting the Kind containers node images, no upgrade needed.
Here's the gist with 3 different terminal sessions:
https://gist.github.com/jspaleta/6f3f5bdadc83790be978999ff9693082
Three scenarios are included:

Kind using k8s 1.27.3 nodes no Cilium :: Result: CoreDNS working after restarting the nodes
Kind using k8s 1.27.3 nodes with Cilium : Result CoreDNS issue after restarting the nodes
Kind using k8s 1.28.0 nodes with Cilium: Result CoreDNS issue after restarting the nodes.

All i needed to do to create the CoreDNS problem was to restart the Kind node containers.
I don't know if its a race or what, but its effectively a reboot of the cluster nodes similar to the clush command in the previous comment.

Thanks. Managed to reproduce it locally with @jspaleta 's way:
./contrib/scripts/kind.sh "" 2 "" "kindest/node:v1.28.0" "none" "ipv4"                                                                                cilium install
docker restart kind-control-plane kind-worker kind-worker2

Important to note that Cilium's kube-proxy replacement is installed.
A relevant bit from the pwru output (pwru --output-tuple 'host $CORE_DNS_POD_IP'):
0xffff918e963e0ee8     19        [<empty>]             ip_local_out 10.244.1.143:58040->10.96.0.1:443(tcp)
0xffff918e963e0ee8     19        [<empty>]           __ip_local_out 10.244.1.143:58040->10.96.0.1:443(tcp)
0xffff918e963e0ee8     19        [<empty>]                ip_output 10.244.1.143:58040->10.96.0.1:443(tcp)

It suggests that the socket LB xlation didn't happen. The BPF LB map is properly populated on the core DNS pod's node:
root@kind-worker:/home/cilium# cilium bpf lb list
SERVICE ADDRESS     BACKEND ADDRESS (REVNAT_ID) (SLOT)
10.96.0.1:443       172.18.0.4:6443 (3) (1)
                    0.0.0.0:0 (3) (0) [ClusterIP, non-routable]
...

Next suspect is the cgroup BPF progs (used by the socket LB) attached to the wrong cgroup root. The core DNS is in the following cgroup:
> sudo cat /proc/$(ps aux | grep coredns | head -n1 | awk '{print $2}')/cgroup
0::/system.slice/docker-51314633cf3e7e7c1d6265978d19f0c0d456b02c4d74dcde9a4ff2e7107e1fc0.scope/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod1e780c35_26eb_4b9a_ada8_ea32bc0f680b.slice/cri-containerd-41e621d81a680dd398e029ddb737c294be4505d22e5af6891d03a50911cf01a3.scope

Meanwhile bpftool cgroup tree gives:
/sys/fs/cgroup/system.slice/docker-51314633cf3e7e7c1d6265978d19f0c0d456b02c4d74dcde9a4ff2e7107e1fc0.scope/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod2472f208_e443_4e02_8300_ecd005f91ff2.slice/cri-containerd-ec66d8070ee0b7ee00f512316d7811220de2f996d29c70557fcfe8e46576ad6e.scope
    12117    cgroup_device   multi
    12777    cgroup_inet4_connect multi           cil_sock4_connect
    12778    cgroup_inet6_connect multi           cil_sock6_connect
    12783    cgroup_inet4_post_bind multi           cil_sock4_post_bind
    12776    cgroup_inet6_post_bind multi           cil_sock6_post_bind
    12782    cgroup_udp4_sendmsg multi           cil_sock4_sendmsg
    12785    cgroup_udp6_sendmsg multi           cil_sock6_sendmsg
    12779    cgroup_udp4_recvmsg multi           cil_sock4_recvmsg
    12784    cgroup_udp6_recvmsg multi           cil_sock6_recvmsg
    12780    cgroup_inet4_getpeername multi           cil_sock4_getpeername
    12781    cgroup_inet6_getpeername multi           cil_sock6_getpeername

Indeed, the socket LB programs attached to the wrong cgroup common root, so the service xlation is not taking place = no connectivity to the kube-apiserver.

For the completion, the working case (before the restart):
> sudo cat /proc/$(ps aux | grep coredns | head -n1 | awk '{print $2}')/cgroup
0::/system.slice/docker-23f180831ee5c872d2b35dfe438c8f1d0422052a363c3dd8c8525dbbe104d60f.scope/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-podf74f6eac_62dc_4d07_b484_a0eca44f9418.slice/cri-containerd-5fc07b1a4bac35ceb9f48b4cf069cbf052a41f4001a5c73debcdd176de546c86.scope

> sudo bpftool cgroup tree
...
/sys/fs/cgroup/system.slice/docker-23f180831ee5c872d2b35dfe438c8f1d0422052a363c3dd8c8525dbbe104d60f.scope
    13052    cgroup_device   multi
    13837    cgroup_inet4_connect multi           cil_sock4_connect
    13839    cgroup_inet6_connect multi           cil_sock6_connect
    13845    cgroup_inet4_post_bind multi           cil_sock4_post_bind
    13838    cgroup_inet6_post_bind multi           cil_sock6_post_bind
    13846    cgroup_udp4_sendmsg multi           cil_sock4_sendmsg
    13842    cgroup_udp6_sendmsg multi           cil_sock6_sendmsg
    13841    cgroup_udp4_recvmsg multi           cil_sock4_recvmsg
    13844    cgroup_udp6_recvmsg multi           cil_sock6_recvmsg
    13843    cgroup_inet4_getpeername multi           cil_sock4_getpeername
    13840    cgroup_inet6_getpeername multi           cil_sock6_getpeername

@azzid @kvaster Could you re-run my experiment to see whether you are hitting the same issue?

Cc @ti-mo wrt cgroup links

@brb, will re-run experiment in approx 4 hours! Thanks!

Trying to confirm my reproducer is related to #27848 by installing Cilium 1.13.6 and then restarting all the Kind nodes..and it still looks like that fails for me...
So that's a bit of a head-scratcher as commentary in the other issue seems to correlate with the group attachment findings from @brb here
Reproducer procedure:

spin up multi-node kind node with kubeproxy and cni disabled and version of k8s you want I've tried 1.27 and 1.28
install cilium versions I've tried: 1.13.6,1.14.1, 1.15.0-pre.0
confirm normal operation with running coredns
docker restart all the kind node containers
confirm coredns pods never come back up into a ready state and logs for those pods indicate
Kubernetes API connection failure

@azzid @kvaster Could you re-run my experiment to see whether you are hitting the same issue?

root@k8scp:~# cat /proc/$(ps aux | grep coredns | head -n1 | awk '{print $2}')/cgroup
0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6bc9ed_fe53_410f_bd20_f54e59f0c048.slice/cri-containerd-43d307d27e57ee95b7b05eaec4ff4aba66f49df09ba9db4c6bcbfb71adc6e066.scope
root@k8scp:~# bpftool cgroup tree | tail -2
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6bc9ed_fe53_410f_bd20_f54e59f0c048.slice/cri-containerd-43d307d27e57ee95b7b05eaec4ff4aba66f49df09ba9db4c6bcbfb71adc6e066.scope
    453      device          multi
root@k8scp:~#

Not used to reading this type of output - is it correct to say that the 0::-prefix is the problem?

For me:
k8s-1 /home/kva # sudo cat /proc/$(ps aux | grep core | head -n1 | awk '{print $2}')/cgroup
0::/kubepods/besteffort/pod79e1b178-95b5-4dc6-9d5b-1ad4ee64d190/b20a6bf61417d0747653b1a6137d618c3432ce4b6f2520b4b761639af84a82f7

and
k8s-1 /home/kva # bpftool cgroup tree
CgroupPath
ID       AttachType      AttachFlags     Name
/sys/fs/cgroup
386      cgroup_inet4_connect multi           cil_sock4_connect
391      cgroup_inet6_connect multi           cil_sock6_connect
385      cgroup_inet4_post_bind multi           cil_sock4_post_bind
390      cgroup_inet6_post_bind multi           cil_sock6_post_bind
392      cgroup_udp4_sendmsg multi           cil_sock4_sendmsg
384      cgroup_udp6_sendmsg multi           cil_sock6_sendmsg
389      cgroup_udp4_recvmsg multi           cil_sock4_recvmsg
388      cgroup_udp6_recvmsg multi           cil_sock6_recvmsg
387      cgroup_inet4_getpeername multi           cil_sock4_getpeername
383      cgroup_inet6_getpeername multi           cil_sock6_getpeername
...
/sys/fs/cgroup/kubepods/besteffort/pod79e1b178-95b5-4dc6-9d5b-1ad4ee64d190/b20a6bf61417d0747653b1a6137d618c3432ce4b6f2520b4b761639af84a82f7
    633      cgroup_device   multi

From my setup:
k8s-1 /home/kva # sudo cat /proc/$(ps aux | grep core | head -n1 | awk '{print $2}')/cgroup
0::/kubepods/besteffort/pod79e1b178-95b5-4dc6-9d5b-1ad4ee64d190/e7d1027117911c63fef8582d28e1c544b729c470d378b7e7d9fbc9bf9fbf6c00

and
k8s-1 /home/kva # bpftool cgroup tree
CgroupPath
ID       AttachType      AttachFlags     Name
/sys/fs/cgroup/kubepods/burstable/podf21937e2-50b5-452c-8a62-90d313eee5b4/19619a967ee0e5b1f2da717839139e4f925576c28645ba8b29872419000f1961
    52       cgroup_device   multi
    368      cgroup_inet4_connect multi           cil_sock4_connect
    374      cgroup_inet6_connect multi           cil_sock6_connect
    372      cgroup_inet4_post_bind multi           cil_sock4_post_bind
    370      cgroup_inet6_post_bind multi           cil_sock6_post_bind
    373      cgroup_udp4_sendmsg multi           cil_sock4_sendmsg
    371      cgroup_udp6_sendmsg multi           cil_sock6_sendmsg
    377      cgroup_udp4_recvmsg multi           cil_sock4_recvmsg
    375      cgroup_udp6_recvmsg multi           cil_sock6_recvmsg
    369      cgroup_inet4_getpeername multi           cil_sock4_getpeername
    376      cgroup_inet6_getpeername multi           cil_sock6_getpeername
/sys/fs/cgroup/kubepods/burstable/podf21937e2-50b5-452c-8a62-90d313eee5b4/2dc2d678b10f7fbcb549b6206ec8f08960d92d77189ca97a2643ed3589257736
    43       cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/podfeef4e54045161bcbf29022bb161242c/090af00fc865ed08fe561c0f7d3ead7e2e350169e0d8b40c4aac801949d86458
    7        cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/podfeef4e54045161bcbf29022bb161242c/5519ee689f1cf05e78c01bde889d4242595ef1bbe8af13f9d02c21ea7d6b10df
    25       cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pod43dbcbd1f898c59a29d5e21b81bc9c30/1b0f2a8a84d0526f33f6f92f40f4a466ab2338f93c80eb5bbabdd9e0bebdb042
    19       cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pod43dbcbd1f898c59a29d5e21b81bc9c30/ba173fe87a20af3b50a7a4f0ea0668b1cd4be232ad837429c854169d4677af5d
    4        cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pod49149134978c7c5a6df38d3a5cdde5b3/e6a4414d90fce7b9b5547cbb1bec8a846fd9f086b7b3f8ec3e48fd56814d1cce
    10       cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pod49149134978c7c5a6df38d3a5cdde5b3/6e626dadafe9ad9c6a350b724fcc647a069bb438af83df373458d9e2e1130303
    22       cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pode26b2f696e1f0489bc7d852a98b7f4df/28a51e57f88eed3e96f919ff9333c51a0ef944f9a812d49556bd325ed08ff3da
    13       cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pode26b2f696e1f0489bc7d852a98b7f4df/1a501e704db7c21814541abf82372b07d7c26d02fd6f240642a6506bb2b4fcd5
    16       cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pod8dd29228-6a55-4d65-b725-de7a4f19bac9/28d6fa00a07c9b8114d6ad51ca550db04afa1fe0e84d913e7b3a9005bcc47652
    680      cgroup_device   multi
/sys/fs/cgroup/kubepods/burstable/pod8dd29228-6a55-4d65-b725-de7a4f19bac9/6cd2afab895631095ba78fe2349f5e7c9288893a0103ec6a75c661d0880d7f7e
    677      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podac594ac5-234d-42dd-ab99-2198ad3094fd/6e35b002c58080684385aa36fa36747f7b784ef7a2c8c3edcc71b333e1a143fd
    46       cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podac594ac5-234d-42dd-ab99-2198ad3094fd/3c4d14c9d97ebea48df6b336efe629edbaf3a488c8c200987768479b10576985
    34       cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod37f448da-134d-499a-9c7b-616c8d130e2b/0a9fde75026dbdebac006dfe53144d3b4234eb414a803dd9e0e48254cee075b2
    636      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod37f448da-134d-499a-9c7b-616c8d130e2b/7f7de1b279e1b9137c8a378a818413f4175bc1ddcfbc9cae8ca492e53237df5b
    615      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod37f448da-134d-499a-9c7b-616c8d130e2b/ce20fe2bdd48e66e488a5113a5cf34c1200614501f3c0359a95bbbc5ae4b856f
    591      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod1380c6e9-a9cd-43f6-9f97-16133da790d4/eeadfe119ec76c957338bde13f63603fbc4ae96e658ef480b625ec7c2375c22c
    612      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod1380c6e9-a9cd-43f6-9f97-16133da790d4/15799589c788138ee842ee824910c3163a69ffe0aa3369432e770458f2d6e96e
    563      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod4afde13c-558e-4847-9a49-a10ac97302da/23bdc088884367a8377526f733503baff4d7de6a07f1f4f84a57b21571bdc557
    560      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod4afde13c-558e-4847-9a49-a10ac97302da/ba475cc88303d98145fa4d150cc7e08c2e8c5c258879902e002fbea15e9ce2f1
    630      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod54eb5888-5033-456b-97f6-5b48e3f9422a/ef07d24abf0799a31c04df4f0ba3f8379eb0a6100c3a2e84341ffa8440b78cb6
    40       cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod54eb5888-5033-456b-97f6-5b48e3f9422a/65e025ebc593d256d5fb778f30cc614f126e6f7dab6acaf010bc5b26c04e3be1
    37       cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podee38f93a-7450-4607-8dab-aabe14e05216/d66e83ba864716cf8c5bf0be00df84297da37e7f571613a77cdaaf9482ea8540
    683      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podee38f93a-7450-4607-8dab-aabe14e05216/c0c5df035a9a91aeffbbe9fbd3061d134b968cad7844f0e12d3dcd8af6e379c4
    633      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podee38f93a-7450-4607-8dab-aabe14e05216/f849eb40bce769c7022e842d35052c40f280d213698ca224c628b1f249e2f79a
    535      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podee38f93a-7450-4607-8dab-aabe14e05216/ecde1b0ec37b2f00f28ca87e7f8aea340e16b696d26e114e742b1ca51294ce0d
    621      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podb264fcda-0685-46b9-bd7f-c9f41aed28dc/d03800378d3a43ed80b59d0f10c76dcff9b0b28cca3144c59f74b7f17f2de37a
    600      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podb264fcda-0685-46b9-bd7f-c9f41aed28dc/c3b68367701731c6c15d8fae1d37d9cb661a6d58d3090c78c0c4e90200bbfdbe
    518      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod79e1b178-95b5-4dc6-9d5b-1ad4ee64d190/b30ee92e68fcc0334f26dda81a0668232162d4ab677d417e50af75bb8a9180d6
    645      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod79e1b178-95b5-4dc6-9d5b-1ad4ee64d190/e7d1027117911c63fef8582d28e1c544b729c470d378b7e7d9fbc9bf9fbf6c00
    603      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod79e1b178-95b5-4dc6-9d5b-1ad4ee64d190/d3d5bdb50a044da963f8b65e366580dd1a2af8bf6497b76012ee8ac5bf407c83
    663      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod79e1b178-95b5-4dc6-9d5b-1ad4ee64d190/6a21ab1101a53ceb8d9b6b6d5fcbefa486795361897e6c2e3ea72a58ef150b64
    532      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod385f667b-7c87-440d-8d23-22304a35094b/b790c6df37f14722ce4d06b741636643824e60004ff131fb7fd638759c327c59
    515      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod385f667b-7c87-440d-8d23-22304a35094b/22a4400832d1eb7b9711e0020e2163311501ac5855db9ecfbaafb74089dabc98
    597      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod434bbb56-8a33-47c7-9ac2-71be0bb8f2c8/e6c91edb92a86a27c5905d6bce2078b1f2aa4675c8d4ad2718acaa6e42a227f1
    512      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod434bbb56-8a33-47c7-9ac2-71be0bb8f2c8/ccea10d9940dcb43fc2e0bd72389fa2a36b4acf52ded0181666b690d6af9ef75
    594      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod434bbb56-8a33-47c7-9ac2-71be0bb8f2c8/b2770b637dd712a9764b9b53d74fedd0900b4e0879a01514778f127a02b5d7f2
    624      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod434bbb56-8a33-47c7-9ac2-71be0bb8f2c8/3a4c3e9c19dd31cfa59b3b4fce37a4abe83f0b9f18a66068cb0c67e720c00dcb
    639      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podd72fabe0-f4bf-49e6-bd5d-709c0b5cbaa8/d22acdb1caf7579e610da7d76394e135af0fdd6340336774adc2dfe835231198
    588      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podd72fabe0-f4bf-49e6-bd5d-709c0b5cbaa8/d13d7d17bcdcafca2e705cdbf18b0e65c53c246dab5a354c9366b8290cfb7bf1
    648      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/podd72fabe0-f4bf-49e6-bd5d-709c0b5cbaa8/292ad4b76ead8ec0f6411a09b2794255c19d69bc44515366058a72447e8e94ab
    642      cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod2fe53cea-1e60-459b-be5a-420790364cdc/06688e8343e5f7b6877932700177f0b41c3236c61d6d143b7c997815d62757d9
    28       cgroup_device   multi
/sys/fs/cgroup/kubepods/besteffort/pod2fe53cea-1e60-459b-be5a-420790364cdc/0f4c4ae05f01ad6c9fe496afcd8da173fc00ee366f3e70435e08f16c5055a35f
    31       cgroup_device   multi

@azzid There seems to be a regression introduced on k8s v1.28 that involves app containers getting started before init containers - https://github.com/kubernetes/kubernetes/issues/120247.
If you are running with cgroup v2 and cilium socket-lb config enabled, the mount-cgroup init container must run prior to the agent. This ensures that the BPF cgroup programs are attached at the right hierarchy. See the kube-proxy replacement guide 1 and 2 for more details.
As an alternative/workaround, you can mount the cgroup2 fs on the host, see this note in the guide -
Cilium will automatically mount cgroup v2 filesystem required to attach BPF cgroup programs by default at the path /run/cilium/cgroupv2. To do that, it needs to mount the host /proc inside an init container launched by the DaemonSet temporarily. If you need to disable the auto-mount, specify --set cgroup.autoMount.enabled=false, and set the host mount point where cgroup v2 filesystem is already mounted by using --set cgroup.hostRoot. For example, if not already mounted, you can mount cgroup v2 filesystem by running the below command on the host, and specify --set cgroup.hostRoot=/sys/fs/cgroup.
...

/cc @aojea Which k8s versions will the regression fix be available in?

My reproducer is probably an entirely different issue than initially reported, same symptoms but high likelihood a different problem..maybe even a Kind specific problem.
Should I file a new issue to track that?
I'm going to retest now using my local k3s home lab and avoid the complication of cluster node as containers sharing a common kernel.

@azzid There seems to be a regression introduced on k8s v1.28 that involves app containers getting started before init containers - kubernetes/kubernetes#120247.
If you are running with cgroup v2 and cilium socket-lb config enabled, the mount-cgroup init container must run prior to the agent. This ensures that the BPF cgroup programs are attached at the right hierarchy. See the kube-proxy replacement guide 1 and 2 for more details.
As an alternative/workaround, you can mount the cgroup2 fs on the host, see this note in the guide -
Cilium will automatically mount cgroup v2 filesystem required to attach BPF cgroup programs by default at the path /run/cilium/cgroupv2. To do that, it needs to mount the host /proc inside an init container launched by the DaemonSet temporarily. If you need to disable the auto-mount, specify --set cgroup.autoMount.enabled=false, and set the host mount point where cgroup v2 filesystem is already mounted by using --set cgroup.hostRoot. For example, if not already mounted, you can mount cgroup v2 filesystem by running the below command on the host, and specify --set cgroup.hostRoot=/sys/fs/cgroup.
...

/cc @aojea Which k8s versions will the regression fix be available in?

I'm not entirely sure I understand what workaround you're proposing.
My nodes are running crgroup v2:
$ clush -w @k8s stat -fc %T /sys/fs/cgroup/
k8scp: cgroup2fs
k8scp3: cgroup2fs
k8sworker: cgroup2fs
k8scp2: cgroup2fs

And they all have a cgroup2 mountpoint:
$ clush -w @k8s 'mount | grep /sys/fs/cgroup'
k8scp: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
k8scp2: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
k8scp3: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
k8sworker: cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

I can't find any cgroup.hostRoot in the cilium-config.
There is however a cgroup-root, which seem to point to something good on one of the control planes - but not the others.
$ k -n kube-system get cm cilium-config -o yaml | grep -i cgroup
  cgroup-root: /run/cilium/cgroupv2
$ clush -w @k8s ls -l /run/cilium/cgroupv2
k8scp: total 0
k8scp: -r--r--r--  1 root root 0 Sep  6 15:51 cgroup.controllers
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 cgroup.max.depth
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 cgroup.max.descendants
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:51 cgroup.procs
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 cgroup.stat
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:51 cgroup.subtree_control
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 cgroup.threads
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 cpu.pressure
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 cpuset.cpus.effective
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 cpuset.mems.effective
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 cpu.stat
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 dev-hugepages.mount
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 dev-mqueue.mount
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:51 init.scope
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 io.cost.model
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 io.cost.qos
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 io.pressure
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 io.prio.class
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 io.stat
k8scp: drwxr-xr-x  4 root root 0 Sep  6 15:52 kubepods.slice
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 memory.numa_stat
k8scp: -rw-r--r--  1 root root 0 Sep  6 15:52 memory.pressure
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 memory.stat
k8scp: -r--r--r--  1 root root 0 Sep  6 15:52 misc.capacity
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 proc-fs-nfsd.mount
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 proc-sys-fs-binfmt_misc.mount
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 sys-fs-fuse-connections.mount
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 sys-kernel-config.mount
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 sys-kernel-debug.mount
k8scp: drwxr-xr-x  2 root root 0 Sep  6 15:52 sys-kernel-tracing.mount
k8scp: drwxr-xr-x 51 root root 0 Sep 11 07:44 system.slice
k8scp: drwxr-xr-x  3 root root 0 Sep  6 15:58 user.slice
k8scp2: total 0
k8scp3: total 0
k8sworker: total 0

Tried upgrading to 1.28.2 today:
$ k get no
NAME        STATUS   ROLES           AGE    VERSION
k8scp       Ready    control-plane   153d   v1.28.2
k8scp2      Ready    control-plane   153d   v1.28.2
k8scp3      Ready    control-plane   153d   v1.28.2
k8sworker   Ready    <none>          153d   v1.28.2

Upgrading alone does not seem to fix the dns:
$ k get po -A -l k8s-app=kube-dns -o wide
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-wxfwm   0/1     Running   0          5m45s   10.0.1.124   k8scp2      <none>           <none>
kube-system   coredns-5dd5756b68-zbt2c   0/1     Running   0          5m45s   10.0.0.39    k8sworker   <none>           <none>

Neither does deleting the old pods:
$ k delete po -A -l k8s-app=kube-dns
pod "coredns-5dd5756b68-wxfwm" deleted
pod "coredns-5dd5756b68-zbt2c" deleted
$ k get po -A -l k8s-app=kube-dns -o wide
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-fkcv5   0/1     Running   0          66s   10.0.3.145   k8scp3   <none>           <none>
kube-system   coredns-5dd5756b68-z5skz   0/1     Running   0          66s   10.0.2.127   k8scp    <none>           <none>

But a cluster reboot after the upgrade seem to get everything in shape:
$ clush -w @k8s,lb reboot
$ sleep 120
$ k get po -A -l k8s-app=kube-dns -o wide
NAMESPACE     NAME                       READY   STATUS    RESTARTS       AGE     IP           NODE        NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-fkcv5   1/1     Running   1 (104s ago)   5m28s   10.0.3.170   k8scp3      <none>           <none>
kube-system   coredns-5dd5756b68-hjws8   1/1     Running   0              24s     10.0.0.68    k8sworker   <none>           <none>

Tried upgrading to 1.28.2 today:
$ k get no
NAME        STATUS   ROLES           AGE    VERSION
k8scp       Ready    control-plane   153d   v1.28.2
k8scp2      Ready    control-plane   153d   v1.28.2
k8scp3      Ready    control-plane   153d   v1.28.2
k8sworker   Ready    <none>          153d   v1.28.2

Upgrading alone does not seem to fix the dns:
$ k get po -A -l k8s-app=kube-dns -o wide
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE     IP           NODE        NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-wxfwm   0/1     Running   0          5m45s   10.0.1.124   k8scp2      <none>           <none>
kube-system   coredns-5dd5756b68-zbt2c   0/1     Running   0          5m45s   10.0.0.39    k8sworker   <none>           <none>

Neither does deleting the old pods:
$ k delete po -A -l k8s-app=kube-dns
pod "coredns-5dd5756b68-wxfwm" deleted
pod "coredns-5dd5756b68-zbt2c" deleted
$ k get po -A -l k8s-app=kube-dns -o wide
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-fkcv5   0/1     Running   0          66s   10.0.3.145   k8scp3   <none>           <none>
kube-system   coredns-5dd5756b68-z5skz   0/1     Running   0          66s   10.0.2.127   k8scp    <none>           <none>

But a cluster reboot after the upgrade seem to get everything in shape:
$ clush -w @k8s,lb reboot
$ sleep 120
$ k get po -A -l k8s-app=kube-dns -o wide
NAMESPACE     NAME                       READY   STATUS    RESTARTS       AGE     IP           NODE        NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-fkcv5   1/1     Running   1 (104s ago)   5m28s   10.0.3.170   k8scp3      <none>           <none>
kube-system   coredns-5dd5756b68-hjws8   1/1     Running   0              24s     10.0.0.68    k8sworker   <none>           <none>

$ cilium connectivity test
(...)
✅ All 42 tests (306 actions) successful, 13 tests skipped, 0 scenarios skipped.

For completion, the bpf-stuff post upgrade:
root@k8scp3:~# cat /proc/$(ps aux | grep coredns | head -n1 | awk '{print $2}')/cgroup
0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbef6a2_439a_4e3d_b861_c07dc2f2e567.slice/cri-containerd-8c0b79f8288484ac45ea2c35fe5c66eee446fe6bd2ef24086c90ab632994f9c1.scope

root@k8scp3:~# bpftool cgroup tree | grep -A1 8c0b79f8288484ac45ea2c35fe5c66eee446fe6bd2ef240
/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbef6a2_439a_4e3d_b861_c07dc2f2e567.slice/cri-containerd-8c0b79f8288484ac45ea2c35fe5c66eee446fe6bd2ef24086c90ab632994f9c1.scope
    466      device          multi

Thanks  @azzid that you tested it. Had the same problem being discussed here on a bare metal cluster of two nodes and also after rebooting the control plane everything stopped. I updated Kubernetes to 1.28.2 then rebooted and all back to normal.

So what is the solution here? I'm receiving the same issue on 1.27.6 and my podcidr is 192.168.0.0/16. coredns gives me the same output as well. Is the solution to just update kubeadm?

I'm still experiencing the same...
RHEL 9
Kubernetes 1.28.8
Cillium 1.15.3
CoreDNS 1.10.0

@kvaster I ended up moving to Calico after couple of days of frustration ... 😢

We observed such behaviour on kubernetes 1.28.8 aswell. Upgrading to kubernetes 1.29.x fixed it.

@kvaster Do you mean a wrong cgroup root for the bpf_sock attachment?

might be unrelated, but had exact error i/o timeout to kubeapi endpoint in coredns logs and all pods not able to communicate outside the cluster. I have suspected nat issue, but could not see anything wrong. I suspected something with ipv6 so I've disabled it completely via cmdline, then I noticed coredns error changed that it was not able to communicate to ipv6 address i.e. cluster had dual stack (tho I didn't set it up). I solved my network error with:
net.ipv6.conf.all.forwarding=1

environment:
kube v1.29.4
fedora40
kubeadm init with pod-network-cidr=172.20.0.0/16, service-cidr=172.30.0.0/16 using crio socket
crio cni
:wave:  @\brb

IS there a resolution eventually for the above issue?
I have exactly the same issue when upgrading from k8s .127 to 1.28 with kubespray and cilium chart v1.14 and kubeproxy replacement enabled.
Upgrading cilium to 1.15 doesnt change anything neither.

In my case, with K3s 1.29 and Cilium 1.15, enabling the host firewall might be causing this issue.

I ran into this issue when using legacy masquerading mode (Masquerading:           IPTables [IPv4: Enabled, IPv6: Disabled]) and specifying and egress interface that didn't exist on my nodes: egressMasqueradeInterfaces: eth0. I solved it by switching to bpf masquerading mode: bpf.masquerade: true (Masquerading:            BPF   [enp71s0, pod-id-link0]   100.76.0.0/24 [IPv4: Enabled, IPv6: Disabled])
(https://github.com/cilium/cilium/blob/7007334438cd7a0dc446df46c0d22442a8fb2312/install/kubernetes/cilium/values.yaml#L559)

Reached to get a good state on any kubernetes and cilium version by enabling nftables and disabling it just after
I don't know how it could have an impact on cilium but it seems related to a conflict with firewall.

same issue here, switching to calico helped 😢
kube 1.30.3, cilium 1.62.2, coredns 1.1.3

Is there any consistent reasoning for this? I'm also having the same problem.
Running crio, cilium for L3 and istio for L7 would be nice. Guess I may have to switch to calico as well?

For what its worth, I was suffering from this exact same issue and discovered that it wasn't a Cilium issue.
My API server pods were advertising the wrong IP addresses that weren't routable from the other nodes. Look at the endpoint resources to see what addresses are backing the Kubernetes API service. I ended up having to change the --advertise-address flag in /etc/kubernetes/manifests/kube-apiserver.yaml to the one I intended to use.

Other pods can call the kube-apiserver through the service IP, but I don't understand why only the CoreDNS pod cannot. My Kubernetes version is 1.30, and I am using Cilium version 1.16.9.
My K8s is on GCE.

authors: T0MASD aanm aditighag alephpt azzid borkmann brb danpe gaffneyd4 jspaleta khalidsafwat kolovo kotyara85 kvaster loic-roux-404 longlx8 nueavv ryantking unixbird youngnick


text:
KNP: Egress default deny all does not work for world.

List of endpoints:
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])               IPv6                 IPv4            STATUS
           ENFORCEMENT        ENFORCEMENT
3365       Disabled           Disabled          37299      k8s:id=app3                               f00d::a0f:0:0:d25    10.15.191.0     ready
                                                           k8s:io.kubernetes.pod.namespace=default
11004      Disabled           Disabled          60284      k8s:id=app1                               f00d::a0f:0:0:2afc   10.15.54.125    ready
                                                           k8s:io.kubernetes.pod.namespace=default
29898      Disabled           Disabled          39640      reserved:health                           f00d::a0f:0:0:74ca   10.15.242.54    ready
42910      Disabled           Disabled          55717      k8s:id=app2                               f00d::a0f:0:0:a79e   10.15.236.254   ready
                                                           k8s:io.kubernetes.pod.namespace=default
49508      Disabled           Disabled          60284      k8s:id=app1                               f00d::a0f:0:0:c164   10.15.114.197   ready
                                                           k8s:io.kubernetes.pod.namespace=default

Try to access world from pod
Manalis-MBP:cilium manalibhutiyani$ kubectl exec app2 sh -- -c "curl -s google.com"
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE>

301 Moved

The document has moved
here.

Works as expected.
Applying egress deny all
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
#for k8s <1.7 use:
#apiVersion: extensions/v1beta1
metadata:
  name: access-backend
spec:
  podSelector: {}
  policyTypes:
  - Egress

State of cilium endpoints:
root@minikube:~# cilium endpoint list
ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])               IPv6                 IPv4            STATUS
           ENFORCEMENT        ENFORCEMENT
3365       Disabled           Enabled           37299      k8s:id=app3                               f00d::a0f:0:0:d25    10.15.191.0     ready
                                                           k8s:io.kubernetes.pod.namespace=default
11004      Disabled           Enabled           60284      k8s:id=app1                               f00d::a0f:0:0:2afc   10.15.54.125    ready
                                                           k8s:io.kubernetes.pod.namespace=default
29898      Disabled           Disabled          39640      reserved:health                           f00d::a0f:0:0:74ca   10.15.242.54    ready
42910      Disabled           Enabled           55717      k8s:id=app2                               f00d::a0f:0:0:a79e   10.15.236.254   ready
                                                           k8s:io.kubernetes.pod.namespace=default
49508      Disabled           Enabled           60284      k8s:id=app1                               f00d::a0f:0:0:c164   10.15.114.197   ready
                                                           k8s:io.kubernetes.pod.namespace=default
root@minikube:~#

Try to access world again:
 Manalis-MBP:cilium manalibhutiyani$ kubectl exec app2 sh -- -c "curl -s google.com"
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE>

301 Moved

 The document has moved
 here.

  Manalis-MBP:cilium manalibhutiyani$

Also tried it with
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "deny-all-egress"
spec:
  endpointSelector:
  egress:
  - {}

Does not work as expected


text:
Allow for non default runtime socket path

Please ensure your pull request adheres to the following guidelines:

[x] For first time contributors, read Submitting a pull request
[x] All code is covered by unit and/or runtime tests where feasible.
[x] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[x] All commits are signed off. See the section Developer’s Certificate of Origin
[x] Provide a title or release-note blurb suitable for the release notes.
[x] Thanks for contributing!

global.containerRuntime.socketPath is used to set a non default
runtime socket path. This property should not be ignored as it was
working on v1.6. This PR ensures that if a socket path is set and
the runtime is not set to none the socket path is set in the
daemonset manifest.
Fixes: #10544
Allow for non default runtime socket path in k8s installation

This change is

Coverage decreased (-0.01%) to 45.627% when pulling e3a750e8523d6c210927069ef374e4f65115d90e on ktsakalozos:bug/socketpath into 6d29bba62d7cef9d340848d360da2effc09f2485 on cilium:master.

Ignoring the socketpath was not the real problem.

source: gharchive/pull-request

authors: coveralls ktsakalozos


text:
docs: Add SmileDirectClub to Cilium users

Please ensure your pull request adheres to the following guidelines:

[x] For first time contributors, read Submitting a pull request
[x] All code is covered by unit and/or runtime tests where feasible.
[x] All commits contain a well written commit description including a title,
description and a Fixes: #XXX line if the commit addresses a particular
GitHub issue.
[x] All commits are signed off. See the section Developer’s Certificate of Origin
[x] Provide a title or release-note blurb suitable for the release notes.
[x] Thanks for contributing!

Add SmileDirectClub to the list of users in the "Who is using Cilium?"
doc, along with contact info.

Coverage decreased (-0.02%) to 36.63% when pulling 02986754f45790a433c8a3d9086b233ec19339ef on particledecay:add_smiledirectclub into ac9732a29a85c97fa4a5b4a7033ca07fef81b0e1 on cilium:master.

authors: coveralls particledecay


text:
Spiffe Integration

What is this?
This PR contains the Cilium-SPIFFE integration that has been proposed in this design document. This PR covers the "Identity Generation Hardening" part of such proposal. We already have some prototypes of the "Upgrading connections to mTLS" part but we prefer start discussing only the former one to keep it more scoped.
How to test?
If you want to test it by yourself you can follow these steps. Otherwise you can watch this recorded demo.

Download, compile and install cilium-agent based on this PR
Add the --enable-spiffe flag to cilium-agent
Deploy Spire. I prepared a yaml manifest with all components needed to deploy it. It's based on the spire k8s quickstart tutorial but uses a custom spire-agent image that implements a POC of the spire privileged API.

$ kubectl apply -f https://gist.githubusercontent.com/rscampos/1e1e537c5183ccc32f2f64a949f37f41/raw/c5543cd286be7dda5d3940a09129378465de0dac/spire.yaml

Check that spire pods are running

$ kubectl get pods -n spire
NAME                READY   STATUS    RESTARTS   AGE
spire-agent-rqsp9   1/1     Running   0          14s
spire-server-0      1/1     Running   0          14s

Create some registration entries. This creates some registration entries for testing.

$ ./create_registration_entries.sh

Create some pods

$ kubectl run podubuntu --image=ubuntu:latest -- sleep 1000000
$ kubectl run podpraqma --image=praqma/network-multitool:latest -- sleep 100000

Restrict all egress traffic

$ kubectl apply -f https://gist.githubusercontent.com/rscampos/1e1e537c5183ccc32f2f64a949f37f41/raw/c5543cd286be7dda5d3940a09129378465de0dac/deny-all-egress.yaml

Check that ping is failing

$ kubectl exec podpraqma -- ping <podubuntu IP>

Deploy image based policy that allows those two pods to talk to each other

$ kubectl apply -f https://gist.githubusercontent.com/rscampos/1e1e537c5183ccc32f2f64a949f37f41/raw/c5543cd286be7dda5d3940a09129378465de0dac/image-based.yaml

Ping now works

$ kubectl exec podpraqma -- ping <podubuntu IP>
PING 10.11.0.173 (10.11.0.173) 56(84) bytes of data.
64 bytes from 10.11.0.173: icmp_seq=1 ttl=63 time=0.124 ms

TODO

[x] Update code once Spire "privileged" PR is ready (https://github.com/spiffe/spire/issues/2246 & https://github.com/spiffe/spire-api-sdk/pull/8 & https://github.com/spiffe/spire/pull/2481)
[x] Improve label validation (undo validation relax done for this POC)
[x] Rebase once https://github.com/cilium/cilium/pull/15123 is merged
[x] Rebase once https://github.com/cilium/cilium/pull/16585 is merged
[x] Spiffe label format - use spiffe as source in Spiffe label - discussed here #16626
[ ] Tests (both unit and integration)
[ ] Documentation
[ ] How to deploy spire. Create helm templates?, leave it up to the user?
[ ] How to initialize the spiffe watcher in a lazy way (avoid trying to connect to it when spire is not deployed yet)

ref #4016
old PR #16626
cc @nyrahul @jrajahalme @jrfastab @navarrothiago @mauriciovasquezbernal

Folks,
Just sent a commit updating to the last version of Delegated Identity API. Also, this commit introduces the single watch approach: for each endpoint,  one stream between Cilium and Spire is created.

@rscampos can you rebase against master? I assume this PR can be marked for review if it's in draft people will tend to ignore it unless it is marked as ready :-)

hey @nathanjsweet, just rebased it :D

I've been looking forward to this integration. Just curious if you have any estimate on when this could get merged?

Hey @aanm !
Since this is tagged as high priority, I apologise for duplicating @HighWatersDev's message but is it still a high priority?
Do you have any news to share that would go in this PR direction or another initiative toward Spiffe integration?
I believe we're numerous to look for this feature (as highlighted in the recent cilium blog post).

@thylong, regarding the initiative toward Spiffe integration, take a look at https://github.com/accuknox/cilium-spire-tutorials if you want to try it out. Maybe this could help to understand a little bit more about the integration.

Thanks for your insights @navarrothiago , definitely appreciated !
I'll take a look at it 😊

Is the Spiffe integration occurring anywhere else? Been looking forward to this going through.

Hey folks, what is missing for the integration here? Happy to figure out how to help and move this forward.

Let me put forward the status here based on what I know:

The changes to SPIRE SDK and other spire accessory tooling that was necessary is in place and upstreamed. (the links are in the main comment).
The changes in this PR were done during 1.9.x rev and the code base has substantially changed since then. However, the links to the document/design should still hold.
Note that It would still be a substantial effort to handle the changes and upstream it.

If anyone wants to take this forward, here would be the steps:

Reproduce the setup following this tutorial.
Check with the cilium community if there is any other effort underway
Rebase the code from this PR and fix the conflicts. This step is the most imp and non-trivial. If the cilium code base has changed significantly or any of its earlier design principles in the core have changed then this could be a major rework. This is the primary analysis that has to be done to get this integration done based on this PR.
Reproduce the setup from step 1 with the code from step 3. If all goes well till this point, most of the work will be in place. You might need to add more tests and get the reviews from the community.

Even though some of the folks developing this PR have now moved on to do something else, I am hoping I should be able to check and pull some of them together in a discussion/call if anyone is interested to take this forward.

authors: HighWatersDev aanm elinesterov navarrothiago nyrahul rscampos securitysamurai thylong


text:
vagrant: bump all Vagrant box versions

Mostly to pick up Go 1.17
Supersedes #17352 which used the wrong branch name, see https://github.com/cilium/cilium/pull/17352#issuecomment-919257321 Otherwise no changes to the PR contents.

test-me-please

test-1.16-netnext

test-1.16-netnext

Looks like this is timing out on waiting for the kubelet to become ready: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-net-next/1445/ I think this is also what happened on the previous run and on the previous PR #17352.
I've tried to reproduce this locally but so far wasn't able to (i.e. the VMs always come up as expected with the new VM image).

test-me-please

test-1.16-netnext

Looks like this is timing out on waiting for the kubelet to become ready: https://jenkins.cilium.io/job/Cilium-PR-K8s-1.16-net-next/1445/ I think this is also what happened on the previous run and on the previous PR #17352.
I've tried to reproduce this locally but so far wasn't able to (i.e. the VMs always come up as expected with the new VM image).

I was using the dev VM instructions instead of the CI VM instructions (thanks @nbusseneau for pointing this out). Using
K8S_VERSION=1.16 K8S_NODES=1 NETNEXT=1 ./vagrant-local-start.sh I was able to reproduce the VM provisioning failure. It looks like kubelet doesn't come up:
vagrant@k8s1:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-09-16 13:21:21 UTC; 86ms ago
     Docs: https://kubernetes.io/docs/home/
  Process: 11042 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 11042 (code=exited, status=255)
vagrant@k8s1:~$ journalctl -xeu kubelet
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is RESULT.
Sep 16 13:21:52 k8s1 kubelet[11246]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:21:52 k8s1 kubelet[11246]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:21:52 k8s1 kubelet[11246]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:21:52 k8s1 kubelet[11246]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:21:52 k8s1 kubelet[11246]: I0916 13:21:52.841662   11246 server.go:410] Version: v1.16.15
Sep 16 13:21:52 k8s1 kubelet[11246]: I0916 13:21:52.842505   11246 plugins.go:100] No cloud provider specified.
Sep 16 13:21:52 k8s1 kubelet[11246]: I0916 13:21:52.842575   11246 server.go:773] Client rotation is on, will bootstrap in background
Sep 16 13:21:52 k8s1 kubelet[11246]: I0916 13:21:52.853385   11246 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 16 13:21:53 k8s1 kubelet[11246]: I0916 13:21:53.069853   11246 server.go:636] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Sep 16 13:21:53 k8s1 kubelet[11246]: F0916 13:21:53.070344   11246 server.go:271] failed to run Kubelet: strconv.Atoi: parsing "1\n\x00": invalid syntax
Sep 16 13:21:53 k8s1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 16 13:21:53 k8s1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 16 13:22:03 k8s1 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 16 13:22:03 k8s1 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 103.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit kubelet.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Sep 16 13:22:03 k8s1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit kubelet.service has finished shutting down.
Sep 16 13:22:03 k8s1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is RESULT.
Sep 16 13:22:03 k8s1 kubelet[11313]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:22:03 k8s1 kubelet[11313]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:22:03 k8s1 kubelet[11313]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:22:03 k8s1 kubelet[11313]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:22:03 k8s1 kubelet[11313]: I0916 13:22:03.399365   11313 server.go:410] Version: v1.16.15
Sep 16 13:22:03 k8s1 kubelet[11313]: I0916 13:22:03.400017   11313 plugins.go:100] No cloud provider specified.
Sep 16 13:22:03 k8s1 kubelet[11313]: I0916 13:22:03.400067   11313 server.go:773] Client rotation is on, will bootstrap in background
Sep 16 13:22:03 k8s1 kubelet[11313]: I0916 13:22:03.404339   11313 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 16 13:22:03 k8s1 kubelet[11313]: I0916 13:22:03.606099   11313 server.go:636] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Sep 16 13:22:03 k8s1 kubelet[11313]: F0916 13:22:03.607174   11313 server.go:271] failed to run Kubelet: strconv.Atoi: parsing "1\n\x00": invalid syntax
Sep 16 13:22:03 k8s1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 16 13:22:03 k8s1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 16 13:22:13 k8s1 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 16 13:22:13 k8s1 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 104.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit kubelet.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Sep 16 13:22:13 k8s1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit kubelet.service has finished shutting down.
Sep 16 13:22:13 k8s1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is RESULT.
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See htt
Sep 16 13:22:13 k8s1 kubelet[11384]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See http
Sep 16 13:22:13 k8s1 kubelet[11384]: I0916 13:22:13.995746   11384 server.go:410] Version: v1.16.15
Sep 16 13:22:13 k8s1 kubelet[11384]: I0916 13:22:13.996551   11384 plugins.go:100] No cloud provider specified.
Sep 16 13:22:13 k8s1 kubelet[11384]: I0916 13:22:13.996614   11384 server.go:773] Client rotation is on, will bootstrap in background
Sep 16 13:22:14 k8s1 kubelet[11384]: I0916 13:22:14.009873   11384 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 16 13:22:14 k8s1 kubelet[11384]: I0916 13:22:14.234620   11384 server.go:636] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Sep 16 13:22:14 k8s1 kubelet[11384]: F0916 13:22:14.235267   11384 server.go:271] failed to run Kubelet: strconv.Atoi: parsing "1\n\x00": invalid syntax
Sep 16 13:22:14 k8s1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 16 13:22:14 k8s1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

The following looks like the culprit for failing to start kubelet:
Sep 16 13:22:14 k8s1 kubelet[11384]: F0916 13:22:14.235267   11384 server.go:271] failed to run Kubelet: strconv.Atoi: parsing "1\n\x00": invalid syntax

Not sure where that 1\n\x00 is coming from. Maybe command line of a config file? Will try to dig further...

Starting kubelet with --v=8 unfortunately doesn't give any further clues.
But it smells a bit like that value could come from a sysctl or sysfs file. And indeed, if I check for uses of strconv.Atoi in code around the last log messages I noticed this:
https://github.com/kubernetes/kubernetes/blob/2adc8d7091e89b6e3ca8d048140618ec89b39369/pkg/kubelet/cm/cpuset/cpuset.go#L258-L297
which gets called from here:
https://github.com/kubernetes/kubernetes/blob/2adc8d7091e89b6e3ca8d048140618ec89b39369/pkg/kubelet/cm/cpumanager/topology/topology.go#L310-L325
Comparing the content of that file for the net-next kernel :
vagrant@k8s1:~$ xxd /sys/devices/system/node/node0/cpulist
00000000: 302d 310a 00                             0-1..

with my development machine running kernel 5.11:
tklauser@orbital:~ $ xxd /sys/devices/system/node/node0/cpulist
00000000: 302d 370a                                0-7.

Notice the trailing \0 on the net-next kernel, the difference in the CPU numbers is expected. This looks like this is a kernel regression to me.

Notice the trailing null byte 00 in the output on the net-next kernel (the difference in the CPU numbers is expected as these are different systems). This looks like a kernel regression to me.

It's indeed a kernel regression which incidentally breaks kubectl. It looks like the regression was introduced by the series at https://lore.kernel.org/all/20210806110251.560-1-song.bao.hua@hisilicon.com/
I've sent a potential fix upstream: https://lore.kernel.org/lkml/20210916222705.13554-1-tklauser@distanz.ch/T/#u

I've sent a potential fix upstream: https://lore.kernel.org/lkml/20210916222705.13554-1-tklauser@distanz.ch/T/#u

Patch was applied to the driver-core tree and should hopefully hit Linus' tree for 5.15-rc3.
Marking this PR as draft until that is the case and bpf-next picked up the fix as well at which point we'd need to build new net-next VM images to pick up into this PR.

/test

/test

/test

/test

test-runtime

authors: aanm tklauser