text:
"File name too long" when mounting Lustre PV in AKS

│   Normal   Scheduled    2m48s                default-scheduler  Successfully assigned workbench/workbench-78cf48884-m7x56 to aks-ds2v2-38810676-vmss000 │
│ 000                                                                                │
│   Warning  FailedMount  45s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[rstudio-home-storage], unattached │
│  volumes=[etc-rstudio rstudio-user license-file rstudio-home-storage rstudio-pam rstudio-session-secret kube-api-access-zk688 rstudio-prestart rstudio- │
│ ser-startup rstudio-session-config rstudio-secret rstudio-launcher-startup rstudio-config rstudio-rsw-startup session-templates[]: timed out waiting fo │
│ r the condition                                                                                │
│   Warning  FailedMount  37s (x8 over 2m45s)  kubelet            MountVolume.SetUp failed for volume "pvc-fef8f1e6-66c9-47d2-9a53-f1a57336bbc3" : rpc er │
│ ror: code = Internal desc = Could not mount "10.1.0.6@tcp:/lustreAKS" at "/var/lib/kubelet/pods/81e87d1b-ac73-458a-90f2-1a548f142dc5/volumes/kubernetes │
│ .io~csi/pvc-fef8f1e6-66c9-47d2-9a53-f1a57336bbc3/mount": mount failed: exit status 36                                                                   │
│ Mounting command: mount                                                                                │
│ Mounting arguments: --no-mtab--no-mtab -t lustre -o noatime,flock 10.1.0.6@tcp:/lustreAKS /var/lib/kubelet/pods/81e87d1b-ac73-458a-90f2-1a548f142dc5/vo │
│ lumes/kubernetes.io~csi/pvc-fef8f1e6-66c9-47d2-9a53-f1a57336bbc3/mount                                                                                │
│ Output: mount.lustre: mount 10.1.0.6@tcp:/lustreAKS at /var/lib/kubelet/pods/81e87d1b-ac73-458a-90f2-1a548f142dc5/volumes/kubernetes.io~csi/pvc-fef8f1e │
│ 6-66c9-47d2-9a53-f1a57336bbc3/mount failed: File name too long

This also applies when shortening the PV name, i.e. the file name is still too long:
│ Output: mount.lustre: mount 10.1.0.6@tcp:/lustreAKS at /var/lib/kubelet/pods/e7028a7d-91d0-4070-9441-4e8f802cb134/volumes/kubernetes.io~csi/pv-wb-lustre/mount failed: file name too long │
│ e/mount failed: File name too long

What happened:
Error during PVC mount.
What you expected to happen:
Mount succeeds.
How to reproduce it:
Mount a dynamically created PVC using a Lustre SC as described in https://github.com/kubernetes-sigs/azurelustre-csi-driver/blob/main/docs/static-provisioning.md.
Anything else we need to know?:
Environment:

CSI Driver version: 0.1.8
Kubernetes version (use kubectl version): 1.25.5
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

Lustre FS name has an 8 character max limit.  If you are using AMLFS with AKS, the default fs-name is "lustrefs".  This is static FS name for AMLFS.

This CSI is only currently supported for Azure Managed Lustre, using the guide here: https://learn.microsoft.com/en-us/azure/azure-managed-lustre/use-csi-driver-kubernetes

authors: joe-atzinger lianger pat-s

repo: kubernetes-sigs/azurelustre-csi-driver


text:
Create unmanaged kubernetes cluster on CloudStack

Issue #, if available:
Description of changes:
Creates an unmanaged kubernetes cluster on Cloudstack. This provides a nice view of CAPC managed kubernetes cluster in CloudStack UI.
Depends on: https://github.com/apache/cloudstack/pull/7515 https://github.com/apache/cloudstack-go/pull/59 (go.mod needs to be updated once this is merged)
Testing performed:
Tested locally by creating a kubernetes cluster using cluster api.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hi @chrisdoherty4
This creates an entry for a kubernetes cluster in CloudStack. This helps provide a good view of CAPC managed clusters on CloudStack and the VMs associated with the cluster in the CloudStack UI. This PR is dependent on apache/cloudstack-go PR merge and release. So, it will be some time before this PR can be merged.

@vishesh92 So populating this value would bind VMs to an Apache CloudStack Cluster ('cluster' is hyperloaded in this context)?
Am I right in saying we expect users to have pre-created the cluster?

cluster has a different meaning when it comes to cloudstack infrastructure. In context of this PR, cluster == kubernetes cluster.
We are doing two operation when the CAPC managed kubernetes cluster comes up:

Create an entry in CloudStack's kubernetes dashboard. This is just a simple insert into db and cloudstack doesn't do anything else on this resource.
When the machine deployments get created or deleted, we link or de-link the VM created by CAPC on cloudstack with kubernetes cluster (on CloudStack) entry created earlier.

In short, we are just adding some information about the kubernetes cluster which is managed by CAPC on CloudStack.

ping @vishesh92 is this up to date?

Codecov Report
Patch coverage: 6.34% and project coverage change: -0.45% :warning:

Comparison is base (b860d9f) 24.95% compared to head (d640335) 24.51%.
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #250      +/-   ##
==========================================
- Coverage   24.95%   24.51%   -0.45%
==========================================
  Files          59       60       +1
  Lines        5558     5687     +129
==========================================
+ Hits         1387     1394       +7
- Misses       4035     4156     +121
- Partials      136      137       +1

Files Changed
Coverage Δ

api/v1beta3/cloudstackcluster_types.go
100.00% <ø> (ø)

controllers/cloudstackcluster_controller.go
16.66% <0.00%> (-4.63%)
:arrow_down:

controllers/cloudstackfailuredomain_controller.go
61.00% <ø> (ø)

pkg/cloud/client.go
48.29% <ø> (ø)

pkg/cloud/cluster.go
0.00% <0.00%> (ø)

controllers/cloudstackmachine_controller.go
54.87% <87.50%> (+0.56%)
:arrow_up:

pkg/cloud/isolated_network.go
58.89% <100.00%> (ø)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Tests were aborted.

Tests were aborted.

/run-e2e -c 4.18

@vishesh92 a jenkins job has been kicked to run test with following paramaters:

kubernetes version: 1.27.2
CloudStack version: 4.18
hypervisor: kvm
template: ubuntu-2004-kube
Kubernetes upgrade from: 1.26.5 to 1.27.2

Test Results : (tid-298)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.18
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr250-sl-298.zip
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
[PASS] When the specified resource does not exist Should fail due to the specified account is not found [TC4a]
[PASS] When the specified resource does not exist Should fail due to the specified domain is not found [TC4b]
[PASS] When the specified resource does not exist Should fail due to the specified zone is not found [TC3]
[PASS] When the specified resource does not exist Should fail due to the public IP can not be found

Summarizing 24 Failures:

[Fail] When testing machine remediation [It] Should replace a machine when it is destroyed
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing node drain timeout [It] A node should be forcefully removed if it cannot be drained in time
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing multiple CPs in a shared network with kubevip [It] Should successfully create a cluster with multiple CPs in a shared network
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing with custom disk offering [It] Should successfully create a cluster with a custom disk offering
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing app deployment to the workload cluster with network interruption [ToxiProxy] [It] Should be able to create a cluster despite a network interruption during that process
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing Kubernetes version upgrades [It] Should successfully upgrade kubernetes versions when there is a change in relevant fields
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing app deployment to the workload cluster with slow network [ToxiProxy] [BeforeEach] Should be able to download an HTML from the app deployed to the workload cluster
/jenkins/workspace/capc-e2e-new/test/e2e/toxiproxy/toxiProxy.go:203

[Fail] When testing affinity group [It] Should have host affinity group when affinity is pro
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing affinity group [It] Should have host affinity group when affinity is anti
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing subdomain [It] Should create a cluster in a subdomain
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing app deployment to the workload cluster [TC1][PR-Blocking] [It] Should be able to download an HTML from the app deployed to the workload cluster
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] with two clusters [It] should successfully add and remove a second cluster without breaking the first cluster
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] [It] Should successfully scale machine replicas up and down horizontally
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing with disk offering [It] Should successfully create a cluster with disk offering
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When the specified resource does not exist [It] Should fail due to the specified control plane offering is not found [TC7]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified template is not found [TC6]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified disk offering is not found
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the compute resources are not sufficient for the specified offering [TC8]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified disk offer is not customized but the disk size is specified
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified disk offer is customized but the disk size is not specified
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist When starting with a healthy cluster [BeforeEach] Should fail to upgrade worker machine due to insufficient compute resources
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When the specified resource does not exist When starting with a healthy cluster [BeforeEach] Should fail to upgrade control plane machine due to insufficient compute resources
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing resource cleanup [It] Should create a new network when the specified network does not exist
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing MachineDeployment rolling upgrades [It] Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

Ran 28 of 29 Specs in 7147.886 seconds
FAIL! -- 4 Passed | 24 Failed | 0 Pending | 1 Skipped
--- FAIL: TestE2E (7147.90s)
FAIL

/run-e2e -c 4.18

@vishesh92 a jenkins job has been kicked to run test with following paramaters:

kubernetes version: 1.27.2
CloudStack version: 4.18
hypervisor: kvm
template: ubuntu-2004-kube
Kubernetes upgrade from: 1.26.5 to 1.27.2

Test Results : (tid-299)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.18
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr250-sl-299.zip
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest

Summarizing 28 Failures:

[Fail] When testing node drain timeout [It] A node should be forcefully removed if it cannot be drained in time
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing MachineDeployment rolling upgrades [It] Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] [It] Should successfully scale machine replicas up and down horizontally
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing with custom disk offering [It] Should successfully create a cluster with a custom disk offering
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing multiple CPs in a shared network with kubevip [It] Should successfully create a cluster with multiple CPs in a shared network
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing machine remediation [It] Should replace a machine when it is destroyed
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When the specified resource does not exist [It] Should fail due to the specified account is not found [TC4a]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified domain is not found [TC4b]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified control plane offering is not found [TC7]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified template is not found [TC6]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified zone is not found [TC3]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified disk offering is not found
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the compute resources are not sufficient for the specified offering [TC8]
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified disk offer is not customized but the disk size is specified
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the specified disk offer is customized but the disk size is not specified
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist [It] Should fail due to the public IP can not be found
/jenkins/workspace/capc-e2e-new/test/e2e/invalid_resource.go:253

[Fail] When the specified resource does not exist When starting with a healthy cluster [BeforeEach] Should fail to upgrade worker machine due to insufficient compute resources
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When the specified resource does not exist When starting with a healthy cluster [BeforeEach] Should fail to upgrade control plane machine due to insufficient compute resources
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing app deployment to the workload cluster with network interruption [ToxiProxy] [BeforeEach] Should be able to create a cluster despite a network interruption during that process
/jenkins/workspace/capc-e2e-new/test/e2e/toxiproxy/toxiProxy.go:203

[Fail] When testing affinity group [It] Should have host affinity group when affinity is pro
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing affinity group [It] Should have host affinity group when affinity is anti
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing subdomain [It] Should create a cluster in a subdomain
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing Kubernetes version upgrades [It] Should successfully upgrade kubernetes versions when there is a change in relevant fields
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing resource cleanup [It] Should create a new network when the specified network does not exist
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] with two clusters [It] should successfully add and remove a second cluster without breaking the first cluster
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing app deployment to the workload cluster with slow network [ToxiProxy] [BeforeEach] Should be able to download an HTML from the app deployed to the workload cluster
/jenkins/workspace/capc-e2e-new/test/e2e/toxiproxy/toxiProxy.go:203

[Fail] When testing with disk offering [It] Should successfully create a cluster with disk offering
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

[Fail] When testing app deployment to the workload cluster [TC1][PR-Blocking] [It] Should be able to download an HTML from the app deployed to the workload cluster
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

Ran 28 of 29 Specs in 7944.351 seconds
FAIL! -- 0 Passed | 28 Failed | 0 Pending | 1 Skipped
--- FAIL: TestE2E (7944.36s)
FAIL

/run-e2e -c 4.18

@vishesh92 a jenkins job has been kicked to run test with following paramaters:

kubernetes version: 1.27.2
CloudStack version: 4.18
hypervisor: kvm
template: ubuntu-2004-kube
Kubernetes upgrade from: 1.26.5 to 1.27.2

Test Results : (tid-301)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.18
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr250-sl-301.zip
[PASS] with two clusters should successfully add and remove a second cluster without breaking the first cluster
[PASS] When testing resource cleanup Should create a new network when the specified network does not exist
[PASS] When the specified resource does not exist Should fail due to the specified account is not found [TC4a]
[PASS] When the specified resource does not exist Should fail due to the specified domain is not found [TC4b]
[PASS] When the specified resource does not exist Should fail due to the specified control plane offering is not found [TC7]
[PASS] When the specified resource does not exist Should fail due to the specified template is not found [TC6]
[PASS] When the specified resource does not exist Should fail due to the specified zone is not found [TC3]
[PASS] When the specified resource does not exist Should fail due to the specified disk offering is not found
[PASS] When the specified resource does not exist Should fail due to the compute resources are not sufficient for the specified offering [TC8]
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is not customized but the disk size is specified
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is customized but the disk size is not specified
[PASS] When the specified resource does not exist Should fail due to the public IP can not be found
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade worker machine due to insufficient compute resources
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade control plane machine due to insufficient compute resources
[PASS] When testing affinity group Should have host affinity group when affinity is pro
[PASS] When testing affinity group Should have host affinity group when affinity is anti
[PASS] When testing app deployment to the workload cluster with slow network [ToxiProxy] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing MachineDeployment rolling upgrades Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
[PASS] When testing multiple CPs in a shared network with kubevip Should successfully create a cluster with multiple CPs in a shared network
[PASS] When testing with custom disk offering Should successfully create a cluster with a custom disk offering
[PASS] When testing node drain timeout A node should be forcefully removed if it cannot be drained in time
[PASS] When testing app deployment to the workload cluster [TC1][PR-Blocking] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing machine remediation Should replace a machine when it is destroyed
[PASS] When testing app deployment to the workload cluster with network interruption [ToxiProxy] Should be able to create a cluster despite a network interruption during that process
[PASS] When testing subdomain Should create a cluster in a subdomain
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
[PASS] When testing with disk offering Should successfully create a cluster with disk offering
[PASS] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] Should successfully scale machine replicas up and down horizontally
[PASS] When testing Kubernetes version upgrades Should successfully upgrade kubernetes versions when there is a change in relevant fields

Ran 28 of 29 Specs in 8531.751 seconds
SUCCESS! -- 28 Passed | 0 Failed | 0 Pending | 1 Skipped
PASS

/run-e2e -c 4.17

@weizhouapache a jenkins job has been kicked to run test with following paramaters:

kubernetes version: 1.27.2
CloudStack version: 4.17
hypervisor: kvm
template: ubuntu-2004-kube
Kubernetes upgrade from: 1.26.5 to 1.27.2

Test Results : (tid-302)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.17
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr250-sl-302.zip
[PASS] When testing Kubernetes version upgrades Should successfully upgrade kubernetes versions when there is a change in relevant fields
[PASS] When testing app deployment to the workload cluster with network interruption [ToxiProxy] Should be able to create a cluster despite a network interruption during that process
[PASS] When testing resource cleanup Should create a new network when the specified network does not exist
[PASS] When testing MachineDeployment rolling upgrades Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
[PASS] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] Should successfully scale machine replicas up and down horizontally
[PASS] When testing machine remediation Should replace a machine when it is destroyed
[PASS] When testing node drain timeout A node should be forcefully removed if it cannot be drained in time
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
[PASS] When testing subdomain Should create a cluster in a subdomain
[PASS] When testing affinity group Should have host affinity group when affinity is anti
[PASS] When testing multiple CPs in a shared network with kubevip Should successfully create a cluster with multiple CPs in a shared network
[PASS] When testing with disk offering Should successfully create a cluster with disk offering
[PASS] When testing with custom disk offering Should successfully create a cluster with a custom disk offering
[PASS] with two clusters should successfully add and remove a second cluster without breaking the first cluster
[PASS] When the specified resource does not exist Should fail due to the specified account is not found [TC4a]
[PASS] When the specified resource does not exist Should fail due to the specified domain is not found [TC4b]
[PASS] When the specified resource does not exist Should fail due to the specified control plane offering is not found [TC7]
[PASS] When the specified resource does not exist Should fail due to the specified template is not found [TC6]
[PASS] When the specified resource does not exist Should fail due to the specified zone is not found [TC3]
[PASS] When the specified resource does not exist Should fail due to the specified disk offering is not found
[PASS] When the specified resource does not exist Should fail due to the compute resources are not sufficient for the specified offering [TC8]
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is not customized but the disk size is specified
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is customized but the disk size is not specified
[PASS] When the specified resource does not exist Should fail due to the public IP can not be found
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade worker machine due to insufficient compute resources
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade control plane machine due to insufficient compute resources

Summarizing 3 Failures:

[Fail] When testing affinity group [It] Should have host affinity group when affinity is pro
/jenkins/workspace/capc-e2e-new/test/e2e/common.go:331

[Fail] When testing app deployment to the workload cluster [TC1][PR-Blocking] [It] Should be able to download an HTML from the app deployed to the workload cluster
/jenkins/workspace/capc-e2e-new/test/e2e/deploy_app.go:111

[Fail] When testing app deployment to the workload cluster with slow network [ToxiProxy] [It] Should be able to download an HTML from the app deployed to the workload cluster
/jenkins/workspace/capc-e2e-new/test/e2e/deploy_app_toxi.go:134

Ran 28 of 29 Specs in 8818.848 seconds
FAIL! -- 25 Passed | 3 Failed | 0 Pending | 1 Skipped
--- FAIL: TestE2E (8818.86s)
FAIL

/run-e2e -c 4.18

@vishesh92 a jenkins job has been kicked to run test with following paramaters:

kubernetes version: 1.27.2
CloudStack version: 4.18
hypervisor: kvm
template: ubuntu-2004-kube
Kubernetes upgrade from: 1.26.5 to 1.27.2

Test Results : (tid-303)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.18
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr250-sl-303.zip
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
[PASS] When testing machine remediation Should replace a machine when it is destroyed
[PASS] with two clusters should successfully add and remove a second cluster without breaking the first cluster
[PASS] When testing app deployment to the workload cluster [TC1][PR-Blocking] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing MachineDeployment rolling upgrades Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
[PASS] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] Should successfully scale machine replicas up and down horizontally
[PASS] When testing app deployment to the workload cluster with slow network [ToxiProxy] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing with custom disk offering Should successfully create a cluster with a custom disk offering
[PASS] When the specified resource does not exist Should fail due to the specified account is not found [TC4a]
[PASS] When the specified resource does not exist Should fail due to the specified domain is not found [TC4b]
[PASS] When the specified resource does not exist Should fail due to the specified control plane offering is not found [TC7]
[PASS] When the specified resource does not exist Should fail due to the specified template is not found [TC6]
[PASS] When the specified resource does not exist Should fail due to the specified zone is not found [TC3]
[PASS] When the specified resource does not exist Should fail due to the specified disk offering is not found
[PASS] When the specified resource does not exist Should fail due to the compute resources are not sufficient for the specified offering [TC8]
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is not customized but the disk size is specified
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is customized but the disk size is not specified
[PASS] When the specified resource does not exist Should fail due to the public IP can not be found
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade worker machine due to insufficient compute resources
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade control plane machine due to insufficient compute resources
[PASS] When testing subdomain Should create a cluster in a subdomain
[PASS] When testing affinity group Should have host affinity group when affinity is anti
[PASS] When testing with disk offering Should successfully create a cluster with disk offering
[PASS] When testing multiple CPs in a shared network with kubevip Should successfully create a cluster with multiple CPs in a shared network
[PASS] When testing app deployment to the workload cluster with network interruption [ToxiProxy] Should be able to create a cluster despite a network interruption during that process
[PASS] When testing node drain timeout A node should be forcefully removed if it cannot be drained in time

Summarizing 3 Failures:

[Fail] When testing resource cleanup [AfterEach] Should create a new network when the specified network does not exist
/jenkins/workspace/capc-e2e-new/test/e2e/resource_cleanup.go:101

[Fail] When testing affinity group [It] Should have host affinity group when affinity is pro
/jenkins/workspace/capc-e2e-new/test/e2e/common.go:331

[Fail] When testing Kubernetes version upgrades [It] Should successfully upgrade kubernetes versions when there is a change in relevant fields
/root/go/pkg/mod/sigs.k8s.io/cluster-api/test@v1.2.12/framework/cluster_helpers.go:143

Ran 28 of 29 Specs in 9073.198 seconds
FAIL! -- 25 Passed | 3 Failed | 0 Pending | 1 Skipped
--- FAIL: TestE2E (9073.21s)
FAIL

@vishesh92 This PR looks good, and you've added unit tests as well. Can you add a check in one of the e2e tests to ensure that the resources are created on ACS ?

cluster has a different meaning when it comes to cloudstack infrastructure. In context of this PR, cluster == kubernetes cluster.

I think it would be better to rename any reference to an ACS cluster as an unmanaged cluster (the interface, method names, etc) to avoid any confusion similar to how the API has CloudStackClusterID and not ClusterID

@rohityadavcloud
a) It shouldn't be an issue if the CKS plugin is not enabled or is on an older version.
b) I have introduced a cluster type field which is used to differentiate between the clusters.

Tests were aborted.

Tests were aborted.

/run-e2e -c 4.18

Test Results : (tid-325)
Environment: kvm Rocky8(x3), Advanced Networking with Management Server Rocky8
Kubernetes Version:  v1.27.2
Kubernetes Version upgrade from:  v1.26.5
Kubernetes Version upgrade to:  v1.27.2
CloudStack Version:  4.18
Template:  ubuntu-2004-kube
E2E Test Run Logs: https://github.com/blueorangutan/capc-prs/releases/download/capc-pr-ci-cd/capc-e2e-artifacts-pr250-sl-325.zip
[PASS] When testing app deployment to the workload cluster with network interruption [ToxiProxy] Should be able to create a cluster despite a network interruption during that process
[PASS] When testing resource cleanup Should create a new network when the specified network does not exist
[PASS] When testing horizontal scale out/in [TC17][TC18][TC20][TC21] Should successfully scale machine replicas up and down horizontally
[PASS] When testing with custom disk offering Should successfully create a cluster with a custom disk offering
[PASS] When testing creation of unmanaged CKS cluster in ACS Should create a workload cluster
[PASS] When testing multiple CPs in a shared network with kubevip Should successfully create a cluster with multiple CPs in a shared network
[PASS] When testing machine remediation Should replace a machine when it is destroyed
[PASS] When testing MachineDeployment rolling upgrades Should successfully upgrade Machines upon changes in relevant MachineDeployment fields
[PASS] When testing K8S conformance [Conformance] Should create a workload cluster and run kubetest
[PASS] When testing affinity group Should have host affinity group when affinity is pro
[PASS] When testing affinity group Should have host affinity group when affinity is anti
[PASS] When testing node drain timeout A node should be forcefully removed if it cannot be drained in time
[PASS] When the specified resource does not exist Should fail due to the specified account is not found [TC4a]
[PASS] When the specified resource does not exist Should fail due to the specified domain is not found [TC4b]
[PASS] When the specified resource does not exist Should fail due to the specified control plane offering is not found [TC7]
[PASS] When the specified resource does not exist Should fail due to the specified template is not found [TC6]
[PASS] When the specified resource does not exist Should fail due to the specified zone is not found [TC3]
[PASS] When the specified resource does not exist Should fail due to the specified disk offering is not found
[PASS] When the specified resource does not exist Should fail due to the compute resources are not sufficient for the specified offering [TC8]
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is not customized but the disk size is specified
[PASS] When the specified resource does not exist Should fail due to the specified disk offer is customized but the disk size is not specified
[PASS] When the specified resource does not exist Should fail due to the public IP can not be found
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade worker machine due to insufficient compute resources
[PASS] When the specified resource does not exist When starting with a healthy cluster Should fail to upgrade control plane machine due to insufficient compute resources
[PASS] When testing app deployment to the workload cluster [TC1][PR-Blocking] Should be able to download an HTML from the app deployed to the workload cluster
[PASS] When testing Kubernetes version upgrades Should successfully upgrade kubernetes versions when there is a change in relevant fields
[PASS] When testing subdomain Should create a cluster in a subdomain
[PASS] with two clusters should successfully add and remove a second cluster without breaking the first cluster
[PASS] When testing with disk offering Should successfully create a cluster with disk offering

Summarizing 1 Failure:

[Fail] When testing app deployment to the workload cluster with slow network [ToxiProxy] [It] Should be able to download an HTML from the app deployed to the workload cluster
/jenkins/workspace/capc-e2e-new/test/e2e/deploy_app_toxi.go:134

Ran 28 of 30 Specs in 8968.702 seconds
FAIL! -- 27 Passed | 1 Failed | 0 Pending | 2 Skipped
--- FAIL: TestE2E (8968.71s)
FAIL

It seems like this could be done by anything with access to the cloudstack API and the kube api server, right? We don't need special information that is only available inside the capc controller or to hook into particular events that are hidden in the reconcilers. This just needs to watch the CloudstackCluster resources and possibly the CloudstackMachines's?

I want to make the integration a part of the CAPC controller since it has enough information about all the resources and the account and domain it needs to be part of. In future we might add even more information regarding the CAPC cluster in CKS as well and having it as part of the controller would help.

I might be missing the point of this feature but it feels a little bit like unnecessary complexity. It's not only just adding this feature, now we need to maintain it and test it and the benefits seem dubious to be.
Given that the CAPI API is our pane of glass to describe k8s clusters, adding a second one that works in parallel and not built on top of it appears like duplicated functionality.

I'm with you, this is adding a lot of CKS-specific logic to the controller, whose only task it should be to reconcile CAPI based clusters. I think there are bigger fish to fry (keeping up-to-date with CAPI and k8s versions, releasing more often etc.)

Hi @g-gaston @hrak I see your points and completely understand why would anybody want to make CAPC complex. But hear us out.
Apache CloudStack has always been a community-led, turnkey IaaS platform. The emphasis on turn-key, easy to use, setup and deploy, with a great UI and that it just works (out of the box). While CAPC can be used as a building-block, such as for EKS-A, we think this is also used directly as a first-class tool in itself with clusterctl.
Before CAPC, CloudStack's only supposed Kubernetes integration was in the form of CKS (CloudStack Kubernetes Service) wherein CloudStack orchestrates k8s clusters for its users. However, the CKS built clusters make some assumption and deploy specific CNIs to make this work out of the box - easy to deploy k8s cluster for most users. And CKS is available for all user account types, if the service providers, operator or enterprise admin enables and configures CKS in their environment.
With CAPC, there is more flexibility but this isn't available for all user types (currently, domain and root admins) and a group of contributors think it would be good idea if an end user can simply deploy a CAPC cluster and there is just a linked-resource created in the form of unmanaged CKS cluster (for example, to give you an idea head over to https://qa.cloudstack.cloud/simulator/#/kubernetes to see such a resource, this shoudn't be confused with a CloudStack Cluster of a zone).
To address these, we've created a general mechanism via https://github.com/apache/cloudstack/pull/7515 and series of fixes such the ability for normal user accounts to pick/use a public IP say in a shared network with https://github.com/apache/cloudstack/issues/7817. This could help fix https://github.com/kubernetes-sigs/cluster-api-provider-cloudstack/issues/303 in future releases. The final lacking piece if a normal user account deploy a CAPC cluster, can we give him a control plane view to say there exists a k8s cluster in CloudStack but it's externally (CAPC) managed - that's where this feature comes in.
Now, you're both right - we don't need this PR to do that, there could be an external script and tool for this, however, it would make it complex for users and operators to set this up and the overall solution wouldn't be turnkey. Such an external tool/script would need to be aware of CloudStack and aware/access of the CAPC cluster and do that for all accounts would just introduce multiple competing control plane paths.
So, here's my proposal - @vishesh92 can introduce an new optional/parameter in the CAPC cluster generated yaml (or via a support cmd-line arg that generate the same) to enable/disable the feature. By default, this is kept enabled so any CAPC user deploying a k8s cluster see an unmanaged CKS cluster in the UI. And downstream tools and users who don't want this behaviour can turn it off via the option/setting in the generated yaml. This way you can control that the new code introduced in this PR doesn't get into the operational path in downstream use-cases, and upstream CAPC and ACS releases are turnkey.
Thoughts?

Hi @rohityadavcloud,
Thanks for sharing your thoughts and elaborating on intent. I do have reservations about this feature. Maintaining the scope outlined in the Cluster API documentation for infrastructure providers ensures we don't feature creep and have an easier time maintaining CAPC. Additional functionality, such as this PR, can be added through auxiliary components, as you highlighted, which would honor the single responsibility principle. Furthermore, considering the roadmap for CKS integration is somewhat unclear to the CAPC community, having a best effort integration doesn't feel like the right path forward.

With CAPC, there is more flexibility but this isn't available for all user types ... it would be good idea if an end user can simply deploy a CAPC cluster and there is just a linked-resource created in the form of unmanaged CKS cluster

This sounds like 2 different asks:

"I want to deploy CAPC clusters using any user type"
"I want a cluster to appear in my CKS plugin UI"

On (1), I think we need to ask why said user types are required and broaden the scope we're considering. For example, perhaps a more flexible permissions model in Apache CloudStack would enable finer permission control so users can create whatever kind of user they wish?
On (2), I empathize with this desire, but I question what a user can do with an unmanaged CKS cluster (not to mention it isn't a CKS cluster) -  it's read-only if I'm not mistaken? Have we established why this is needed? In any case, it feels like it falls outside the scope of CAPC.

however, it would make it complex for users and operators to set this up and the overall solution wouldn't be turnkey

I think that's dependent on how you create a solution. Cluster API in itself isn't turnkey: you have to deploy Cluster API, then you can create your clusters. There's nothing preventing building a solution that wraps deployment of all the necessary components, including a component that embodies the integration in this PR, which would be truly turnkey; though I don't think that's a concern for CAPC.

Such an external tool/script would need to be aware of CloudStack and aware/access of the CAPC cluster and do that for all accounts would just introduce multiple competing control plane paths.

That seems like an appropriate set of dependencies for this kind of component, though I don't understand the "do it for all accounts" or "competing control plane paths" parts; perhaps you could elaborate?
I think its worth pointing out that all existing components built for Kubernetes have this kind of dependency set. Monitoring tools like Jaeger and Prometheus come to mind.

Hi @vishesh92. It sounds like you may have tried gathering feedback prior to completing the upstream CloudStack work and the work in this PR. I'm sorry I missed that and have provided feedback late in the day including leaving this PR untouched for a prolonged period. We should be aiming to avoid this again, lets chat in the Kubernetes CAPC Slack on how to best address the communication inefficiencies.

source: gharchive/pull-request

authors: blueorangutan chrisdoherty4 codecov-commenter davidjumani hrak rohityadavcloud vishesh92 weizhouapache

license_type: permissive

license_source: github-api


text:
Validate container status in volume mount tests

What type of PR is this?
/kind cleanup
What this PR does / why we need it:
We fixed an issue in CRI-O returning an invalid container status on different mount propagations: https://github.com/cri-o/cri-o/pull/5981
To avoid such failures everywhere, we now also validate the container status in critest.
Which issue(s) this PR fixes:

Refers to https://github.com/cri-o/cri-o/pull/5981
Special notes for your reviewer:
Does this PR introduce a user-facing change?

Verify container status on all three "Container Mount Propagation" critest cases.

Ready for review :+1:

repo: kubernetes-sigs/cri-tools


text:
[draft] Kubernetes 1.19 Release Cycle

This outlines the descheduler release plans for the upstream 1.19 release cycle (April 13 - August 4), ending with a rebase to the upstream 1.19 dependencies soon after 1.19 GA (on August 4).
At the start of this cycle, the descheduler was still on 1.17 dependencies so we needed to take some steps to catch up:

[x] Create release-1.17 branch
[ ] Tag v0.17.0 release
[ ] Create release-1.18 branch, with updated 1.18 deps
[ ] Tag v0.18.0 release (this should actually probably be v0.18.2 since that's the matching upstream tag for what we bumped to in https://github.com/kubernetes-sigs/descheduler/pull/280)
[ ] (Soon after 1.19 GA) Create release-1.19 branch with updated 1.19 deps and tag v0.19.0 release

This will have us caught up so that, at the conclusion of the 1.20 upstream release cycle, we only need to update our deps, branch, and tag

/assign @aveshagarwal @ravisantoshgudimetla @seanmalloy @ingvagabund
This is a summary of the steps we discussed in https://github.com/kubernetes-sigs/descheduler/issues/273 for our branching/release cycle and gets us caught up so that we ideally only have to do this once per k8s release for now on.
If this looks good, could you all please give a +1 to acknowledge it, or add any comments we should address here? Once we have a consensus we can begin this process

+1
I don't think the patch version for descheduler needs to match the k8s version. The patch version for descheduler should be used to keep track of descheduler bug fixes. I recommend releasing v0.18.0 instead of v0.18.2.

What is the benefit of descheduler 0.17 based on k8s 1.17? I am pretty sure descheduler 0.18 based on k8s 1.18 should work for 1.17 too without any issue, or are there any 1.18 specific APIs we are using in descheduler might not work for k8s 1.17? I mean k8s 1.18 is the latest release, and if going forward we want to map every descheduler release with corresponding k8s release thats fine. So why not just start with 1.18? But don't have strong opinion either way, your choice if you guys have cycles.
Also if we are doing the above, should we also clarify, perhaps make it explicit, about descheduler's supportability statement in README, may be by providing some sort of matrix that even though there is going to be descheduler release for every k8s release, but each descheduler release should work fine on multiple releases of k8s? What are your thoughts? I am just hoping that users do not get confused that descheduler 0.x would only work for k8s 1.x.

I don't think the patch version for descheduler needs to match the k8s version.

I agree.

The patch version for descheduler should be used to keep track of descheduler bug fixes. I recommend releasing v0.18.0 instead of v0.18.2.

Matching patch releases might be too much it seems and might not make much sense IMO, as APIs dont change in k8s patch releases.

In general, +1.

One thing I noticed that PRs by people (me, @damemi, and i think ravi) who have approval permission are getting self approved. Could we look into it so that PRs dont get self approval?

What is the benefit of descheduler 0.17 based on k8s 1.17? I am pretty sure descheduler 0.18 based on k8s 1.18 should work for 1.17 too without any issue

Good point, we don't really need to catch ourselves up when we can just start with a v0.18.0 release and keep it matched up from there.
@seanmalloy what do you think? Would you be willing to write up v0.18.0 release notes for us?

@seanmalloy what do you think? Would you be willing to write up v0.18.0 release notes for us?

@damemi in general this sounds good to me. Yes, I can write up the release notes for v0.18.0. I'll add a comment to this issue with the release notes when I have them ready.

@damemi here are the release notes for v0.18.0. I made some slight changes to the format since that last release. The release notes are also available in this gist.
Docker images are available on Google Container Registry:
docker run asia.gcr.io/k8s-artifacts-prod/descheduler/descheduler:v0.18.0 --help
docker run eu.gcr.io/k8s-artifacts-prod/descheduler/descheduler:v0.18.0 --help
docker run us.gcr.io/k8s-artifacts-prod/descheduler/descheduler:v0.18.0 --help

New Features :rainbow:

#254 Add new RemovePodsHavingTooManyRestarts strategy @damemi
#274 Add new PodLifeTime strategy @seanmalloy

Bug Fixes :bug:

#181 LowNodeUtilization strategy takes node taints into consideration @jw-s
#240 Fix serialization for NodeResourceUtilizationThresholds parameters @ingvagabund
#242 Fix container image location in Job and CronJob YAML @seanmalloy
#249 Update nodes on every iteration with CLI option --descheduling-interval @ingvagabund
#256 Fix early return from PodFitsAnyNode check @damemi
#275 RemoveDuplicates strategy now takes container image name into account @damemi
#276 Fix serialization for RemovePodsHavingTooManyRestarts parameters @damemi

Others :running:

#233 Update release documentation @seanmalloy
#239 Update email address in Dockerfiles @aveshagarwal
#244 Create initial user and contributor guides @seanmalloy
#255 Update to Go 1.13.9 @seanmalloy
#258 Add CI test to verify vendor changes @damemi
#256 Fix early return from PodFitsAnyNode check @damemi
#260 Update link to contributor cheat sheet @CriaHu
#262 Improved LowNodeUtilization performance @ingvagabund
#264 Update documentation for RemovePodsViolatingNodeAffinity strategy @ingvagabund
#265 Refactor taint code to reduce duplicate code @ingvagabund
#266 Refactor maximum pods per nodes check to reduce duplicate code @ingvagabund
#278 Update list of strategies in README @lixiang233
#279 Add namespace to logs for evicted pods @seanmalloy
#280 Update do k8s v1.18 dependencies @damemi

Also if we are doing the above, should we also clarify, perhaps make it explicit, about descheduler's supportability statement in README, may be by providing some sort of matrix that even though there is going to be descheduler release for every k8s release, but each descheduler release should work fine on multiple releases of k8s? What are your thoughts? I am just hoping that users do not get confused that descheduler 0.x would only work for k8s 1.x.

@aveshagarwal I'll get a PR submitted soon to update the Compatibility Matrix section of the README.

@seanmalloy thanks for the great notes, are those production images already promoted for v0.18?

are those production images already promoted for v0.18?

@damemi no they are not. Someone needs to follow the release guide. We need to add a step to the release guide about creating release branches.

@seanmalloy ok, just checking.
I think we've addressed everything that was brought up so if there are no more suggestions I'll create the release-1.18 branch at EOD today (to give a last call for anyone to speak up).
Then I will publish the draft v0.18.0 release, which should create the tag. @seanmalloy after that do you want to take responsibility for opening the image promotion PR? (Also, what's the difference in the "manual" and "semi-auto" sections of the release guide, besides pushing a staging image? Do we need to push a staging image?)

@damemi I created PR #288 to update the compatibility matrix and PR #289 to update the release guide with some details on creating the release branches.

Also, what's the difference in the "manual" and "semi-auto" sections of the release guide

The container images are built and pushed to the staging registry by an automated prow job with the "semi-auto" process. Using the "manual" process a person builds and pushes the container images from their laptop to the staging registry. The "manual" process would only need to be used if the automated prow job fails for some reason.

@seanmalloy after that do you want to take responsibility for opening the image promotion PR?

Yes, I can take care of that step.

Branch release-1.18 cut here: https://github.com/kubernetes-sigs/descheduler/tree/release-1.18, and I have the 1.18 release draft ready (not sure how to share it besides marking it pre-release or just publishing it, but it's just copy-pasted from the notes above)
@seanmalloy once the image promotions are ready I'll publish the release

@damemi I see the release-1.18 branch. But I don't see a new tag pushed, https://github.com/kubernetes-sigs/descheduler/tags. The newest tag is still v0.10.0. My guess is creating a draft release does not create a tag. I think you need to create and push the v0.18.0 tag. Pushing the tag will trigger the prow job to build the container image.

@seanmalloy I just pushed v0.18.0 tag.

Here is the v0.18.0 container image in the staging registry that needs to promoted gcloud container images describe gcr.io/k8s-staging-descheduler/descheduler:v20200521-v0.18.0. I'm hoping the submit the PR soon to promote the image.

Pull request for v0.18.0 image promotion https://github.com/kubernetes/k8s.io/pull/888

@damemi the --help CLI option needs to be removed from the docker run commands that I put in the release notes. You should be able to edit the v0.18.0 draft release to remove the --help CLI option.

@seanmalloy thanks for catching that, I see the image promotion has merged so I've published the release here: https://github.com/kubernetes-sigs/descheduler/releases/tag/v0.18.0 (I also included a note explaining the versioning change)
Now that we're caught up we will do the same process again around 8/4 for the 1.19 GA. Thanks everyone!

@damemi do you think we should define the list if features we would like to target for the v0.19.0 release?
Here is a starter list:

Topology spread strategy #146
Run by namespace #251
Run by label selector #195
Update to klog v2 #286

Keeping in mind there is only about 2 months until v0.19.0.

With 1.19 code freeze today, I opened https://github.com/kubernetes-sigs/descheduler/pull/337 to test bumping our k8s deps to a 1.19 branch (currently beta.2). This is just to test our compatibility right now to prepare for a smooth release when it's time, there are still features we're working on for 1.19 and we aren't strictly beholden to the code freeze today

Started working on release notes for 1.19, if anything's missing or in the wrong spot so far let me know. Obviously we still have a couple things that are being worked on so this is incomplete right now
## New Features :rainbow:
* #338 Filter pods by namespaces @ingvagabund

## Bug Fixes :bug:
* #285 Change break condition and thresholds validation for lowUtilization @lixiang233
* #310 Skip evicting when no evictable pod found on node @lixiang233
* #312 Pass golint check for pkg/descheduler/strategies/ @lixiang233
* #319 Remove redundant eviction log message and add "Reason" param to EvictPod @damemi
* #330 Fix examples typo @paulfantom
* #336 avoid appending list multiple times in RemoveDuplicates @lixiang233
* #361 remove unnecessary line feed in log messages @jjmengze
* #362 Update version parsing to exclude helm-chart tags @damemi

## Others :running:
* #296 Make DeschedulerStrategy.Params a pointer @ingvagabund
* #297 Add verify-gofmt make target @ingvagabund
* #299 Standardize node affinity strategy logs @damemi
* #300 Add more verbose logging to IsEvictable checks @damemi
* #307 Add parent make verify target @damemi
* #321 Add Pod Eviction Reason To Events @seanmalloy
* #322 Move sortPodsBasedOnPriority to pod util @lixiang233
* #325 Add initial GitHub issue templates @seanmalloy
* #327 Update klog to v2 @farah
* #328 Update To Go 1.14.4 @seanmalloy
* #332 Add maxPodsToEvictPerNode to LowNodeUtilization testcase struct @lixiang233
* #333 Support only one sorting strategy in lowNodeUtilization @lixiang233
* #340 Update e2e script to use Kind setup @damemi
* #344 have k8s version configurable when creating cluster through kind @ingvagabund
* #342 Remove Travis CI Configuration @seanmalloy
* #298 Add helm	chart @stevehipwell
* #343 Clean e2e test so it's easier to extend it @ingvagabund
* #351 Update Release Documentation For Helm Charts @seanmalloy
* #356 Update Helm release action to work on release branches @damemi
* #359 More Helm Documentation @seanmalloy
* #360 Add NPD+CA autohealing use case to user guide @dharmab
* #337 Rebase k8s dependencies to 1.19-rc.2 @damemi

I opened https://github.com/kubernetes-sigs/descheduler/issues/367 to track updating to Go 1.15. The k8s v1.19 release is going to use Go 1.15. This should probably be done prior to releasing descheduler v0.19.0.

We talked about it today and decided to plan on tagging the 1.19 release on Monday, August 31.
This is due to upstream 1.19 GA still being planned for Tuesday (8/25), and some of us planning to be OOO next week. Releasing the following Monday will ensure there are people available to catch any big fallout.
This also gives us time to finish up any ongoing work for this release, and prepare/review PRs for the GA bump.
If this sounds good to everyone, please prioritize any last minute PRs. Thanks!

I updated the release notes draft in https://github.com/kubernetes-sigs/descheduler/issues/284#issuecomment-665856609. If these look good we'll publish the 1.19 tag EOD today

The descheduler v0.19.0 release is done.
/close

source: gharchive/issue

authors: aveshagarwal damemi seanmalloy

repo: kubernetes-sigs/descheduler


text:
Helm chart rename - Step #3

IMPORTANT - This needs to be merged after PR #457 and PR #459.
This PR will release the v0.19 chart with the new descheduler name.

/assign @damemi

/hold
for steps 1 & 2

/ok-to-test

@damemi I made a change in the README but it looks like my formatter tidied it up too. I'm not by my machine at the moment so is it possible to take the change or do I need to update the PR? This is time sensitive as v0.19 will be missing from the hub so if I need to I can get back to my machine.

@stevehipwell I suppose those changes aren't a big problem, and it does correct some formatting that was messed up (like extra backtics in code blocks)
Since this is time sensitive, I'm fine merging the changes in formatting now too.
/hold cancel
Once this merges, please check that everything looks good on your end too and let us know. Thank you!

/approve
/lgtm

@damemi this all looks good to me.

@damemi it looks like the GitHub release action has failed for the v0.19 release.

@stevehipwell thanks, looks like a CI check was blocking it. Re-ran and it looks like it completed

@damemi I think you either need to delete the descheduler-helm-chart-0.19.2 tag and then re run the action or manually modify the index.yaml file on the gh-pages branch; as it looks like the failure wasn't cleaned up correctly so resulted in a NOP.

@stevehipwell oh, that makes sense. I was able to delete the branch/release and re-run the job. The right version appears to be in index.yaml now: https://github.com/kubernetes-sigs/descheduler/blob/97a0ab9/index.yaml
and the release is back https://github.com/kubernetes-sigs/descheduler/releases/tag/descheduler-helm-chart-0.19.2
If you can double check that I got this right and I'm not missing anything else that would be great!

@damemi I can now see the v0.19 chart on artifacthub.io so it looks good to me.

authors: damemi stevehipwell


text:
Cordon Drifted nodes before processing evictions

Tell us about your request
We currently use a tool which will roll all nodes in an ASG when the launch template configuration changes. This is usually in response to AMI Upgrades.
One of the features of this tool is that when it detects this change, it will cordon all nodes in the ASG before it starts replacing nodes. This ensures that workloads on evicted nodes do not reschedule onto nodes that are about to get killed again.
Can Karpenter implement this for it's Drift calculations?
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We have quite a few services that do not handle restart well or rapid restarts well and would like to minimize how many times they get restarted.
Are you currently working around this issue?
No feature like this in Karpenter.
Additional Context
No response
Attachments
No response
Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

/remove-lifecycle rotten

/assign

#1314 will address this issue, albeit through a slightly different mechanism. Rather than cordoning the nodes with a NoSchedule taint, Karpenter will apply a PreferNoSchedule taint to voluntary disruption candidates, and it will apply an in-memory NoSchedule taint to drifted nodes. We don't apply this in-memory taint to underutilized nodes since additional pods scheduling is the signal we use to cancel a consolidation decision. The in-memory taint combined with the PreferNoSchedule taint ensures that Karpenter will pre-spin new capacity for drifted nodes, rather than evicting and rescheduling pods to other drifted nodes.

/remove-lifecycle stale
/lifecycle frozen
Going to freeze this issue, #1314 should address this issue but there is a race condition that needs to be fixed first. Hopefully I should be able to prioritize this in the next few weeks.

#1314 has now been closed off without merging so where does that leave this issue?

authors: andyspiers jmdeal jukie sidewinder12s

repo: kubernetes-sigs/karpenter


text:
Validation fails for files with only ---

What happened?
Validating a file containing only --- will cause validation to fail.
❯ kubectl-validate has3hyphen.yaml

has3hyphen.yaml...ERROR
GVK cannot be empty
Error: validation failed

What did you expect to happen?
This file is essentially an empty file, and I expect the validation to succeed as well as a file containing only comments.
How can we reproduce it (as minimally and precisely as possible)?
has3hyphen.yaml
---

Validate the above file.
❯ kubectl-validate has3hyphen.yaml

has3hyphen.yaml...ERROR
GVK cannot be empty
Error: validation failed

Anything else we need to know?
No response
Kubernetes version

❯ kubectl version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

I agree this is a bug. Lots of files by convention contain empty doccuments. We should ignore them
/label good-first-issue
/assign

Hi @alexzielenski, i would like to take a look at this if its ok?

/assign @ah8ad3

repo: kubernetes-sigs/kubectl-validate


text:
Set pending condition on QueuedWorkload with message

What type of PR is this?
/kind feature

What this PR does / why we need it:
Which issue(s) this PR fixes:

Initial implementation for #102
sets:

The Queue doesn't exist
The ClusterQueue doesn't exist

Still missing:

The QW's namespace is not allowed by the ClusterQueue
The workload was attempted for scheduling but it didn't fit.

@ArangoGutierrez: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name	Commit	Details	Required	Rerun command
pull-kueue-test-integration-main	c6a4a94 link	true	/test pull-kueue-test-integration-main
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

I know I know..... looking what I might be missing

Although this is still a WIP, I would like to have some eyes here @alculquicondor @ahg-g @denkensk

I will work on a diff approach to this
/close

repo: kubernetes-sigs/kueue


text:
[wip]Add docsy-example site for testing

Add docsy-example site for testing generated Markdown pages.
The generated _index.md page contains a reference to the glossary_definition shortcode.
To get around this, I copied a generic _index.md file to the root location.
The file tree could be improved/cleaned up.

/approve cancel

/approve cancel

Just now I've not got time to look at the implementation, but this sounds great.
/approve

Whoops! I should have marked this PR as WIP. I planned to make several more changes.
Will create a follow-up PR.

I am making corrections and testing. Looks like the theme submodule is not set up in my last commit.

authors: kbhawkey sftim

repo: kubernetes-sigs/reference-docs


text:
Fix typo in service bindings deletion message

This PR is a

[ ] Feature Implementation
[x] Bug Fix
[ ] Documentation

What this PR does / why we need it:
It fixes a typo in a user-facing string.
Which issue(s) this PR fixes
None.
Please leave this checklist in the PR comment so that maintainers can ensure a good PR.
Merge Checklist:

[ ] New feature

[ ] Tests
[ ] Documentation

[ ] SVCat CLI flag
[ ] Server Flag for config

[ ] Chart changes
[ ] removing a flag by marking deprecated and hiding to avoid
breaking the chart release and existing clients who provide a
flag that will get an error when they try to update

/ok-to-test

/lgtm

authors: akelsch jhvhs piotrmiskiewicz

repo: kubernetes-sigs/service-catalog

license_source: bigquery


text:
Removing Kubernetes version from Antrea install

Fixes #https://github.com/kubernetes-sigs/sig-windows-dev-tools/issues/146
Fixes #https://github.com/kubernetes-sigs/sig-windows-dev-tools/issues/138

/lgtm

authors: dougsland knabben

repo: kubernetes-sigs/sig-windows-dev-tools


text:
[stable/mariadb] Use storageClassName for mariadb.

Fixed mariadb for #1869
/cc @prydonius

In the charts meeting we said there's no reason this can't be merged. I had a question about the need for the - abstraction, but we agreed I'll open a separate discussion issue about the differences in go template between nil and "" (empty string), and if it turns out we can easily detect the difference between the two, I could open a refactoring PR for this. 👍

/lgtm

authors: gyliu513 mgoodness scottrigby

repo: kubernetes/charts


text:
Using client-go returns errors of "cannot find package"

I wrote a simple application to print number of pods:
package myapp

import (
	"fmt"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
)

func main() {
	config, err := rest.InClusterConfig()
	if err != nil {
		panic(err.Error())
	}
	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		panic(err.Error())
	}

	pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{})
	fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
}

When I build my application I am gettings lots of errors:
..\k8s.io\client-go\discovery\discovery_client.go:28:2: cannot find package "github.com/golang/protobuf/proto" in any of:
        C:\Go\src\github.com\golang\protobuf\proto (from $GOROOT)
        C:\Users\myuser\go\src\github.com\golang\protobuf\proto (from $GOPATH)
..\k8s.io\apimachinery\pkg\apis\meta\v1\micro_time.go:23:2: cannot find package "github.com/google/gofuzz" in any of:
        C:\Go\src\github.com\google\gofuzz (from $GOROOT)
        C:\Users\myuser\go\src\github.com\google\gofuzz (from $GOPATH)
..\k8s.io\client-go\discovery\discovery_client.go:29:2: cannot find package "github.com/googleapis/gnostic/OpenAPIv2" in any of:
        C:\Go\src\github.com\googleapis\gnostic\OpenAPIv2 (from $GOROOT)
        C:\Users\myuser\go\src\github.com\googleapis\gnostic\OpenAPIv2 (from $GOPATH)
..\k8s.io\apimachinery\pkg\runtime\serializer\json\json.go:25:2: cannot find package "github.com/json-iterator/go" in any of:
        C:\Go\src\github.com\json-iterator\go (from $GOROOT)
        C:\Users\myuser\go\src\github.com\json-iterator\go (from $GOPATH)
..\k8s.io\apimachinery\pkg\runtime\serializer\json\json.go:26:2: cannot find package "github.com/modern-go/reflect2" in any of:
        C:\Go\src\github.com\modern-go\reflect2 (from $GOROOT)
        C:\Users\myuser\go\src\github.com\modern-go\reflect2 (from $GOPATH)
..\k8s.io\client-go\plugin\pkg\client\auth\exec\exec.go:35:2: cannot find package "golang.org/x/crypto/ssh/terminal" in any of:
        C:\Go\src\golang.org\x\crypto\ssh\terminal (from $GOROOT)
        C:\Users\myuser\go\src\golang.org\x\crypto\ssh\terminal (from $GOPATH)
..\k8s.io\apimachinery\pkg\util\net\http.go:34:2: cannot find package "golang.org/x/net/http2" in any of:
        C:\Go\src\golang.org\x\net\http2 (from $GOROOT)
        C:\Users\myuser\go\src\golang.org\x\net\http2 (from $GOPATH)
..\k8s.io\client-go\transport\round_trippers.go:25:2: cannot find package "golang.org/x/oauth2" in any of:
        C:\Go\src\golang.org\x\oauth2 (from $GOROOT)
        C:\Users\myuser\go\src\golang.org\x\oauth2 (from $GOPATH)
..\k8s.io\client-go\util\flowcontrol\throttle.go:25:2: cannot find package "golang.org/x/time/ra..\k8s.io\client-go\transport\round_trippers.go:25:2: cannot find package "golang.org/x/oauth2"
in any of:
        C:\Go\src\golang.org\x\oauth2 (from $GOROOT)
        C:\Users\myuser\go\src\golang.org\x\oauth2 (from $GOPATH)
..\k8s.io\client-go\util\flowcontrol\throttle.go:25:2: cannot find package "golang.org/x/time/rate" in any of:
        C:\Go\src\golang.org\x\time\rate (from $GOROOT)
        C:\Users\myuser\go\src\golang.org\x\time\rate (from $GOPATH)
..\k8s.io\apimachinery\pkg\api\resource\amount.go:23:2: cannot find package "gopkg.in/inf.v0" in any of:
        C:\Go\src\gopkg.in\inf.v0 (from $GOROOT)
        C:\Users\myuser\go\src\gopkg.in\inf.v0 (from $GOPATH)
..\k8s.io\apimachinery\pkg\labels\selector.go:29:2: cannot find package "k8s.io/klog" in any of:
        C:\Go\src\k8s.io\klog (from $GOROOT)
        C:\Users\myuser\go\src\k8s.io\klog (from $GOPATH)
..\k8s.io\client-go\util\flowcontrol\backoff.go:24:2: cannot find package "k8s.io/utils/integer" in any of:
        C:\Go\src\k8s.io\utils\integer (from $GOROOT)
        C:\Users\myuser\go\src\k8s.io\utils\integer (from $GOPATH)
..\k8s.io\apimachinery\pkg\util\yaml\decoder.go:30:2: cannot find package "sigs.k8s.io/yaml" in any of:
        C:\Go\src\sigs.k8s.io\yaml (from $GOROOT)
        C:\Users\myuser\go\src\sigs.k8s.io\yaml (from $GOPATH)

I need to install each package individually, I think there should be a configuration file that install all these libraries to the user.
It doesn't make sense that I need to install something like 20 libraries just to be able to use client-go library.

see the install instructions in https://github.com/kubernetes/client-go/blob/master/INSTALL.md#for-the-casual-user which describe how to use go modules without needing to install each package manually
/close

I stumbled on this issue whilst using Bazel on a project that had dependencies that used clint-go, and it took ages to solve, so I thought I'd drop my solution here since it's the top search result for:
external/io_k8s_apimachinery/pkg/util/net/http.go:174:13: undefined: http2.ConfigureTransports

Which was the symptom I had, and others I imagine may stumble on the same issue.
If you have this with Bazel, make sure your WORKSPACE declares your own registered Go dependencies as early as you can, since some things (Gazelle in my case) will pre-empt you and set up dependencies first, and the earlier one is preferred. So you may have a modern dependency on x/net but you end up using Gazelle's from 2019.

authors: ashleydavies liggitt mydockergit

repo: kubernetes/client-go


text:
type net.IP has no field or method DeepCopy

When generating clients for types that refer to net.IP the code generator generates invalid go code which is not able to compile. See for instance https://github.com/giantswarm/apiextensions/pull/16. I am wondering how this is supposed to work. Can somebody elaborate on this? Our workaround was to rather use type string than net.IP but I feel like this should be fixed upstream. Cheers.

I believe I am seeing the same behaviour when trying to use a struct inside my CRD spec that comes from another package

repo: kubernetes/code-generator


text:
Ansible: fix make-ca-cert

Fixes #17911 (in kubernetes/kubernetes):
Export empty proxy no more
Cut CN's over 64 bytes

Both of these are awesome. Thank you! Just some tiny tiny nits trying to make it easier for me to support this long term.

Ehi @eparis, is there anything I can do to make this feature land to the master branch?
I was afflicted by the CN issue but applying this fix to my working copy I've been able to successfully move forward the provisioning.
Thank you very much.

@atosatto send a new PR which starts here and fixes my comments and I'll merge it.

@eparis I think we can now close this PR since all its changes has been merged with PR#440.

authors: Arano-kai atosatto eparis

repo: kubernetes/contrib


text:
Logic for the token is wrong

The use of token or basic auth was a conditional inside the TLS verify loop.
You can use TLS or skip the verification and use token or basic auth.
Just a tab issue. Put the token conditional outside the kubeInsecure check.
Also handled the case that we actually switch the boolean for skip TLS and forced it to replace true or false.

I tested this by building my own resourcifier image locally and it works fine.
I am finally able to deploy templates.

@jackgr

Thanks, @runseb!

authors: jackgr runseb vaikas-google

repo: kubernetes/deployment-manager


text:
feat(chart): Add a chart/ package.

This package provides a chart struct, a chartfile struct, and methods to load charts either from directories or from tgz files. The testdata directory contains some variations on charts .
Also, added a very thin log abstraction layer to give errors, warnings, info, and debug messages some consistency.

@adamreese If you have a second to look at this, I'd appreciate it.

seems sane to me

Would like to see what the refactoring of the template management code (i.e., expandybird/expander/expander.go) looks like with this package in place. Can you push a WIP for that?

The previous URL package has been moved to chart.Locator (to distinguish it from a full URL). It's now in the PR. I also added some tests on Windows paths, following a blog from Microsoft on the topic (given that I know nothing about Windows).

Merging to get these changes in. Please fix the remaining issues.

authors: adamreese jackgr technosophos


text:
[Dashboard] No Graphs, followed guidance

Looking for the exact install instructions to get Heapster, InfluxDB, Grafana working and Dashboard to include graphs. I am using kubeadm to install a basic 3 node cluster. Typically I use the ansible scripts to do the full install of kubernetes on Centos 7.3. However, decided to try kubeadm. To get Heapster/InfluxDB/Grafana in stalled I followed the guidance of : kubectl create -f deploy/kube-config/influxdb/
Services were successfully created. However, Grafana throws an error on the UI that no server found and Dashboard still missing the graphs, etc.
I have looked online and found very little if any step-by-step instructions of getting this working when using kubeadm as the base installer for the cluster.
Kubeadm installs latest from repo which is v1.6.2 of kubernetes. Using Calico for the networking. Dashboard functional (minus graphs) along with all services are running with no issues
Heapster: v1.3.0
InfluxDB: v1.1.1
Grafana: v4.0.2
Please advise
Thanks
DB

Are there any issues in the Heapster logs?  Have you tried asking the dashboard repo?

I will ask the dashboard repo since this issue seems to get punted along and no one can even spell out the exact instructions as referenced above

So here is how I have installed this:

Git clone heapster project

Just use the kube.sh in deploy folder to initiate. Suppose I could use kubectl directly as well but result is the same when I have done so

I modify grafana.yaml in (/deploy/kube-config/influxdb) to use the api server proxy
value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/

no other changes. Grafana can see the template, shows Cluster and Pod as the choices. The InfluxDB data source is connected with no issues, however, there is no data points showing in Grafana

In checking heapster it is running. I query it with curl via:
curl localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/metrics

The result is a long list of metrics

Checking further...
curl localhost:8001/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/debug/allkeys

I get a return of:
[
"cluster"
]
What more can be done to check to see if heapster is feedin influxdb the data. This seems like perhaps the issue possibly.
Thanks
DB

Also is cAdvisor part of the equation ?

Heapster makes use of the cAdvisor built in to the Kubelet.
If you run Heapster at high verbosity (--v=10), what do the Heapster logs say?

repo: kubernetes/heapster


text:
Fix default value information

proxy-read-timeout and proxy-send-timeout default value is 60 seconds, not 30.

This change is

/lgtm

@eduardobaitello thanks!

Coverage remained the same at 46.232% when pulling f737cdcae50cdef610ad4032e2a3f919f9f57b60 on eduardobaitello:patch-1 into 5fb45c962a51402fecfa66379aeed618e5e0b18d on kubernetes:master.

authors: aledbf coveralls eduardobaitello k8s-reviewable

repo: kubernetes/ingress


text:
Add firewall configuration section to README

Adds a section on how to configure GCP firewall rules to enable cluster-api to orchestrate GCP VMs.

This change is

repo: kubernetes/kube-deploy