text:
No removable pods on node，why?

I0912 09:17:02.750207       1 nodeutilization.go:204] "Node is underutilized" node="home0123" usage={"cpu":"600m","memory":"306Mi","pods":"4"} usagePercentage={"cpu":0.75,"memory":0.33,"pods":3.64}
│ I0912 09:17:02.750371       1 nodeutilization.go:207] "Node is overutilized" node="cloud0406" usage={"cpu":"950m","memory":"290Mi","pods":"15"} usagePercentage={"cpu":47.5,"memory":17.32,"pods":13.64}
│ I0912 09:17:02.750414       1 nodeutilization.go:204] "Node is underutilized" node="home0122" usage={"cpu":"200m","memory":"250Mi","pods":"10"} usagePercentage={"cpu":3.33,"memory":1.62,"pods":9.09}
│ I0912 09:17:02.750448       1 lownodeutilization.go:135] "Criteria for a node under utilization" CPU=100 Mem=5 Pods=100
│ I0912 09:17:02.750483       1 lownodeutilization.go:136] "Number of underutilized nodes" totalNumber=2
│ I0912 09:17:02.750506       1 lownodeutilization.go:149] "Criteria for a node above target utilization" CPU=100 Mem=10 Pods=100
│ I0912 09:17:02.750539       1 lownodeutilization.go:150] "Number of overutilized nodes" totalNumber=1
│ I0912 09:17:02.750567       1 nodeutilization.go:261] "Total capacity to be moved" CPU=85200 Mem=10678022143 Pods=206
│ I0912 09:17:02.750655       1 nodeutilization.go:264] "Evicting pods from node" node="cloud0406" usage={"cpu":"950m","memory":"290Mi","pods":"15"}
│ I0912 09:17:02.750978       1 nodeutilization.go:267] "Pods on node" node="cloud0406" allPods=15 nonRemovablePods=15 removablePods=0
│ I0912 09:17:02.751021       1 nodeutilization.go:270] "No removable pods on node, try next node" node="cloud0406"
│ I0912 09:17:02.751082       1 profile.go:349] "Total number of pods evicted" extension point="Balance" evictedPods=0

I learned the go , looked at the source code, and Add a annotations here
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: wccloud-web-rust
  name: wccloud-web-rust
spec:
  replicas: 10
  selector:
    matchLabels:
      app: wccloud-web-rust
  template:
    metadata:
      annotations:
        "descheduler.alpha.kubernetes.io/evict": "1"
      labels:
        app: wccloud-web-rust
    spec:
      containers:
        - name: wccloud-web-rust

Now there is a new problem，After the pod was expelled, it was deployed again on the original node

│ 0913 03:02:02.597709       1 nodeutilization.go:204] "Node is underutilized" node="home0123" usage={"cpu":"600m","memory":"306Mi","pods":"9"} usagePercentage={"cpu":0.75,"memory":0.33,"pods":8.18}
│ 0913 03:02:02.597743       1 lownodeutilization.go:135] "Criteria for a node under utilization" CPU=100 Mem=2 Pods=100
│ 0913 03:02:02.597779       1 lownodeutilization.go:136] "Number of underutilized nodes" totalNumber=2
│ 0913 03:02:02.597803       1 lownodeutilization.go:149] "Criteria for a node above target utilization" CPU=100 Mem=5 Pods=100
│ 0913 03:02:02.597834       1 lownodeutilization.go:150] "Number of overutilized nodes" totalNumber=1
│ 0913 03:02:02.597863       1 nodeutilization.go:261] "Total capacity to be moved" CPU=85200 Mem=5120122060 Pods=202
│ 0913 03:02:02.597902       1 nodeutilization.go:264] "Evicting pods from node" node="cloud0406" usage={"cpu":"950m","memory":"290Mi","pods":"15"}
│ 0913 03:02:02.598126       1 nodeutilization.go:267] "Pods on node" node="cloud0406" allPods=15 nonRemovablePods=13 removablePods=2
│ 0913 03:02:02.598167       1 nodeutilization.go:274] "Evicting pods based on priority, if they have same priority, they'll be evicted based on QoS tiers"
│ 0913 03:02:02.688816       1 evictions.go:170] "Evicted pod" pod="default/wccloud-web-rust-68ddb477d9-267ss" reason="" strategy="LowNodeUtilization" node="cloud0406" profile="ProfileName"
│ 0913 03:02:02.688910       1 nodeutilization.go:316] "Evicted pods" pod="default/wccloud-web-rust-68ddb477d9-267ss"
│ 0913 03:02:02.688937       1 nodeutilization.go:341] "Updated node usage" node="cloud0406" CPU=950 Mem=304087040 Pods=14
│ 0913 03:02:02.857815       1 evictions.go:170] "Evicted pod" pod="default/wccloud-web-rust-68ddb477d9-dx9hw" reason="" strategy="LowNodeUtilization" node="cloud0406" profile="ProfileName"
│ 0913 03:02:02.857912       1 nodeutilization.go:316] "Evicted pods" pod="default/wccloud-web-rust-68ddb477d9-dx9hw"
│ 0913 03:02:02.857943       1 nodeutilization.go:341] "Updated node usage" node="cloud0406" CPU=950 Mem=304087040 Pods=13
│ 0913 03:02:02.858015       1 profile.go:349] "Total number of pods evicted" extension point="Balance" evictedPods=2
│ 0913 03:02:02.858171       1 descheduler.go:170] "Number of evicted pods" totalEvicted=2
│ 0913 03:02:02.858486       1 reflector.go:302] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:160
│ 0913 03:02:02.858777       1 tlsconfig.go:255] "Shutting down DynamicServingCertificateController"
│ 0913 03:02:02.858831       1 event_broadcaster.go:279] "Unable to write event (may retry after sleeping)" err="Post "https://10.96.0.1:443/apis/events.k8s.io/v1/namespaces/default/events\": context canceled"
│ 0913 03:02:02.858942       1 secure_serving.go:258] Stopped listening on :10258
│ 0913 03:02:02.859024       1 reflector.go:302] Stopping reflector *v1.Node (0s) from k8s.io/client-go/informers/factory.go:160
│ 0913 03:02:02.859079       1 reflector.go:302] Stopping reflector *v1.Namespace (0s) from k8s.io/client-go/informers/factory.go:160

@hwyrq glad you figured it out, for future reference the docs lists the reasons why a pod isn't evictable and you can set --v=4 to log the specific reason why a pod wasn't evicted

source: gharchive/issue

authors: damemi hwyrq

repo: kubernetes-sigs/descheduler


text:
docs: guide index updated

What type of PR is this?

/kind documentation
What this PR does / why we need it:
The standard release channel now includes GA resources as well, and the guide has been updated accordingly.
Which issue(s) this PR fixes:

Fixes #
Does this PR introduce a user-facing change?:

NONE

/approve

/lgtm

source: gharchive/pull-request

authors: mlavacca tao12345666333 youngnick

repo: kubernetes-sigs/gateway-api

license_type: permissive

license_source: github-api


text:
Added failed jobs in replicated job statuses

Fixes #170
/kind feature

@stuton please sign CLA following the instructions linked above https://github.com/kubernetes-sigs/jobset/pull/178#issuecomment-1579259232

/check-cla

/retest

authors: danielvegamyhre stuton

repo: kubernetes-sigs/jobset


text:
Kubernetes Networking E2E Test Failures

Currently, we are noticing good number of failures for Kubernetes E2E networking SIG tests. We need to triage and hopefully resolve them soon.

Closing this issue as based on further investigation it turned out that none of the networking SIG related failures were due to scheduling (Firmament or default scheduler). There were some environment related issues.

repo: kubernetes-sigs/poseidon


text:
[stable/openvpn] add kubevpn simple CLI

The pod now generates the kubevpn CLI so you don't overflow your terminal with so many lines.
Copy once from the pod, use anywhere (if you want it... with the correct kubectl context, of course)

/cc @jfelten

nice! - Works like a charm
A couple things:
1.) Increment minor version in Chart.yaml
2.) Can you add a section in README.md that shows the kubevpn script in detail and how to use it.
Otherwise looks good.

@jfelten Version bumped.
What do you mean with a more detailed explanation? There is some script usage at the Readme already.
Thanks for the time to review it, man :)

@pfeodrippe Are you going to come up with the suggested improvement?

@unguiculus I'll catch on it until the next week... sorry for the delay

Closing as stale. Please re-open if/when you have a chance to work on this.

authors: jfelten pfeodrippe unguiculus viglesiasce

repo: kubernetes/charts


text:
Parameterize postgres chart's imagePullPolicy.

The postgres chart is unusual in its imagePullPolicy being hardcoded to Always. We probably don't want to do this by default unless there are really compelling reasons to do so, especially for stable charts.
This PR parameterizes the imagePullPolicy in a way that is consistent with a growing number of our charts. The default is also now the Kubernetes default of IfNotPresent.
:eyeglasses: @swordbeta @viglesiasce @prydonius

The same could be done for the mysql chart; I took that one as an example when creating this one.

Thanks @gtaylor. I'll review

lgtm! thanks @gtaylor, looks like @swordbeta got to mysql in #196 :)

authors: gtaylor lachie83 prydonius swordbeta


text:
[stable/fluent-bit] Cater for full set of Elastic Output Plugin parameters (Auth & TLS)

Caters for the full set of parameters for the Fluent-Bit Elastic Output Plugin. Useful when logging to an external Elastic platform.
Parameters include:

X-Pack Authentication
TLS

Mentioning maintainers
@kfox1111
@edsiper

/ok-to-test
/lgtm

/lgtm

/assign @unguiculus
Danke schön Reinhard

@timstoop @unguiculus update RBAC and ServiceAccount to standards. Merged with master again and bumped the chart version

Please bump the minor version. Also, the cluster role binding needs to be updated for the service account.

@timstoop @unguiculus bumped minor version and set ClusterRoleBinding to use serviceAccountName

/lgtm

authors: MnrGreg timstoop unguiculus


text:
Update Dask to version 0.18.1

What this PR does / why we need it:
Updates the Dask chart to the latest stable version of Dask (0.18.1).
Special notes for your reviewer:
Tested by running a non-trivial job (scanpy on 1 million rows of zarr data using 5 worker nodes) on a Kubernetes cluster.

I've signed the CLA now. /check-cla

/assign @foxish

Any news on this PR?

/ok-to-test
/lgtm

/lgtm

authors: davidkarlsen jakirkham tomwhite unguiculus


text:
Remove MAINTAINERS.md

What this PR does / why we need it:
The Kubernetes development community favors OWNERS with approver and reviewer entries over maintainers file and entries. Remove outdated and unmaintained MAINTAINERS.md file.
Which issue this PR fixes
NONE
Special notes for your reviewer:
https://github.com/kubernetes/community/blob/master/community-membership.md#maintainer
Release note:

NONE

/approve
/lgtm

authors: dims hogepodge

repo: kubernetes/cloud-provider-openstack

license_source: bigquery


text:
Remove unnucessary log and adjust some logic

actually, err is not nil means node is not "",so
no need to double check here to make logic complicate
What this PR does / why we need it:
minor refactory
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
minor refactory to remove complex
Release note:
NONE

/ok-to-test

/override openlab/cloud-provider-openstack-acceptance-test-flexvolume-cinder

/hold
@jichenjc please respond to @adisky 's question

/assign @adisky @hogepodge

@jichenjc please address the comment and rebase the PR.

/hold cancel

@jichenjc Please rebase, i marked my comment as resolved

done :)

@mrhillsman any insight on why all of the jobs are failing? It looks like it could be a volumes problem?

@hogepodge let me have a look. it looks like it may have been an issue with volume just at the moment of trying to write some dependencies to disk and then it resolved. i am not sure at the moment but hope to have some info soon.

/retest

Build failed.

cloud-provider-openstack-acceptance-test-flexvolume-cinder  : FAILURE in 35m 10s

2019-06-04 18:53:01.636203 | ubuntu-xenial | pkg/csi/cinder/nodeserver.go:313:2: undefined: glog
2019-06-04 18:53:01.645719 | ubuntu-xenial | Makefile:77: recipe for target 'cinder-csi-plugin' failed
2019-06-04 18:53:01.645903 | ubuntu-xenial | make: *** [cinder-csi-plugin] Error 2
2019-06-04 18:53:02.129036 | ubuntu-xenial | ERROR

Build failed.

cloud-provider-openstack-acceptance-test-keystone-authentication-authorization  : FAILURE in 25m 23s

2019-06-04 18:41:46.192715 | ubuntu-xenial | pkg/csi/cinder/nodeserver.go:313:2: undefined: glog
2019-06-04 18:41:46.212963 | ubuntu-xenial | Makefile:77: recipe for target 'cinder-csi-plugin' failed
2019-06-04 18:41:46.213072 | ubuntu-xenial | make: *** [cinder-csi-plugin] Error 2

Build failed.

cloud-provider-openstack-acceptance-test-standalone-cinder  : POST_FAILURE in 27m 40s

2019-06-04 18:43:52.907201 | ubuntu-xenial | pkg/csi/cinder/nodeserver.go:313:2: undefined: glog
2019-06-04 18:43:52.915088 | ubuntu-xenial | Makefile:77: recipe for target 'cinder-csi-plugin' failed
2019-06-04 18:43:52.915207 | ubuntu-xenial | make: *** [cinder-csi-plugin] Error 2

@hogepodge i believe the volume issue was temporary as mentioned. i have checked all of the jobs and except for the unittest they are all failing due to:
pkg/csi/cinder/nodeserver.go:313:2: undefined: glog
Makefile:77: recipe for target 'cinder-csi-plugin' failed
make: *** [cinder-csi-plugin] Error 2

sorry，I rebase the code and forget to update the glog to klog

/approve

/test cloud-provider-openstack-acceptance-test-e2e-conformance

@adisky can you help to lgtm ?

/lgtm

authors: adisky dims hogepodge jichenjc mrhillsman


text:
Fix the user guide to work for namespaced clusters.

/sig multicluster

/lgtm

authors: perotinus pmorie

repo: kubernetes/cluster-registry


text:
glog doesn't like it when non-standard flag package is used

Note the ERROR: logging before flag.Parse:
client-gen \
>         --input-base "github.com/atlassian/smith/pkg/apis/" \
>         --input "smith/v1" \
>         --clientset-path "github.com/atlassian/smith/pkg/client/clientset_generated/" \
>         --clientset-name "clientset"

ERROR: logging before flag.Parse: F1029 14:26:52.019063   88249 client_generator.go:312] Failed loading boilerplate: open /Users/ash2k/gopath/src/k8s.io/kubernetes/hack/boilerplate/boilerplate.go.txt: no such file or directory
goroutine 1 [running]:
github.com/golang/glog.stacks(0xc42000e000, 0xc4264e8c60, 0xc1, 0x109)
        vendor/github.com/golang/glog/glog.go:769 +0xcf
github.com/golang/glog.(*loggingT).output(0x148ffa0, 0xc400000003, 0xc4253b5ce0, 0x146030e, 0x13, 0x138, 0x0)
        vendor/github.com/golang/glog/glog.go:720 +0x345
github.com/golang/glog.(*loggingT).printf(0x148ffa0, 0xc400000003, 0x1306ffd, 0x1e, 0xc42ad21928, 0x1, 0x1)
        vendor/github.com/golang/glog/glog.go:655 +0x14c
github.com/golang/glog.Fatalf(0x1306ffd, 0x1e, 0xc42ad21928, 0x1, 0x1)
        vendor/github.com/golang/glog/glog.go:1148 +0x67
k8s.io/code-generator/cmd/client-gen/generators.Packages(0xc424dfc2a0, 0xc420010240, 0x12ff74b, 0x6, 0xc424dfc2a0)
        vendor/k8s.io/code-generator/cmd/client-gen/generators/client_generator.go:312 +0xdf
k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc420010240, 0xc420080f00, 0x12ff74b, 0x6, 0x13151b8, 0x1, 0x4)
        vendor/k8s.io/gengo/args/args.go:176 +0x1d4
main.main()
        vendor/k8s.io/code-generator/cmd/client-gen/main.go:199 +0x65e

I just encountered this as well
/remove-lifecycle stale

/reopen
/remove-lifecycle rotten
/lifecycle frozen

@ash2k Is this still relevant with klog?

I haven't tested but I guess it is not. Worth testing before closing I think. I haven't updated all our stuff to use 1.13/klog libraries - this is in progress. Once we are there, I can see it this issue still exists.

authors: ash2k frankgreco nikhita

repo: kubernetes/code-generator


text:
bug fix: add node state check in getting node list

fixed ref https://github.com/kubernetes/heapster/issues/787
when structuring node list used for getting metrics, heapster needs to check node's state.

Can one of the admins verify this patch?

This is a good check but i think there must be some other problem too in the scraping loop in the case where a node becomes unavailable during scrape?

yes, i agree.
When a node becomes unavailable during scrape, heapster will have some other problem. Like this:
E0107 04:32:55.001706    8037 kube_nodes.go:59] Failed to get container stats from Kubelet on node "8.8.8.239"

Because, the unnormal node still exists in cache, and heapster will get metrics as before.
But I think when the node is delete from cache, in runGc(). Heapster will work normally.

LGTM
Thanks for the change, although I'm not sure whether it actually fixes #787, so no closing the issue before it's verified.

add to whitelist

@k8s-bot ok to test

thanks, @piosz

Fail not related.

authors: cadvisorJenkinsBot huangyuqi jimmidyson piosz

repo: kubernetes/heapster


text:
Failed at the kubernetes-ui@0.3.0 poststart script 'gulp'.

need some help, cloned the master repo, npm install and npm start in master dir, yet I still get this error. any ideas? thanks
➜  master git:(master) ✗ npm start

> kubernetes-ui@0.3.0 prestart /Users/jeane/Development/go/src/kube-ui/master
> bower install

> kubernetes-ui@0.3.0 start /Users/jeane/Development/go/src/kube-ui/master
> npm install

> kubernetes-ui@0.3.0 poststart /Users/jeane/Development/go/src/kube-ui/master
> gulp

[18:11:47] Using gulpfile ~/Development/go/src/kube-ui/master/gulpfile.js
[18:11:47] Starting 'sync group6:0'...
[18:11:47] Starting 'clean'...
[18:11:47] Finished 'clean' after 18 ms
[18:11:47] Finished 'sync group6:0' after 19 ms
[18:11:47] Starting 'sync group6:1'...
[18:11:47] Starting 'sync group7:0'...
[18:11:47] Starting 'copy:shared-assets'...
[18:11:47] Finished 'copy:shared-assets' after 44 ms
[18:11:47] Finished 'sync group7:0' after 45 ms
[18:11:47] Starting 'sync group7:1'...
[18:11:47] Starting 'copy:components'...
[18:11:47] Tested 26 tests, 26 passes, 0 failures: PASS
[18:11:47] Finished 'copy:components' after 67 ms
[18:11:47] Finished 'sync group7:1' after 67 ms
[18:11:47] Starting 'sync group7:2'...
[18:11:47] Starting 'sync group3:0'...
[18:11:47] Starting 'content:html:app'...
[18:11:47] Finished 'content:html:app' after 11 ms
[18:11:47] Finished 'sync group3:0' after 12 ms
[18:11:47] Starting 'sync group3:1'...
[18:11:47] Starting 'content:html:views'...
[18:11:47] Finished 'content:html:views' after 10 ms
[18:11:47] Finished 'sync group3:1' after 11 ms
[18:11:47] Starting 'content:html'...
[18:11:47] Finished 'content:html' after 2.49 μs
[18:11:47] Finished 'sync group7:2' after 23 ms
[18:11:47] Starting 'sync group7:3'...
[18:11:47] Starting 'sync group1:0'...
[18:11:47] Starting 'scripts:vendor:base'...
[18:11:47] Tested 16 tests, 16 passes, 0 failures: PASS
[18:11:47] Finished 'scripts:vendor:base' after 34 ms
[18:11:47] Finished 'sync group1:0' after 35 ms
[18:11:47] Starting 'sync group1:1'...
[18:11:47] Starting 'scripts:vendor:app'...
[18:11:47] Tested 4 tests, 4 passes, 0 failures: PASS
[18:11:47] Finished 'scripts:vendor:app' after 8.3 ms
[18:11:47] Finished 'sync group1:1' after 8.73 ms
[18:11:47] Starting 'scripts:vendor'...
[18:11:47] Finished 'scripts:vendor' after 2.73 μs
[18:11:47] Finished 'sync group7:3' after 44 ms
[18:11:47] Starting 'sync group7:4'...
[18:11:47] Starting 'sync group0:0'...
[18:11:47] Starting 'bundle-manifest'...
[18:11:47] Finished 'bundle-manifest' after 898 μs
[18:11:47] Finished 'sync group0:0' after 1.37 ms
[18:11:47] Starting 'sync group0:1'...
[18:11:47] Starting 'bundle-manifest-routes'...
[18:11:47] Finished 'bundle-manifest-routes' after 1.16 ms
[18:11:47] Finished 'sync group0:1' after 1.5 ms
[18:11:47] Starting 'sync group0:2'...
[18:11:47] Starting 'sync group2:0'...
[18:11:47] Starting 'config:base'...
[18:11:47] Finished 'config:base' after 5.39 ms
[18:11:47] Finished 'sync group2:0' after 5.68 ms
[18:11:47] Starting 'sync group2:1'...
[18:11:47] Starting 'config:copy'...
[18:11:47] ✗ FAIL: Missing 1 expected files: shared/config/development.json
[18:11:47] Tested 1 tests, 0 passes, 1 failures: FAIL
Error in plugin 'gulp-expect-file'
Failed 1 expectations

npm ERR! Darwin 14.4.0
npm ERR! argv "node" "/usr/local/bin/npm" "start"
npm ERR! node v0.12.7
npm ERR! npm  v2.11.3
npm ERR! code ELIFECYCLE
npm ERR! kubernetes-ui@0.3.0 poststart: `gulp`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the kubernetes-ui@0.3.0 poststart script 'gulp'.
npm ERR! This is most likely a problem with the kubernetes-ui package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     gulp
npm ERR! You can get their info via:
npm ERR!     npm owner ls kubernetes-ui
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/jeane/Development/go/src/kube-ui/master/npm-debug.log

@preillyme can you check this please?

@Hokutosei I'll check shortly. Thanks for the report.

@Hokutosei @preillyme had the same error after the git clone. This line helps:
cp master/shared/config/development.example.json master/shared/config/development.json

But IMO the build should be green after a fresh clone.

@Hokutosei it should be fixed now can you confirm?

@preillyme  @SebastianM
Greetings,
its working very well now! :) thank you guys :D kube-ui looks really awesome

@preillyme @SebastianM
Greetings,
its working very well now! :) thank you guys :D kube-ui looks really awesome

authors: Hokutosei SebastianM cz-soliva-jeanepaul preillyme

repo: kubernetes/kube-ui


text:
weird behaviour of kubectl debug set-image under ohmyzsh

What would you like to be added:

change the example of set-image option
kubectl debug mypod --copy-to=my-debugger --set-image="*=busybox"
kubectl debug mypod -it --copy-to=my-debugger --image=debian --set-image="app=app:debug,sidecar=sidecar:debug"

Why is this needed:
when i follow the examples in  kubectl debug -h, when i try to set-image for debug containers, a mistake occured like following:

and i found a releated issue of ohmyzsh: https://github.com/ohmyzsh/ohmyzsh/issues/31
after add double quote, it worked as expecte.

so i suggest that correct the example.

This issue is not a kubectl issue, many commands have the same issue when using ohmyzsh, did you consider this comment?
https://github.com/ohmyzsh/ohmyzsh/issues/31#issuecomment-359728582

unsetopt nomatch

This issue is not a kubectl issue, many commands have the same issue when using ohmyzsh, did you consider this comment? ohmyzsh/ohmyzsh#31 (comment)

unsetopt nomatch
yeah, maybe the maintainer of kubectl can update the example, this will be useful for people who not very familiar with kubectl. When i first use kubectl debug command, it make me confused.

Going to close the issue but we would accept a PR if you want to make one and tag me.
/close

authors: ah8ad3 eddiezane warjiang

repo: kubernetes/kubectl


text:
Search Results page is quite ugly

http://kubernetes.github.io/docs/search/?q=nodes
Too much whitespace, results not aligned with header.
@CreaturesInUnitards

Did a first pass at fixing this — let me know what you think.

Needs more work.

Example: http://kubernetes.github.io/docs/search/?q=nodes
cc @CreaturesInUnitards

Fixed with commit af265623af5c9e2382b417bd4c8499f29a632624

authors: CreaturesInUnitards johndmulhausen


text:
I wish /docs/user-guide/logging-demo/ existed

I wish /docs/user-guide/logging-demo/ existed

http://kubernetes.io/docs/user-guide/logging/

http://kubernetes.io/docs/user-guide/logging/overview/

authors: gildean jaredbhatti so0k


text:
Deleted bullet points (•)

They're unnecessary punctuation and were rendering as boxes (see screenshot.

This change is

Deploy preview ready!
Built with commit 1d4d0b50ba6ac496bf9ade86a8755ce80318ea38
https://deploy-preview-4341--k8sio-v1-6.netlify.com

/unassign

@chenopis I don't think this needs tech review

authors: chenopis cody-clark pwittrock


text:
Adjust latency pods requirements

Ref #1311
/assign @mborsz

/lgtm

authors: mborsz wojtek-t

repo: kubernetes/perf-tests


text:
Unofficial list of FQDNs or ability to download as tar archive?

Is there an existing issue for this?

[X] I have searched the existing issues

What did you expect to happen?
I was previously proxying k8s.gcr.io with Artifactory which worked fine in my restricted environment.
With the new registry.k8s.io the recommended solution is mirroring which is not feasible because I would need to have the FQDNs allowlisted in order to mirror them to my own local repository.
Debugging Information
n/a
Anything else?
Is there a paid service or some approach I can use to move forward here? I would be fine with downloading a tar archive and importing into my local repository if that was an option.
Thank you for any thoughts you can provide.
Code of Conduct

[X] I agree to follow this project's Code of Conduct

There is no list of FQDNs, this is covered by http://registry.k8s.io/#stability

With the new registry.k8s.io the recommended solution is mirroring which is not feasible because I would need to have the FQDNs allowlisted in order to mirror them to my own local repository.

You need to pull via a less restricted environment and then copy it to your registry or let through the observed FQDNs long enough to populate the mirror.

Is there a paid service or some approach I can use to move forward here? I would be fine with downloading a tar archive and importing into my local repository if that was an option.

You can export images to an archive format using something like crane, but you will have to pick an environment that is not network restricted and run the export there. Any unrestricted network environment will do.

We don't maintain a list because the list is subject to change at any time anyhow in order to keep the registry sustainable and there does not appear to be any standard for maintaining and communicating such a list.
More importantly we don't want to setup a false expectation that there is a stable list or that we'll hesitate to leverage new hosts immediately when resources become available or we otherwise need to reconfigure the infrastructure. The team of contributors operating this infrastructure will leverage new hosts, regions, etc without notice to keep things affordable and maintainable.
You could identify the current set of FQDNs that apply to your traffic by observing what is blocked by your firewall and adding them or by inspecting the code and configs, but they're subject to change at any time.
An alternative would be using a Kubernetes distribution that provide and host images themselves, e.g. most large vendor provided offerings like EKS/AKS/GKE/Anthos/...
There are many artifactory instances currently spamming registry.k8s.io logs with attempts to pull content we don't even host 🙃 , I think typically they're configured to have less restricted network access.
Previously: #127 #145

Also https://github.com/kubernetes/registry.k8s.io/issues/237

Thank you for the information. Unfortunately in my environment transferring via a less restrictive environment means sneaker netting a CD which is quite a time consuming process. I appreciate the feedback regardless and will try to come up with something on my end.

Is there a list of images somewhere that are hosted on the registry?

Is there a list of images somewhere that are hosted on the registry? It seems like that would make mirroring it easier.

The images are controlled by a yaml spec so you can find those in the registry.k8s.io directory in https://github.com/kubernetes/k8s.io repo however we have a LOT of images and you should really only mirror the images you plan to use. To determine that for popular tools see the mirroring guide linked from the README in this repo.

Also I don't know if you all have any plans to offer a paid service for people in similar situations? It seems like this issue is coming up a lot. Maybe an idea to help fund the project would be the ability to pay for a stable registry endpoint? just a thought

Kubernetes and the CNCF aren't really equipped for that but there are many vendors participating in the project that offer paid distributions that they host.

authors: BenTheElder TerryHowe donaldm


text:
specify crane version in deploy.sh

fixes go attempting to use the module version and finding we don't have all the deps in go.sum
latest is fine, we're using an extremely stable command.

/lgtm

authors: BenTheElder dims


text:
Translate services-networking/connect-applications-service.md in Korean.

This is a Feature Request
What would you like to be added
Translate services-networking/connect-applications-service.md in Korean.
Why is this needed
The following files are not translated into Korean.

[ ] /en/docs/concepts/services-networking/connect-applications-service.md

Comments
Page to update: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
I am translating this document.
The translated document will be submitted to dev-1.16-ko.7 or dev-1.17-ko.1 branch.
If it is translated very quickly, it is submitted to dev-1.16-ko.6.
/language ko

/assign

repo: kubernetes/website


text:
Localize “Hello Minikube” into Vietnamese

This is a Bug Report

Problem:
https://kubernetes.io/vi/docs/tutorials/hello-minikube/ is a missing page; however, https://kubernetes.io/docs/tutorials/hello-minikube/ is part of the required content for a localization.
Proposed Solution:
Localize “Hello Minikube” into Vietnamese (Tiếng Việt).
Page to Update:
https://kubernetes.io/vi/docs/tutorials/hello-minikube/

Additional Information:
/language vi
/kind bug

/remove-lifecycle rotten


text:
Mention scheduling profiles in Configure Multiple Schedulers

This is a Feature Request

What would you like to be added
https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/ should signpost readers to
https://kubernetes.io/docs/reference/scheduling/config/#multiple-profiles as they
Why is this needed
Readers might want to use an additional scheduling profile rather than implement a fully custom scheduler.
Comments
You can also use a custom scheduler configuration, or custom container image, for the cluster's main scheduler by modifying its static pod manifest on the relevant control plane host.
/language en
/kind feature
/help
/sig scheduling

/assign

/triage accepted

/remove-lifecycle rotten

Fixed in PR #27027

/close

authors: kbhawkey sftim tengqm vaibhav2107


text:
Improvement for k8s.io/docs/tasks/tools/install-kubectl-linux/

It would be good to count link/instructions to install curl: "sudo apt install curl"

Instead of providing (Debian-specific) instructions, we can - maybe should - explicitly state that the curl tool is a prerequisite for the Install kubectl binary with curl on Linux method.
Readers should be able to figure out for themselves how to install curl; if not, I dare say they won't find Kubernetes easy overall.

/language en

/assign @Anilkumar56

/triage accepted

/remove-lifecycle stale

/close

authors: CarlosMendozaB RA489 sftim


text:
Getting started page format is not synced with other pages

This is a Bug Report
Problem:

All the subsections should be described and linked to correspond pages.
a)  Like Learning environment and Production environment , best practices as well described in this page.
b)  Like concepts  page all subsection should be linked to corresponding pages.

remove Learn about best practice for cluster setup from What's next
Reason : As per point 1, it would be a subsection of this page

Incorrect context in  What's next section
[Download and install tools including kubectl] to be changed to [Download and install tools such as  kubectl  and kind etc]

The sentence between Learn about best practices for cluster setup and Learn to set up clusters with Windows nodes  in the  What's next section should be  removed and copied to Windows in Kubernetes page.

Problem:
Proposed Solution:
Page to Update:
https://kubernetes.io/docs/setup/

/language en
/sig docs
/assign

3. Incorrect context in  What's next section

I don't think that content is incorrect.

4. The sentence between Learn about best practices for cluster setup and Learn to set up clusters with Windows nodes  in the  What's next section should be  removed and copied to Windows in Kubernetes page.

I disagree here. People might want to learn that Kubernetes is designed for its control plane to run on Linux (only), and that nodes but not the control plane can also run Windows.
/priority awaiting-more-evidence
I'd like to be convinced about why we should make the changes you're suggesting @tamilselvan1102
Was there something you didn't enjoy about the existing page and how it is written?
@tami

[Download and install tools including kubectl] under this topic having various tools details are there to be installed. But here specifically mentioned only kubectl. That's why thought to rewrite.

[What's next] section have to focus on pinpoint to features not detailed description. That's why thought to move the description to Windows in Kubernetes section. Otherwise move the content as attached file.

@sftim
could you please check the above proposal?

I'm afraid that I don't fully understand what you're proposing @tamilselvan1102
You're welcome to open a pull request to illustrate how you think the page should look; that might be easier. Please do consider that we might prefer the existing page (I remember updating it to the current format within the last couple of years, and at the time I was happy with those improvements). Of course, there are lots of reviewers and there is more than one opinion, not just mine.
So long as you're willing for the answer to be "no", we're very very happy to consider making changes.

@sftim
Sorry, This was an oversight.
I thought to make only bullet points in the What's next section.


text:
Job error handling has confusing subtitles

/sig apps
Problem:
We have the following structure:

Handling Pod and container failures

Pod backoff failure policy

Advanced usage

Pod failure policy

The headings are closely related and have similar names that are difficult to differentiate
Proposed Solution:

Handling Pod and container failures

Pod failure backoff limit
Pod failure policy

Page to Update:
https://kubernetes.io/docs/concepts/workloads/controllers/job/

/assign

/retitle Confusing headings in Job concept page

/close
Closing the issue as https://github.com/kubernetes/website/pull/40999 is merged.

authors: alculquicondor mimowo sftim tamilselvan1102


text:
Unable to view POD_NAME environment variable using PowerShell on Windows

Get the name of the Pod and store it in the POD_NAME environment variable:
export POD_NAME="$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')"
echo "Name of the Pod: $POD_NAME"
Windows 11. using Powershell 5.1
export isnt a command so I have tried "    $env:POD_.... " for powershell and "set:...." in cmd
when using Powershell I get error:
error: error parsing template {{range .items}}{{.metadata.name}}{{\n}}{{end}}, template: output:1: unexpected "\" in command
export : The term 'export' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path
is correct and try again.
At line:1 char:1

export POD_NAME=$(kubectl get pods -o go-template --template '{{range ...

  + CategoryInfo          : ObjectNotFound: (export:String) [], CommandNotFoundException
  + FullyQualifiedErrorId : CommandNotFoundException

and when I use cmd with set, it proceeds with the command set:...... but when I do the echo call  it's like this:

echo Name of the Pod: $POD_NAME
Name of the Pod: $POD_NAME

any ideas why?

Page related to the issue: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/
/retitle Unable to view POD_NAME environment variable using PowerShell on Windows
/language en
/kind support

@rdt-nick Could your try echo %POD_NAME% to see if it displays something useful?

Hi @rdt-nick, you can find about setting up environment variable in PoweShell here -
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_environment_variables?view=powershell-7.3

We could clarify that https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/ is written for people using a POSIX-style shell.

In Case of PowerShell
#set environment variable
$env:POD_NAME="$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')"
#print environment variable
$env:POD_NAME

In case Windows cmd
set environment variable
set POD_NAME="$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')"
print an environment variable
set POD_NAME
or
echo %POD_NAME%

/close
This issue is more about how to use PowerSell for environment variables.
It is beyond the scope of k8s documentation.

@tengqm This is not only an issue with environment variables. Just executing this command using Powershell:
kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'
Will yield this error:
error: error parsing template {{range .items}}{{.metadata.name}}{{\n}}{{end}}, template: output:1: unexpected "\\" in command

This is again an issue caused by the PowerShell parsing of quotes and control sequence. I don't think is it possible to provide two flavored command line guidance everywhere. The two Shells are so different.

For PowerShell users, the simplest way to fix this is to use the following command:
$env:POD_NAME=$(kubectl get pods -o name).replace('pod/', '')

Using windows terminal, this should work:
for /f "delims=*" %i in ('kubectl get pods -o go-template --template "{{range .items}}{{.metadata.name}}{{\"\n\"}}{{end}}"') do set POD_NAME=%i
echo Name of the Pod: %POD_NAME%

authors: PowerOfCreation dakotacookenmaster dipesh-rawat rc6600 rdt-nick sftim tamilselvan1102 tengqm utkarsh-singh1


text:
Please enhance the section description at tutorials/stateful-application/basic-stateful-set/#on-delete

This is a Feature Request

What would you like to be added
At this section, it introduces using OnDelete by the command below.
kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"type":"OnDelete"}}}'

But in the case of a server like minikube, where defaulting spec.updateStrategy.rollingUpdate field in the create step,
it shows an error like this.
➜  minikube kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"type":"OnDelete"}}}'
The StatefulSet "web" is invalid: spec.updateStrategy.rollingUpdate: Invalid value: apps.RollingUpdateStatefulSetStrategy{Partition:0, MaxUnavailable:(*intstr.IntOrString)(nil)}: only allowed for updateStrategy 'RollingUpdate'

Therefore I recommend to change this part like this.
//As-is
kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"type":"OnDelete"}}}'
//To-be
kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"type":"OnDelete","rollingUpdate":null}}}'

I think it is important because most newbie of k8s will use minikube and they can suffer from similar issues.
Comments

I referred this page for debugging this error.

/language en

Page related to the issue : https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/

/triage accepted

Hi @Ritikaa96
How did you decide that the page needs a fix? I wasn't sure that it does.

Hi @sftim ,
As per https://github.com/kubernetes/website/pull/42767 indicating need to revise the page and review discussion on Pr: https://github.com/kubernetes/website/pull/44730#issuecomment-1890983853 i accepted the Triage .
I can definitely be wrong in case there was some pending discussion for acceptance of this issue. My great apologies in that case.

Also when I use minikube i can replicate this and as per https://github.com/kubernetes/kubernetes/issues/100151#issuecomment-801073848  this behavior is noticed earlier, if this create confusion maybe a small note for this behaviour or the same as author mentioned can clarify this.

I encountered the same problem when I learned the tutorial using k3s.


text:
PVCProtection feature not documented

This is a...
[x ] Feature Request
Problem:
design doc:  https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/postpone-pvc-deletion-if-used-in-a-pod.md
implementation: kubernetes/kubernetes#55824
Proposed Solution:
Add the following documentation to encourage people to try it.

admission control
behavior change

Page to Update:
http://kubernetes.io/docs/concepts/storage/persistent-volumes
Kubernetes v1.9

I'm still new to these parts but I think this was covered in #6415

@aburdenthehand thanks, this in fact can be closed now.

authors: aburdenthehand tengqm


text:
zh-trans: add overview/_index.md

zh-trans: add overview/_index.md

/lgtm
/approve

authors: AdamDang tengqm


text:
Blog NCW Shanghai: image captions and copyediting

Use Hugo figure shortcodes to display image captions, along with copyediting fixes

https://deploy-preview-11590--kubernetes-io-master-staging.netlify.com/blog/2018/12/05/new-contributor-workshop-shanghai/

/lgtm
/approve

Thanks @idealhack ! :+1: :100:

authors: idealhack kbarnard10 xiangpengzhao


text:
Fix word 'anda' to 'kamu' for Indonesian translation

Fixing words suggested in Kubernetes contributor workshop in Openinfra Day Surabaya Indonesia
/language.id
/lgtm
/cc @girikuncoro

Hi @sevenzy, thanks for the PR! Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA so that we can review this PR, thanks!

authors: sevenzy tfogo


text:
Update custom-resource-definitions.md

The example on line 746-754 has an image tag; with the image tag, the validation will fail with the error "validating data: ValidationError(CronTab.spec): unknown field "image"".  Solution : Adding the image field on the CustomResourceDefinitions.

It seems like an error of ValidationError(CronTab.spec): unknown field "image" would indicate that the field was specified unnecessarily.
👋 @kubernetes/sig-api-machinery-api-reviews would it be possible to get a technical /lgtm on this minor change? Thanks!

Ping @sttts . Would you have time to review this change? Thank you.

I think it's fine to fix the problem either by adding this or removing the reference.

/lgtm
/approve

authors: jimangel kbhawkey lavalamp mokua zacharysarah


text:
[zh] Rework debugging application task

The current translation is missing English context, with content not
updated to match English source. Some translation were inaccurate.
This PR fixes these problems, along with links.

/lgtm
/approve

authors: tengqm xichengliudui


text:
Update outdated files in the dev-1.20-ko.3 branch (2)

#26107
Update outdated files in the upstream/dev-1.20-ko.3 branch.
16 files are updated

[x]  M24.  content/en/docs/reference/command-line-tools-reference/feature-gates.md  | 2(+XS) 1(-)
[x]  M25.  content/en/docs/reference/kubectl/cheatsheet.md  | 5(+XS) 0(-)
[x]  M26.  content/en/docs/reference/kubectl/conventions.md  | 16(+S) 17(-)
[x]  M27.  content/en/docs/reference/kubectl/docker-cli-to-kubectl.md  | 1(+XS) 6(-)
[x]  M29.  content/en/docs/setup/production-environment/container-runtimes.md  | 75(+M) 11(-)
[x]  M31.  content/en/docs/tasks/access-application-cluster/connecting-frontend-backend.md  | 51(+M) 40(-)
[x]  M32.  content/en/docs/tasks/manage-kubernetes-objects/kustomization.md  | 5(+XS) 4(-)
[x]  M33.  content/en/docs/tutorials/kubernetes-basics/deploy-app/deploy-intro.html  | 1(+XS) 3(-)
[x]  M34.  content/en/examples/application/deployment.yaml  | 1(+XS) 1(-)
[x]  M35.  content/en/examples/application/guestbook/frontend-deployment.yaml  | 1(+XS) 1(-)
[x]  M36.  content/en/examples/application/guestbook/redis-master-deployment.yaml  | 1(+XS) 1(-)
[x]  M37.  content/en/examples/application/guestbook/redis-slave-deployment.yaml  | 1(+XS) 1(-)
[x]  M38.  content/en/examples/application/mysql/mysql-deployment.yaml  | 1(+XS) 1(-)
[x]  M39.  content/en/examples/application/wordpress/mysql-deployment.yaml  | 1(+XS) 1(-)
[x]  M40.  content/en/examples/application/wordpress/wordpress-deployment.yaml  | 1(+XS) 1(-)
[x]  M41.  content/en/examples/service/access/Dockerfile  | 1(+XS) 1(-)

2 files are renamed

[x]  R1.  R091 content/en/examples/service/access/hello.yaml -> content/en/examples/service/access/backend-deployment.yaml
[x]  R2.  R095 content/en/examples/service/access/hello-service.yaml -> content/en/examples/service/access/backend-service.yaml

2 files are not deleted yet (due to build failure)

[ ]  D1.  content/en/examples/service/access/frontend.conf
[ ]  D2.  content/en/examples/service/access/frontend.yaml

2 files not updated which were already applied for Korean

[x]  M28.  content/en/docs/reference/scheduling/config.md  | 1(+XS) 1(-)
[x]  M30.  content/en/docs/tasks/access-application-cluster/configure-access-multiple-clusters.md  | 1(+XS) 1(-)

/assign @yoonian

/assign @yoonian

@pjhwa
D1. content/en/examples/service/access/frontend.conf
D2. content/en/examples/service/access/frontend.yaml
관련된 내용은 connecting-frontend-backend.md 에 포함되어 있네요. 원문을 찾아보니, 원문이 이미 변경되어 있는 상태이군요.
현재 한글화 문서가 원문과 다소 차이가 나므로, D1-D2 도 connecting-frontend-backend.md 와 함께 따로 처리하는 것이 좋을 것 같습니다.
\content/ko/docs/tasks/access-application-cluster/connecting-frontend-backend.md
:{{< codenew file="service/access/frontend.conf" >}}

content/ko/docs/tasks/access-application-cluster/connecting-frontend-backend.md
:{{< codenew file="service/access/frontend.yaml" >}}
:kubectl apply -f https://k8s.io/examples/service/access/frontend.yaml

@pjhwa
D1. content/en/examples/service/access/frontend.conf
D2. content/en/examples/service/access/frontend.yaml
관련된 내용은 connecting-frontend-backend.md 에 포함되어 있네요. 원문을 찾아보니, 원문이 이미 변경되어 있는 상태이군요.
현재 한글화 문서가 원문과 다소 차이가 나므로, D1-D2 도 connecting-frontend-backend.md 와 함께 따로 처리하는 것이 좋을 것 같습니다.
\content/ko/docs/tasks/access-application-cluster/connecting-frontend-backend.md
:{{< codenew file="service/access/frontend.conf" >}}

content/ko/docs/tasks/access-application-cluster/connecting-frontend-backend.md
:{{< codenew file="service/access/frontend.yaml" >}}
:kubectl apply -f https://k8s.io/examples/service/access/frontend.yaml

@pjhwa connecting-frontend-backend 갱신,벌써 진행해주셨네요.. ^^ 감사합니다!!

@pjhwa connecting-frontend-backend 갱신,벌써 진행해주셨네요.. ^^ 감사합니다!!

@pjhwa connecting-frontend-backend 갱신,벌써 진행해주셨네요.. ^^ 감사합니다!!

다시 원문이랑 비교해보니 파일이 추가도 되고, 파일명이 변경도 되었더라고요. :-)

@pjhwa connecting-frontend-backend 갱신,벌써 진행해주셨네요.. ^^ 감사합니다!!

다시 원문이랑 비교해보니 파일이 추가도 되고, 파일명이 변경도 되었더라고요. :-)

@pjhwa 디플로이 문제가 있네요? 확인 좀 부탁드려요~

@pjhwa 디플로이 문제가 있네요? 확인 좀 부탁드려요~

/retest

/retest

@pjhwa @seokho-son 빌드 문제가 있네요? 확인 부탁드려요~

"fatal error: concurrent map read and map write" 에러와 관련한 이슈가 있나봐요. 몇 번 재빌드하면 성공할때가 있다고 하니 다시 해보겠습니다.
https://github.com/gohugoio/hugo/issues/7293

@pjhwa @seokho-son 빌드 문제가 있네요? 확인 부탁드려요~

"fatal error: concurrent map read and map write" 에러와 관련한 이슈가 있나봐요. 몇 번 재빌드하면 성공할때가 있다고 하니 다시 해보겠습니다.
https://github.com/gohugoio/hugo/issues/7293

@yoonian 빌드 테스트는 완료된 것 같습니다 ^^ 리뷰 완료해주시면, 제가 승인 절차에 들어가겠습니다.. ^^

@yoonian 빌드 테스트는 완료된 것 같습니다 ^^ 리뷰 완료해주시면, 제가 승인 절차에 들어가겠습니다.. ^^

@seokho-son 좋습니다~ 아까 올린다는 것이 깜박했네요.
/lgtm

@seokho-son 좋습니다~ 아까 올린다는 것이 깜박했네요.
/lgtm

Thank you !
/lgtm
/approve

Thank you !
/lgtm
/approve

authors: pjhwa seokho-son yoonian


text:
[zh] sync install-kubectl-*.md

synchronize install-kubectl-macos.md file to latest

/close


text:
Change some links in ja to proper en pages

This PR updates some links in ja docs to proper en pages.
related to: #37247
If #37247 is merged, we need to update some links to those pages.

change link ja/docs/reference/command-line-tools-reference/kube-apiserver to docs/reference/command-line-tools-reference/kube-apiserver
change link ja/docs/reference/config-api/apiserver-audit.v1 to docs/reference/config-api/apiserver-audit.v1

I found 1 page for being changed by grep command.
$ grep "ja/docs/reference/command-line-tools-reference/kube-apiserver" -r ./ja -l
./ja/docs/concepts/workloads/controllers/job.md  #<-- just 1 file is matched
$ grep "ja/docs/reference/config-api/apiserver-audit.v1" -r ./ja -l
# no files are matched

That's why I updated ja/docs/concepts/workloads/controllers/job.md in this PR.

/hold
I hold this PR till #37247 will be merged.

I've confirmed that the target is only this file and the fixes are correct.
/lgtm

#37247 is merged for now
/unhold

@nasa9084
Wow, thank you for unholding this PR :smile:


text:
[zh] sync /working-with-objects/labels.md

content/zh-cn/docs/concepts/overview/working-with-objects/labels.md

/lgtm
/approve

authors: tengqm windsonsea


text:
Addition of Bullet points in sub-docs pages at side-menu

Addition of Bullet points in sub-docs pages at side-menu solving this issue https://github.com/kubernetes/website/issues/43092
(Providing link to some of those pages)
Pods page, Storage page, Authorization resources page, API Overview page etc.

Seems like a Netlify build failure, @Ayush9026 try to push another PR -
git reset
git commit --amend --no-edit
git push --force-with-lease

Explicit hold to avoid accidental merges of (current) breaking changes.
/hold

The above solution proposed does not seem to be required for the issue #43092, editing Markdown not seems to be the solution.

Thank you for your contribution to the issue. However, it seems that the changes made do not address the required modifications. If you are willing to work on the issue from scratch again, we suggest focusing on altering the actual layout rather than adding bullet points to the markdown metadata.

/close
Thank you for your efforts, @dipesh-rawat. Let us know if you have any questions!


text:
Concepts -> Configuration -> Secrets      Fix a mistake

Please tell why this is a mistake in this PR's desc. Do not leave it blank.

this section is talking about the kubernetes.io/tls Secret type, but its content mistakenly refers to kubernetes.io/ssh-auth, which is described in the previous section.

@luoxiaohi would you be willing to edit the existing description for this PR? That will help reviewers.

/retitle Fix a mistake in Secrets concept

authors: luoxiaohi sftim


text:
in 1.27, the StatefulsetAutoDeletePVC is default true

in 1.27, the StatefulsetAutoDeletePVC is default true

/cc @dipesh-rawat  Please review this modification

@SuQiucheng, The test cases failed due to Netlify build timeout failures.
please re-push the commit.
git reset
git commit --amend --no-edit
git push --force-with-lease

@SuQiucheng, The test cases failed due to Netlify build timeout failures. please re-push the commit.
git reset
git commit --amend --no-edit
git push --force-with-lease

thanks, i have done it

Thanks.
/lgtm
/approve
May need to fix release-1.27 branch as well.

Thanks. /lgtm /approve
May need to fix release-1.27 branch as well.

ok，I will do it.

/cherrypick release-1.27

@SuQiucheng Looks like the /cherrypick command doesn't work on the website repo. How about a manual PR?

@SuQiucheng @tengqm
If we fix this in the release-1.27 branch then we have to fix this in the release-1.28 branch as well.

@SuQiucheng @tengqm If we fix this in the release-1.27 branch then we have to fix this in the release-1.28 branch as well.

The current live site is 1.28, still evolving. The release-1.27 branch is an archived branch, manual cherry-pick is needed.

@SuQiucheng Looks like the /cherrypick command doesn't work on the website repo. How about a manual PR?

okk.

authors: Shubham82 SuQiucheng tengqm


text:
Fix incorrect YAML format in static-web.yaml example

page:  https://kubernetes.io/zh-cn/docs/tasks/configure-pod-container/static-pod/#configuration-files

preview page: https://deploy-preview-44172--kubernetes-io-main-staging.netlify.app/zh-cn/docs/tasks/configure-pod-container/static-pod/#configuration-files
Fixes: https://github.com/kubernetes/website/issues/44071

The preview looks good.
Thanks.
/approve
/lgtm

authors: mengjiao-liu windsonsea


text:
Disable line wrapping instead of removing new lines

When encoding a file using base64 it's simpler to just use the w flag to remove the line wrapping instead of removing new lines.

/sig auth

/lgtm
Much simpler, and what I've always used.

/lgtm cancel
based on https://github.com/kubernetes/website/pull/44575#pullrequestreview-1799556317

/remove-sig auth

I agree, this probably should not go through. I had forgotten about the macOS differences, so the existing instructions are the safest. This should probably be closed.

authors: dipesh-rawat rizerzero sftim stmcginnis


text:
Rename example pod-with-affinity-anti-affinity.yaml

This https://github.com/kubernetes/website/blob/main/content/en/examples/pods/pod-with-affinity-anti-affinity.yaml example stresses  preferred affinity weights and not the antiAffinity.  The file name and pod name may make some confusing.

@7sunarni Thanks you for your contribution.
Could you please sign the CLA before the PR can be reviewed.
You can follow the steps documented here: https://git.k8s.io/community/CLA.md

/easycla

Explicit hold until this PR only covers one language
(Ok to unhold once that's done)
/hold

/remove-language ja
/remove-language ko
/remove-language ru
/remove-language zh

/hold cancel
(PR now targets only single language)

/check-cla

/lgtm

/approve

authors: 7sunarni dipesh-rawat salaxander tengqm


text:
Update Sysdig logo and description

Submitting this PR to update the description and the logo for Sysdig on the technology partners section of the partners page. We've gone through rebranding and have launched new products since this was last updated, & want to ensure our partner listing is up to date.

This change is

CLA should be signed now.


text:
fix describtion about authz and tutorial

Fix four 404 pages
/assign @tengqm
Thanks !!!

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For 1.11 Features: set Milestone to 1.11 and Base Branch to release-1.11
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
NOTE: After opening the PR, please un-check and re-check the "Allow edits from maintainers" box so that maintainers can work on your patch and speed up the review process. This is a temporary workaround to address a known issue with GitHub.>
Please delete this note before submitting the pull request.

/lgtm
/approve

@tengqm 👋 I figured out why you got this message from k8s-ci-bot:
@tengqm: changing LGTM is restricted to assignees, and only kubernetes/website repo collaborators may be assigned issues.

It turns out that you are not a member of the Kubernetes GH organization. Not sure how that happened--but it's easy to fix! Here are the instructions for becoming a K8s member.
@steveperry-53 and I are happy to sponsor you. I will send you more details via DM on Kubernetes slack.

authors: CaoShuFeng tengqm zacharysarah


text:
🐛 Fix concurrency bugs, stop copying binding resolutions

Summary
This PR changes, primarily the binding controller, to remove the need for ResolutionBroker.GetResolution to copy data. This PR also removes some concurrency bugs in the handling of binding resolutions.
Related issue(s)
Fixes #2418

/cc @vMaroon
/cc @waltforme

/lgtm

/approve

authors: MikeSpreitzer vMaroon

repo: kubestellar/kubestellar


text:
client-side throttling in api server

Describe the bug
The VelaUX page is slow, find many client-side throttling errors in logs.
I0516 02:55:36.801103       1 request.go:665] Waited for 1.041772117s due to client-side throttling, not priority and fairness, request: GET:https://172.16.0.1:443/apis/cluster.core.oam.dev/v1alpha1?timeout=32s
{"level":"info","ts":1652669739.5503702,"caller":"rest/rest_server.go:239","msg":"request log","clientIP":"192.168.7.87","path":"/api/v1/definitions","method":"GET","status":200,"time":"1.343368ms","responseSize":87015}
{"level":"error","ts":1652669740.3559577,"caller":"usecase/rbac.go:548","msg":"get user's perm policies failure context canceled, user is admin","stacktrace":"github.com/oam-dev/kubevela/pkg/apiserver/rest/usecase.(*rbacUsecaseImpl).CheckPerm.func1\n\t/workspace/pkg/apiserver/rest/usecase/rbac.go:548\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/oam-dev/kubevela/pkg/apiserver/rest/webservice.authCheckFilter\n\t/workspace/pkg/apiserver/rest/webservice/authentication.go:115\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/oam-dev/kubevela/pkg/apiserver/rest.(*restServer).requestLog\n\t/workspace/pkg/apiserver/rest/rest_server.go:230\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/emicklei/go-restful/v3.(*Container).OPTIONSFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/options_filter.go:15\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/emicklei/go-restful/v3.CrossOriginResourceSharing.Filter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/cors_filter.go:52\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/emicklei/go-restful/v3.(*Container).dispatch\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/container.go:282\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2425\ngithub.com/emicklei/go-restful/v3.(*Container).ServeHTTP\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/container.go:300\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2879\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1930"}
{"level":"info","ts":1652669740.3560603,"caller":"rest/rest_server.go:239","msg":"request log","clientIP":"192.168.7.87","path":"/api/v1/applications/istio-gateway/records","method":"GET","status":403,"time":"3.523816416s","responseSize":53}
{"level":"error","ts":1652669740.3559704,"caller":"bcode/bcode.go:111","msg":"Business exceptions, error message: context canceled, path:/api/v1/applications/istio-gateway/triggers method:GET","stacktrace":"github.com/oam-dev/kubevela/pkg/apiserver/rest/utils/bcode.ReturnError\n\t/workspace/pkg/apiserver/rest/utils/bcode/bcode.go:111\ngithub.com/oam-dev/kubevela/pkg/apiserver/rest/webservice.(*applicationWebService).appCheckFilter\n\t/workspace/pkg/apiserver/rest/webservice/application.go:1124\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/oam-dev/kubevela/pkg/apiserver/rest/usecase.(*rbacUsecaseImpl).CheckPerm.func1\n\t/workspace/pkg/apiserver/rest/usecase/rbac.go:556\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/oam-dev/kubevela/pkg/apiserver/rest/webservice.authCheckFilter\n\t/workspace/pkg/apiserver/rest/webservice/authentication.go:115\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/oam-dev/kubevela/pkg/apiserver/rest.(*restServer).requestLog\n\t/workspace/pkg/apiserver/rest/rest_server.go:230\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/emicklei/go-restful/v3.(*Container).OPTIONSFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/options_filter.go:15\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/emicklei/go-restful/v3.CrossOriginResourceSharing.Filter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/cors_filter.go:52\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/filter.go:19\ngithub.com/emicklei/go-restful/v3.(*Container).dispatch\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/container.go:282\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2425\ngithub.com/emicklei/go-restful/v3.(*Container).ServeHTTP\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3@v3.0.0-rc2/container.go:300\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2879\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1930"}

To Reproduce

Install the KubeVela in the ASK.
Create a complex application.
Open the status page or instances page.

Expected behavior
The page loads at the normal speed.
KubeVela Version
1.3.4
Cluster information
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.3-aliyun.1", GitCommit:"2d7fa03ee32075acb101bc7286176439c3edddf1", GitTreeState:"clean", BuildDate:"2022-01-18T14:15:16Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

https://github.com/kubernetes/kubernetes/pull/107131

repo: kubevela/kubevela


text:
Document v1.VM and v1.DomainSpec

By adding comments to the structs v1.VM and v1.DomainSpec, the swagger documentation is updated. We miss documentation almost everywhere. This is important, so that we can automatically generate documentation in different formats later on.

Hints on the meaning of the fields can be found here:
https://libvirt.org/formatdomain.html

The PR #347 mostly documented v1.VM. Given that https://github.com/libvirt/libvirt-go-xml exists and we may eventually switch to it, I think the issue could be reduced to v1.VM documentation. Documenting api.DomainSpec could be wasted, easily outsourced (to the other project) effort.

authors: mpolednik rmohr

repo: kubevirt/kubevirt


text:
[flakey test] Creating a VirtualMachineInstance when virt-handler is not responsive [test_id:1634]the node controller should react

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
This test fails with:
/root/go/src/kubevirt.io/kubevirt/tests/vmi_lifecycle_test.go:609
Timed out after 180.000s.
Expected
    <v1.VirtualMachineInstancePhase>: Running
to equal
    <v1.VirtualMachineInstancePhase>: Failed
/root/go/src/kubevirt.io/kubevirt/tests/vmi_lifecycle_test.go:637

It was observed 5 times in last week: https://storage.googleapis.com/kubevirt-prow/reports/flakefinder/kubevirt/kubevirt/flakefinder-2020-06-14-168h.html#row3
It happened on k8s 1.17 and 1.18 lanes.
Example failed run: https://prow.apps.ovirt.org/view/gcs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/3540/pull-kubevirt-e2e-k8s-1.18/1271432801274564608
What you expected to happen:
It should pass.

Two of the failures observed from flakefinder look different than the others, i.e. they are not a timeout after 180 seconds, one is a panic, the other is waiting on condition "Virthandler should be ready to work" to be true.
So the flake count goes down to one on k8s-1.17 and 2 on k8s-1.18.

The panic test run is caused by an oddball race condition that we're doing to ourselves. We have OwnerReferences for things like a VMI which has a VM as a parent. if the VM is deleted, then k8s will cascade that deletion to the VMI and possibly to container disks and the like... the next thing we do is attempt to delete the same VMI -- and then we panic on an "isNotFound" error. https://github.com/kubevirt/kubevirt/pull/3581

What I see is that the virt-launcher pods get actually deleted (see the deletion timestamp):
                "name": "virt-launcher-testvmih5b472qswtlcwlbxpk4mpj7xqln6vf62m4wl5zcj7m",
                "generateName": "virt-launcher-testvmih5b472qswtlcwlbxpk4mpj7xqln6vf62m4wl5grzx48xd864-",
                "namespace": "kubevirt-test-default",
                "selfLink": "/api/v1/namespaces/kubevirt-test-default/pods/virt-launcher-testvmih5b472qswtlcwlbxpk4mpj7xqln6vf62m4wl5zcj7m",
                "uid": "d6f8ce74-a16d-4574-9b70-c3bc00e550f1",
                "resourceVersion": "1119773",
                "creationTimestamp": "2020-06-09T19:49:37Z",
                "deletionTimestamp": "2020-06-09T19:50:33Z",

but the pod does not terminate. If the containers don't exit, our node-controller does not react, since this may corrupt the associated VMI.
It is very likely related to the fact that the Pod status looks like this:
            "status": {
                "phase": "Pending",
[...]
                "containerStatuses": [
                    {
                        "name": "compute",
                        "state": {
                            "waiting": {
                                "reason": "PodInitializing"
                            }
                        },
                        "lastState": {},
                        "ready": false,
                        "restartCount": 0,
                        "image": "registry-proxy.engineering.redhat.com/rh-osbs/container-native-virtualization-virt-launcher@sha256:03a96f4bbd4f2ea1ce089f59e47a382a7383265b0a0d04f5fb2075c0a21fcc5c",
                        "imageID": "",
                        "started": false
                    },
                    {
                        "name": "volumedisk0",
                        "state": {
                            "waiting": {
                                "reason": "PodInitializing"
                            }
                        },
                        "lastState": {},
                        "ready": false,
                        "restartCount": 0,
                        "image": "kubevirt/cirros-container-disk-demo:v0.30.0",
                        "imageID": "",
                        "started": false
                    }

There are known corner cases where it can take a very long time until a pod gets finally deleted. I just wonder why it is still in Pending. We should not mark a VMI as running when the pod is in such a state.

If I understand correctly the issue is that the VMI is marked as "Running" which then lets the rest of the test proceed. But in this time the pod is still initializing the containers. Then when a delete request gets made it is not deleted in within the allowed timeout time since the containers can not exit due to not yet running.
It seems like the proper fix for this flaky test case then is to see why the controller is marking the VMI as running when the pod is still Pending? And instead the VMI should be kept in the state of Scheduled?

It seems like the proper fix for this flaky test case then is to see why the controller is marking the VMI as running when the pod is still Pending? And instead the VMI should be kept in the state of Scheduled?

I think, out of my head, that Scheduled is already only set when the Pod is started. As soon as virt-controller sets Scheduled, virt-handler is taking over and should try to define the VMI.

In that case I am misunderstanding the relationship that you are pointing out in your previous comment. I'm not sure what is being pointed out then between the VMI having the status of Running when the virt-launcher pod is in the state Pending ?

In that case I am misunderstanding the relationship that you are pointing out in your previous comment. I'm not sure what is being pointed out then between the VMI having the status of Running when the virt-launcher pod is in the state Pending ?

I think you got it right, I may have expressed the relationship weird. virt-handler, which is the one setting the phase to Running should not even react at this stage to a VMI. virt-controller explicitly hands over the VMI when the pods is ready (the containers are ready).

So maybe this makes sense but I'm seeing some weirdness in the timestamps.

authors: ashleyschuett dhiller phoracek rmohr stu-gott


text:
virt-launcher  start failing in the version of kubevirt v0.39.0

What happened:
the virt-launcher-pod is Terminating when I start the vm
What you expected to happen:
the error as blow:
[root@member ~]# kubectl logs virt-launcher-cirros-vm-ggdm8 -n vm {"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:68","timestamp":"2021-03-15T12:07:37.497541Z"} {"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:71","timestamp":"2021-03-15T12:07:37.497619Z"} {"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu:///system","pos":"libvirt.go:420","timestamp":"2021-03-15T12:07:37.501868Z"} {"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory')","pos":"libvirt.go:428","timestamp":"2021-03-15T12:07:37.503845Z"} {"component":"virt-launcher","level":"info","msg":"libvirt version: 6.6.0, package: 13.fc32 (Unknown, 2021-01-21-11:33:00, )","subcomponent":"libvirt","thread":"49","timestamp":"2021-03-15T12:07:37.521000Z"} {"component":"virt-launcher","level":"info","msg":"hostname: cirros-vm","subcomponent":"libvirt","thread":"49","timestamp":"2021-03-15T12:07:37.521000Z"} {"component":"virt-launcher","level":"error","msg":"internal error: Child process (dmidecode -q -t 0,1,2,3,4,11,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2741","subcomponent":"libvirt","thread":"49","timestamp":"2021-03-15T12:07:37.521000Z"} {"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:436","timestamp":"2021-03-15T12:07:38.005044Z"} {"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:454","timestamp":"2021-03-15T12:07:38.008558Z"} {"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:73","timestamp":"2021-03-15T12:07:38.009019Z"}
How to reproduce it (as minimally and precisely as possible):
when I use the version of kubevirt v0.37.0 ,everything is ok .
Anything else we need to know?:
Environment:

KubeVirt version (use virtctl version): v0.39.0
Kubernetes version (use kubectl version): v1.17.5
VM or VMI specifications:
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a): centos7
Install tools:
Others:

I'm seeing this same issue with:

kubevirt 0.39.0
kubernetes v1.19
OS k3os version v0.19.5-rc.1

$ kubectl logs virt-launcher-testvm-6llz7 -f --all-containers=true
Wed Mar 17 10:07:11 2021
error: socket does not exist anymore
{"component":"virt-launcher","level":"info","msg":"Collected all requested hook sidecar sockets","pos":"manager.go:68","timestamp":"2021-03-17T10:07:10.071011Z"}
{"component":"virt-launcher","level":"info","msg":"Sorted all collected sidecar sockets per hook point based on their priority and name: map[]","pos":"manager.go:71","timestamp":"2021-03-17T10:07:10.071298Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon: qemu:///system","pos":"libvirt.go:419","timestamp":"2021-03-17T10:07:10.121002Z"}
{"component":"virt-launcher","level":"info","msg":"Connecting to libvirt daemon failed: virError(Code=38, Domain=7, Message='Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory')","pos":"libvirt.go:427","timestamp":"2021-03-17T10:07:10.130242Z"}
{"component":"virt-launcher","level":"info","msg":"libvirt version: 6.6.0, package: 13.fc32 (Unknown, 2021-01-21-11:33:00, )","subcomponent":"libvirt","thread":"39","timestamp":"2021-03-17T10:07:10.180000Z"}
{"component":"virt-launcher","level":"info","msg":"hostname: testvm","subcomponent":"libvirt","thread":"39","timestamp":"2021-03-17T10:07:10.180000Z"}
{"component":"virt-launcher","level":"error","msg":"internal error: Child process (dmidecode -q -t 0,1,2,3,4,11,17) unexpected exit status 1: /dev/mem: No such file or directory","pos":"virCommandWait:2741","subcomponent":"libvirt","thread":"39","timestamp":"2021-03-17T10:07:10.180000Z"}
{"component":"virt-launcher","level":"info","msg":"Connected to libvirt daemon","pos":"libvirt.go:435","timestamp":"2021-03-17T10:07:10.677592Z"}
{"component":"virt-launcher","level":"info","msg":"Registered libvirt event notify callback","pos":"client.go:454","timestamp":"2021-03-17T10:07:10.683652Z"}
{"component":"virt-launcher","level":"info","msg":"Marked as ready","pos":"virt-launcher.go:73","timestamp":"2021-03-17T10:07:10.683826Z"}
{"component":"virt-launcher","level":"info","msg":"Received signal terminated","pos":"virt-launcher.go:442","timestamp":"2021-03-17T10:07:12.741617Z"}
{"component":"virt-launcher","level":"info","msg":"stopping cmd server","pos":"server.go:448","timestamp":"2021-03-17T10:07:12.741732Z"}
{"component":"virt-launcher","level":"error","msg":"failed to read libvirt logs","pos":"libvirt_helper.go:224","reason":"read |0: file already closed","timestamp":"2021-03-17T10:07:12.743775Z"}
{"component":"virt-launcher","level":"error","msg":"timeout on stopping the cmd server, continuing anyway.","pos":"server.go:459","timestamp":"2021-03-17T10:07:13.741965Z"}
{"component":"virt-launcher","level":"info","msg":"Exiting...","pos":"virt-launcher.go:471","timestamp":"2021-03-17T10:07:13.742102Z"}

Hi @wardwouts,
can you provide a pod definition of virt-launcher-testvm-6llz7 ?

The vm description is actually taken from https://kubevirt.io/labs/kubernetes/lab1:
$ cat vm.yaml
apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  name: testvm
spec:
  running: false
  template:
    metadata:
      labels:
        kubevirt.io/size: small
        kubevirt.io/domain: testvm
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            bridge: {}
        resources:
          requests:
            memory: 64M
      networks:
      - name: default
        pod: {}
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/kubevirt/cirros-container-disk-demo
        - name: cloudinitdisk
          cloudInitNoCloud:
            userDataBase64: SGkuXG4=

$ kubectl describe pod $(kubectl get pods|awk '/virt-laun/{print $1}')
Name:                      virt-launcher-testvm-wh5r4
Namespace:                 default
Priority:                  0
Node:                      k3os-18576/192.168.0.169
Start Time:                Wed, 17 Mar 2021 11:32:50 +0100
Labels:                    kubevirt.io=virt-launcher
                           kubevirt.io/created-by=ae587c38-5bdf-45ee-bb1b-6499ef7b1590
                           kubevirt.io/domain=testvm
                           kubevirt.io/size=small
Annotations:               kubevirt.io/domain: testvm
Status:                    Terminating (lasts <invalid>)
Termination Grace Period:  60s
IP:                        10.42.1.246
IPs:
  IP:           10.42.1.246
Controlled By:  VirtualMachineInstance/testvm
Init Containers:
  container-disk-binary:
    Container ID:  containerd://3d13c6b535acf8c64063a10a76c019c5a379e0a6bc2897fe8b2ab0fbbf6b6e11
    Image:         quay.io/kubevirt/virt-launcher@sha256:15d8acc10977d4bcc14deb76da3d38252b5455dff9284a80231347b19a5d73a9
    Image ID:      quay.io/kubevirt/virt-launcher@sha256:15d8acc10977d4bcc14deb76da3d38252b5455dff9284a80231347b19a5d73a9
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/cp
      /usr/bin/container-disk
      /init/usr/bin/container-disk
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Mar 2021 11:32:52 +0100
      Finished:     Wed, 17 Mar 2021 11:32:52 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  40M
    Requests:
      cpu:        10m
      memory:     1M
    Environment:  <none>
    Mounts:
      /init/usr/bin from virt-bin-share-dir (rw)
  volumecontainerdisk-init:
    Container ID:  containerd://00d0d213cf4a850b603f260f20eeaa69129698d75c04448cc2da19097cc62ede
    Image:         quay.io/kubevirt/cirros-container-disk-demo
    Image ID:      quay.io/kubevirt/cirros-container-disk-demo@sha256:0e5ac38b20abcc7752293425b239a147868facd62cd5030dede6da6f2fc526a1
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/container-disk
    Args:
      --no-op
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Mar 2021 11:32:53 +0100
      Finished:     Wed, 17 Mar 2021 11:32:53 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  40M
    Requests:
      cpu:        10m
      memory:     1M
    Environment:  <none>
    Mounts:
      /usr/bin from virt-bin-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks/container-disk-data/ae587c38-5bdf-45ee-bb1b-6499ef7b1590 from container-disks (rw)
Containers:
  compute:
    Container ID:  containerd://f9d293e7021c04d52de18b1a833a4f1026af61d0955953e8311412142232fcba
    Image:         quay.io/kubevirt/virt-launcher@sha256:15d8acc10977d4bcc14deb76da3d38252b5455dff9284a80231347b19a5d73a9
    Image ID:      quay.io/kubevirt/virt-launcher@sha256:15d8acc10977d4bcc14deb76da3d38252b5455dff9284a80231347b19a5d73a9
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/virt-launcher
      --qemu-timeout
      5m
      --name
      testvm
      --uid
      ae587c38-5bdf-45ee-bb1b-6499ef7b1590
      --namespace
      default
      --kubevirt-share-dir
      /var/run/kubevirt
      --ephemeral-disk-dir
      /var/run/kubevirt-ephemeral-disks
      --container-disk-dir
      /var/run/kubevirt/container-disks
      --grace-period-seconds
      45
      --hook-sidecars
      0
      --less-pvc-space-toleration
      10
      --ovmf-path
      /usr/share/OVMF
    State:          Running
      Started:      Wed, 17 Mar 2021 11:32:54 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
    Requests:
      cpu:                            100m
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
      memory:                         242382920
    Environment:                      <none>
    Mounts:
      /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw)
      /var/run/kubevirt/container-disks from container-disks (rw)
      /var/run/kubevirt/hotplug-disks from hotplug-disks (rw)
      /var/run/kubevirt/sockets from sockets (rw)
      /var/run/libvirt from libvirt-runtime (rw)
  volumecontainerdisk:
    Container ID:  containerd://a8b827c52d4ebfa557c65c5e4bf25d2dfbcfaa0776c397e4877bc3aa3c8f34d6
    Image:         quay.io/kubevirt/cirros-container-disk-demo
    Image ID:      quay.io/kubevirt/cirros-container-disk-demo@sha256:0e5ac38b20abcc7752293425b239a147868facd62cd5030dede6da6f2fc526a1
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/container-disk
    Args:
      --copy-path
      /var/run/kubevirt-ephemeral-disks/container-disk-data/ae587c38-5bdf-45ee-bb1b-6499ef7b1590/disk_0
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Mar 2021 11:32:56 +0100
      Finished:     Wed, 17 Mar 2021 11:32:59 +0100
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  40M
    Requests:
      cpu:        10m
      memory:     1M
    Environment:  <none>
    Mounts:
      /usr/bin from virt-bin-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks/container-disk-data/ae587c38-5bdf-45ee-bb1b-6499ef7b1590 from container-disks (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  sockets:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  virt-bin-share-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  libvirt-runtime:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  ephemeral-disks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  container-disks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  hotplug-disks:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  kubevirt.io/schedulable=true
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                 Message
  ----    ------     ----  ----                 -------
  Normal  Scheduled  17s                        Successfully assigned default/virt-launcher-testvm-wh5r4 to k3os-18576
  Normal  Pulled     17s   kubelet, k3os-18576  Container image "quay.io/kubevirt/virt-launcher@sha256:15d8acc10977d4bcc14deb76da3d38252b5455dff9284a80231347b19a5d73a9" already present on machine
  Normal  Created    16s   kubelet, k3os-18576  Created container container-disk-binary
  Normal  Started    16s   kubelet, k3os-18576  Started container container-disk-binary
  Normal  Pulling    16s   kubelet, k3os-18576  Pulling image "quay.io/kubevirt/cirros-container-disk-demo"
  Normal  Pulled     15s   kubelet, k3os-18576  Successfully pulled image "quay.io/kubevirt/cirros-container-disk-demo" in 481.773908ms
  Normal  Created    15s   kubelet, k3os-18576  Created container volumecontainerdisk-init
  Normal  Started    15s   kubelet, k3os-18576  Started container volumecontainerdisk-init
  Normal  Pulled     15s   kubelet, k3os-18576  Container image "quay.io/kubevirt/virt-launcher@sha256:15d8acc10977d4bcc14deb76da3d38252b5455dff9284a80231347b19a5d73a9" already present on machine
  Normal  Created    14s   kubelet, k3os-18576  Created container compute
  Normal  Started    14s   kubelet, k3os-18576  Started container compute
  Normal  Pulling    14s   kubelet, k3os-18576  Pulling image "quay.io/kubevirt/cirros-container-disk-demo"
  Normal  Pulled     13s   kubelet, k3os-18576  Successfully pulled image "quay.io/kubevirt/cirros-container-disk-demo" in 506.743798ms
  Normal  Created    12s   kubelet, k3os-18576  Created container volumecontainerdisk
  Normal  Started    11s   kubelet, k3os-18576  Started container volumecontainerdisk
  Normal  Killing    8s    kubelet, k3os-18576  Stopping container compute

Can you do kubectl get po virt-launcher-testvm-wh5r4 -o yaml? This could  be also SELinux issue. What distribution are you running on node?

I'm running on k3os v1.19.5-rc1. It doesn't have selinux.
Can't run the command right now as I uninstalled kubevirt. I'll try to create a new setup to test this on.

@wardwouts
Ah, I missed the fact you are running k3os. This PR will fix your issue.

Hi @watermelon-brother ,
any chance you have disabled SELinux?

@xpivarc I have disabled the SELinux before ask this question.

@watermelon-brother
Can you check this PR help you? If not we might have some loose end.

I have a try to above solution,but there is no one solution To take effect.Now,I use the version of kubevirt v0.37.0,All versions after that have problems.

There is a 0.39.1 release containing a fix. Can you confirm it is fixed?

docker image not found when I deploy the virt-operator of kubevirt v0.39.1.
bash [root@member kubevirt-v0.39.1]# docker pull quay.io/kubevirt/virt-operator@sha256:2429917d5fa837c5b3f7a8ebcc4df6f9574d3bbab8f9d47c45d1a71d150e0c7f Error response from daemon: manifest for quay.io/kubevirt/virt-operator@sha256:2429917d5fa837c5b3f7a8ebcc4df6f9574d3bbab8f9d47c45d1a71d150e0c7f not found: manifest unknown: manifest unknown

david vossel mentioned that something went wrong with the release manifests, so it'll be working soon but not yet.

authors: maya-r wardwouts watermelon-brother xpivarc


text:
Kubevirt VMI Creation with HostDisk is giving SyncFailed error

Hi,
I am trying to create a kubevirt VMI with guest os ubuntu22.04 along with it trying to create a hostdisk volume. I am using the below yaml file for this.
metadata:
name: testvmi-containerdisk
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
spec:
domain:
resources:
requests:
memory: 105Gi
cpu: 30
devices:
disks:
- name: containerdisk
disk: {}
- name: cloudinitdisk
disk:
bus: virtio
- name: rootdisk
disk:
bus: virtio
volumes:
- name: containerdisk
containerDisk:
image: quay.io/containerdisks/ubuntu:22.04
imagePullPolicy: IfNotPresent
- cloudInitNoCloud:
userData: |-
#cloud-config
chpasswd:
expire: false
password: password
user: fedora
name: cloudinitdisk
- name: rootdisk
hostDisk:
path: /mnt
type: DiskOrCreate
=====================================================================================
While applying the above yaml file Virt-laucher is runnig fine below is the output when i describe virt-laucher pod:
Events:
Type    Reason     Age   From               Message

Normal  Scheduled  22s   default-scheduler  Successfully assigned default/virt-launcher-testvmi-containerdisk-8w8kf to kubernetes-worker01
Normal  Pulled     21s   kubelet            Container image "quay.io/kubevirt/virt-launcher:v0.59.0" already present on machine
Normal  Created    21s   kubelet            Created container container-disk-binary
Normal  Started    21s   kubelet            Started container container-disk-binary
Normal  Pulled     20s   kubelet            Container image "quay.io/containerdisks/ubuntu:22.04" already present on machine
Normal  Created    20s   kubelet            Created container volumecontainerdisk-init
Normal  Started    20s   kubelet            Started container volumecontainerdisk-init
Normal  Pulled     19s   kubelet            Container image "quay.io/kubevirt/virt-launcher:v0.59.0" already present on machine
Normal  Created    19s   kubelet            Created container compute
Normal  Started    19s   kubelet            Started container compute
Normal  Pulled     19s   kubelet            Container image "quay.io/containerdisks/ubuntu:22.04" already present on machine
Normal  Created    19s   kubelet            Created container volumecontainerdisk
Normal  Started    19s   kubelet            Started container volumecontainerdisk
But VMI is not going to running state and getting below error when i describe the VMI:
Events:
Type     Reason            Age                    From                       Message

Normal   SuccessfulCreate  4m48s                  virtualmachine-controller  Created virtual machine pod virt-launcher-testvmi-containerdisk-8w8kf
Warning  SyncFailed        4m43s                  virt-handler               server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2023-03-31T06:56:34.557449Z qemu-kvm: -blockdev {"driver":"file","filename":"/var/run/kubevirt-private/vmi-disks/rootdisk/mnt","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: 'file' driver requires '/var/run/kubevirt-private/vmi-disks/rootdisk/mnt' to be a regular file')"
Warning  SyncFailed        4m43s                  virt-handler               server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2023-03-31T06:56:34.926927Z qemu-kvm: -blockdev {"driver":"file","filename":"/var/run/kubevirt-private/vmi-disks/rootdisk/mnt","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: 'file' driver requires '/var/run/kubevirt-private/vmi-disks/rootdisk/mnt' to be a regular file')"
======================================================================================
Environment:

KubeVirt version (use virtctl version): v0.59.0
Kubernetes version (use kubectl version):  v1.26.2
VM or VMI specifications: ubuntu:22.04

First, try to have a look to this guide. Additionally, I think you need to run the VM with root, otherwise the pod doesn't have the permission to right into the host path directory. Either you find the permission manually on the node, or you can enable root VM with the feature gate Root


text:
Increase verbosity level of 1MiB-alignment log

What this PR does
This Pull Request aims to increase the log verbosity of a log in AlignImageSizeTo1MiB to avoid flooding.
This log is unskippable when using a non-aligned volume size, and appears quite frequently in virt-launcher and virt-handler. We currently log it as a warning, which is overkill for this specific case. We should increase the verbosity level and let it only appear when explicitly debugging.
Some cases where the log is triggered:

In virt-handler, every time PVC/DV volumes are converted to hostDisk in ReplacePVCByHostDisk.
In virt-launcher, when we use disks that allow expansion.

Fixes # https://issues.redhat.com/browse/CNV-43005
Release note

NONE

Make sense, thanks!
/approve

/retest

/test pull-kubevirt-conformance-arm64

/lgtm

/test pull-kubevirt-e2e-k8s-1.28-sig-storage

/test pull-kubevirt-e2e-k8s-1.28-sig-storage

/test pull-kubevirt-e2e-k8s-1.28-sig-storage

/test pull-kubevirt-e2e-k8s-1.28-sig-storage

/unhold

/cherry-pick release-1.3

authors: alicefr alromeros mhenriks


text:
Sync VMI conditions to VM

Previously, we hand picked a few VMI conditions to sync to the VM such as ready and paused. In this PR, all VMI conditions are now synced to the VM. This is done in an effort to streamline debugging efforts.
There is a new "printableStatus" field on the VM object that aggregates a VM's state across multiple objects (VMI, DV, PVC, Pod) into a human readable field. By syncing all VMI conditions to the VM, we further streamline debugging by bubbling up runtime conditions on the VM.
This means that the VM's "printableStatus" field gives us a quick high level status, and the VM's conditions will contain more detailed reasoning behind why the status exists.
VM controller now syncs VMI conditions to corresponding VM object

/cc @rmohr @zcahana
how do you all feel about this? The idea here is to begin consolidating debug info on the VM object by bubbling up conditions from the lower layers.

With this change, we now have two VM conditions, Failure and Synchronized which are basically used to report errors. Do you think there's justification to hold on to both, or should we try to unify them?

Yes, i think unifying them is the right thing. If there's a VM sync error, that should take precedence over a VMI sync error on the VM condition.
looking closely at the VM failure condition, it is not accurate as it is written today, and needs some major work. I'll tackle that in this PR

Looks good!
/lgtm
/retest

/retest

authors: davidvossel zcahana


text:
workspace: Bump alpine-with-tooling image

What this PR does / why we need it:
The last alpine with tooling image contains a proper ping command.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
NONE

/hold @qinqon Can you please check the SRIOV lane? looks like the test fails on ping -6

Is not only SRIOV looks like there are massive failures.

/test pull-kubevirt-e2e-k8s-1.22-ipv6-sig-network
/test pull-kubevirt-e2e-kind-1.23-vgpu-nonroot
(Random sample to avoid burdening the CI too much)

/approve

/hold cancel

/hold
@qinqon Can you please check the SRIOV lane? looks like the test fails on ping -6

Hi Igor, there were a bug con the alpine ping call, now it's fixed, unholding.

/retest

/lgtm

/retest

/lgtm

/retest

authors: AlonaKaplan enp0s3 maya-r qinqon


text:
Rename and authorization for namespaced expand-spec endpoint

What this PR does / why we need it:
The expand-spec endpoint is renamed to expand-vm-spec and made
namespaced to allow proper authorization.
This PR adds code to the virt-api authorizer to check namespaced
base resource endpoints and unittests/functests to verify the authorization.
Based on https://github.com/kubevirt/kubevirt/pull/8399 and https://github.com/kubevirt/kubevirt/pull/8570
/area instancetype
Release note:

The expand-spec subresource endpoint was renamed to expand-vm-spec and made namespaced

Looks good.
Have you considered renaming the endpoint to something that would hint that it takes a VM definition, like expand-vm-spec?
It would be harder to change name once the API is merged.
/lgtm

/retest

@akrejcir Good idea, I renamed it to expand-vm-spec

/lgtm

@akrejcir I had to rebase, can you lgtm again?

/retest

/lgtm

/retest

/retest

@davidvossel I updated the PR and tried to keep the changes to the authorizer minimal. Can you PTAL again?

@lyarwood I addressed your comments, can you PTAL?

@lyarwood I addressed your comments, can you PTAL?

/lgtm
Thanks changes look good!

authors: 0xFelix akrejcir lyarwood


text:
ci: fix image version in the released yaml bundle

What this PR does / why we need it:
The YAML file that was added to the releases contained latest image tag. Now it contains the release tag.
Release note:
None

/cc @codingben

Can you please just have fix: image version in the released yaml bundle or just ci: fix image version in the released yaml bundle?

Can you please just have fix: image version in the released yaml bundle or just ci: fix image version in the released yaml bundle?

Done

/approve

authors: akrejcir codingben

repo: kubevirt/vm-console-proxy


text:
Enable verification of policies signed using sigstore's keyless signing

Policy server must be able to verify policies that have been signed using Sigstore's keyless signing.
The verification requires the user to provide a list of email addresses that are expected to be inside of the certificate issued by Fulcio.
Implementation details
The verification.yaml file should be enriched to have also another top level key: cert_emails.
This is how the file will look like:
---
verification_keys: #optional
  key-name-irrelevant: |
            -----BEGIN PUBLIC KEY-----
            MBFKHFDGHKIJH0CAQYIKoZIzj0DAQcDQgAEX0HFTtCfTtPmkx5p1RbDE6HJSGAVD
            BVDF6SKFSF87AASUspkQsN3FO4iyWodCy5j3o0CdIJD/KJHDJFHDFIu6sA==
            -----END PUBLIC KEY-----
  another-key: |
            -----BEGIN PUBLIC KEY-----
            AUYSDHKJFGSHGSHCAQYIKoZIzj0DAQcDQgAEX0HFTtCfTtPmkx5p1RbtwDE1EVzu
            wjQs1cCRKb5Pz/yUspkQsN3FO4iyWodCy5j3o0CdIJD/1gvq98pf4IG9tA==
            -----END PUBLIC KEY-----
cert_emails: #optional
- flavio@foo.bar.com
- victor@foo.bar.com
verification_annotations: # optional
  env: prod
  foo: bar

All the cert_emails mentioned must be found inside of the policy signatures. This is the same behaviour of multiple verification_keys.
The following table shows the behaviour of the verification check, depending on what the user provides

verification keys
cert emails
annotations
outcome

✅
❌
❌
each policy must be signed with all the specified verification keys

✅
✅
❌
each policy must be signed with all the specified verification keys and with all the specified emails

✅
✅
✅
each policy must be signed with all the specified verification keys and with all the specified emails; plus each signature must satisfy all the annotations provided by the user

✅
❌
✅
each policy must be signed with all the specified verification keys; plus each signature must satisfy all the annotations provided by the user

❌
✅
✅
each policy must be signed with all the specified emails; plus each signature must satisfy all the annotations provided by the user

❌
❌
✅
I don't think scenario makes any sense, we should error out when the configuration looks like that

Thinking about the implementation, the struct holding the contents of verification_keys should have keys, emails and annotations as Option<>.
I think we should write some check against the object we just deserialized and then error if only annotations are provided. I prefer to fail because providing only annotations would lead to accept all policies. That gives a false sense of security, which is not what someone providing an explicit configuration for the verification feature expects.

after discussion today, I would like to go with something as:
allOf: # all info listed here needs to be verified for accepting
  pub_keys:
    key-name-irrelevant: |
              -----BEGIN PUBLIC KEY-----
              MBFKHFDGHKIJH0CAQYIKoZIzj0DAQcDQgAEX0HFTtCfTtPmkx5p1RbDE6HJSGAVD
              BVDF6SKFSF87AASUspkQsN3FO4iyWodCy5j3o0CdIJD/KJHDJFHDFIu6sA==
              -----END PUBLIC KEY-----
    another-key: |
              -----BEGIN PUBLIC KEY-----
              AUYSDHKJFGSHGSHCAQYIKoZIzj0DAQcDQgAEX0HFTtCfTtPmkx5p1RbtwDE1EVzu
              wjQs1cCRKb5Pz/yUspkQsN3FO4iyWodCy5j3o0CdIJD/1gvq98pf4IG9tA==
              -----END PUBLIC KEY-----

anyOf: # at least 1 of all info listed here needs to be verified for accepting
  keyless:
    - oidc_issuer: https://token.actions.githubusercontent.com # default tuple, we add it to our minimum config
      provenance: https://github.com/kubewarden
    - oidc_issuer: https://token.actions.githubusercontent.com
      provenance: https://github.com/kubewarden/pod-privileged-policy
    - oidc_issuer: https://oauth2.suse.com/auth
      provenance: kubewarden@suse.de
  annotations:
    env: prod
    foo: bar

That is, add a root level that can be one of allOf or anyOf, which inside contain any number of pub_keys/keyless/annotations.

Let's move the discussion about the verification.yaml file over there: https://github.com/kubewarden/kubewarden-controller/pull/147
I gotta say, I like the idea of allOf and anyOf; that makes things really explicit.

Since there's a lot of work that comes from consuming the big refactor of sigstore-rs >= 0.2.0, I will start noting here the Epic.
Almost all changes are for policy-fetcher, and it make take a couple of days to settle. So I have created the branch feat-sigstore-0.2 in policy-fetcher, and I will open PRs against that branch.

[ ] https://github.com/kubewarden/policy-fetcher/pull/42
[ ] https://github.com/kubewarden/policy-fetcher/pull/50
[ ] Implement rest of Verifiers for Keyless verification
[ ] Consume policy-fetcher changes in policy-server (use read_verification_file() from policy-fetcher, call  policy-fetcher::verifier::verify())
[ ] tests

closing, because we have broken down it into smaller epics and their sub-tasks

authors: flavio viccuad

repo: kubewarden/policy-server