text:
Any reason mergeDeepWithKey does not compare reference equality

I currently have two issues, the first one (1) is that i'm merging some large objects and checking ramda's code i note that the objects are merged no matter if they are the exact same object.
Is there any reason for not having something like this in R.mergeDeepWithKey?
var mergeDeepWithKey = _curry3(function mergeDeepWithKey(fn, lObj, rObj) {
  if (lObj === rObj) return lObj; // <- compare here
  return mergeWithKey(function(k, lVal, rVal) {
    if (_isObject(lVal) && _isObject(rVal)) {
      return mergeDeepWithKey(fn, lVal, rVal);
    } else {
      return fn(k, lVal, rVal);
    }
  }, lObj, rObj);
});

I been using ramda for a while now, but don't much about the internal implementation, so no idea if something like this is against the general guideline of the library.
The second issue (2) is when i deep merge objects that contain as value moment instances (which are objects) then some of the moment functionalities gets broken.
If i have the same moment instance i keep the left [or right] which is similar to issue (1).
And when they are not the same instance, but at least one of them is a moment instance then i keep left [or right].
Currently have this utils:
export const mergeDeepUnless = R.curry((fnUnless, fnObj, fnVal, lObj, rObj) => {
  const recursiveMergeDeepUnless = (lObj0, rObj0, k0) => {
    if (fnUnless(k0, lObj0, rObj0)) {
      return fnObj(k0, lObj0, rObj0);
    }
    return R.mergeWithKey((k, lVal, rVal) => {
      if (R.is(Object)(lVal) && R.is(Object)(rVal)) {
        return recursiveMergeDeepUnless(lVal, rVal, k);
      }
      return fnVal(k, lVal, rVal);
    }, lObj0, rObj0);
  };
  return recursiveMergeDeepUnless(lObj, rObj);
});
export const mergeDeepWhen = R.curry((fnWhen, fnObj, fnVal, lObj, rObj) => mergeDeepUnless(
  R.complement(fnWhen), fnObj, fnVal, lObj, rObj,
));
export const mergeDeepLeftWhen = R.curry((fnWhen, lObj, rObj) => mergeDeepUnless(
  R.complement(fnWhen), R.nthArg(1), R.nthArg(1), lObj, rObj,
));
export const mergeDeepRightWhen = R.curry((fnWhen, lObj, rObj) => mergeDeepUnless(
  R.complement(fnWhen), R.nthArg(2), R.nthArg(2), lObj, rObj,
));

And i use them like:
const mergeDeepLeftWhenIsNotRefEquals = mergeDeepLeftWhen((k, l, r) => l !== r);

const isNotMoment = R.complement(R.is)(moment);
const mergeDeepLeftWhenIsNotMoment = mergeDeepLeftWhen(
  (k, l, r) => isNotMoment(l) && isNotMoment(r),
);

So the question is just that, is there is a built in way of doing something like this.

Every time Ramda chooses to optimize by returning an unaltered value in a function that usually alters that value, we end up causing problems with JS users' expectations.  Of course this wouldn't be an issue if users never mutated data.  This, unfortunately, is outside our control, and sometimes they do mutate it. If they mutate a property on the merged object that happens to be a reference to the property on an original object, they mutate the original too.  And they blame Ramda, with at least some justification.
That is why we don't tend to do this.
As to Moment, I don't know that there is an alternative.  We do our deep merge without ever trying to handle the prototype chain.  I don't see a way to do this differently, not with complete accuracy.  A constructor that uses its parameters to set up some invariants the object then maintains seems impossible to duplicate without access to the original parameters.  But even if we just wanted to merge {x: new Rectangle(7, 4)} and {x: new Square(5)}, which constructor would we try to use?  Right now we would probably get something like {width: 7, height: 4, side: 5} without the area or perimeter methods from those classes.  If you have a suggestion for how we could fix this, I'm very interested.  I've been trying to conceive of one for some time, with no success.

Yeah i understand that is difficult to decide when to return the original object and when to return a new one (performance vs usability?). But i think if the user is already using ramda, and wants to change some object's value, then it should do it using ramda, and not mutating the object directly, that's the point of assoc and similar methods.
If you have some objects to this:
const objDoNotWantToMutate = { value: 1 };
const objToMerge1 = { deep: { key1: objDoNotWantToMutate } };
const objToMerge2 = { deep: { key1: objDoNotWantToMutate } };
const objToMerge3 = { deep: { key2: objDoNotWantToMutate } };

If you merge objToMerge1 with objToMerge2 and mutate the key1, then the original object is not mutated.
const result = R.mergeDeepLeft(objToMerge1)(objToMerge2);
const obj = R.path(["deep", "key1"])(result);
obj.value = 2; // does not mutate 'objDoNotWantToMutate'

But if you merge objToMerge1 with objToMerge3 and mutate the key1, then the original object is mutated.
const result = R.mergeDeepLeft(objToMerge1)(objToMerge3);
const obj = R.path(["deep", "key1"])(result);
obj.value = 2; // it does mutate 'objDoNotWantToMutate'

If you are mutating the result of a merge, you either have no idea/do not care that the original object is mutated or you know very well the structures of the objects been merged. In the first case the object is not cloned due to performance, and in the second case is merged to be less error prone (due to user misusing), is hard to maintain consistensy, while balancing performance and usability [but it can at least another method to provide more flexibility to the user, see below].

As of the moment thing, i agree, i don't think there is a logical way of implementing something like that. I mean what would be the logical result of merging a new Circle(5) with a new Square(5) (the only difference in the contructors is the name of the argument and is not reliable at all), the area method of both also return different values. So the most logical result of this merge would be the current behavior, an object that doesn't handle prototypes: { radio: 5, side: 5, area: f() } where the area method is the one of the circle or the one of the square, depending whether it was a left or right or a merge with.
The only person that would know how to combine the Circle with the Square should be the user, he/she may prefer circles over squares, circles may be merged by radio average, and squares by max side size.
So i think it should be a method that allows to provide a function like this one (currently not posible since both Circle and Square are objects):
const customMerge = (key, left, right) => {
  if (R.is(Circle)(left) && R.is(Square)(right)) return left; // prefer circles
  if (R.is(Square)(left) && R.is(Circle)(right)) return right; // prefer circles
  if (R.is(Circle)(left) && R.is(Circle)(right)) return new Circle(
    (left.radio + right.radio) / 2.0,
  ); // average circle radio
  if (R.is(Square)(left) && R.is(Square)(right)) { // prefer larger squares
    if (left.side >= right.side) return left; // or new Square(left.side) or left.clone()
    return right; // or new Square(right.side) or right.clone()
  }
}

This would allow the user to decide for instance, what to do when the objects are exacly the same (first example):
const customMerge = (left, right) => {
  if (left === right) return left;
}

So i think it should be a method that allows to provide a function like this one (currently not possible [at least not in deep merges, in simple merges is possible using R.mergeWith] since both Circle and Square are objects)

Have you seen mergeDeepWithKey?

Have you seen mergeDeepWithKey?

function Circle(radio) {
  this.radio = radio;
  this.area = function() {
    return this.radio * this.radio * Math.PI;
  };
  this.area = this.area.bind(this);
};
function Square(side) {
  this.side = side;
  this.area = function() {
    return this.side * this.side;
  };
  this.area = this.area.bind(this);
};

Using the first customMerge in my last comment
const simpleObj1 = { shape: new Circle(5) };
const simpleObj2 = { shape: new Square(5) };

const simpleMerge = R.mergeWithKey(customMerge)(simpleObj1)(simpleObj2);
// { shape: Circle{ radio: 5, area: f } }

simpleMerge === simpleObj1 // false
simpleMerge === simpleObj2 // false
simpleMerge.shape === simpleObj1.shape // true

Here everything it ok.
const complexObj1 = { deep: { shape: new Circle(5) } };
const complexObj2 = { deep: { shape: new Square(5) } };

const complexMerge = R.mergeDeepWithKey(customMerge)(complexObj1)(complexObj2);
// { deep: { shape: { radio: 5, side: 5, area: undefined } } }

complexMerge === complexObj1 // false
complexMerge === complexObj2 // false
complexMerge.deep.shape === complexObj1.deep.shape // false
complexMerge.deep.shape === complexObj2.deep.shape // false

Here is not, and there is no way (that i know of) of doing this in ramda, have to implement the merge myself.

I suppose what I'd like to see is a PR for your proposal.  I'm not sure what the team would think, but i can promise a fair hearing.  I'm getting lost between what you think is desired behavior, and what you do and don't like about the current merge* functions.

Well when i first opened the issue i was just wondering why deep merge didn't make a simple === since the deep merge already do some optimizations; like avoiding to clone objects when is not necessary (e.g. key in just one of the two objects to be merged).
Now i think is just easier to provide a way to customize the behavior of the deep merge, so it doesn't affect any of the already existing methods.
But not sure if you want yet another merge function, or how many people would find it useful. I'm currently using some sort of custom deep merge in 3 of my 4 projects where i use ramda.
The implementation would be almost the same as mergeDeepWithKey except it receives another function. When this function fnObj returns an object, this result is used, and when it doesn't then the deep merge is done as usual.
var mergeDeepUsingKey = _curry3(function mergeDeepUsingKey(fnObj, fnVal, lObj, rObj) {
  return mergeWithKey(function(k, lVal, rVal) {
    if (_isObject(lVal) && _isObject(rVal)) {
	  var r = fnObj(k, lVal, rVal);
	  return _isObject(r) ? r : mergeDeepUsingKey(fnObj, fnVal, lVal, rVal);
    } else {
      return fnVal(k, lVal, rVal);
    }
  }, lObj, rObj);
});
export default mergeDeepUsingKey;

The functionality of mergeDeepWithKey is the same as mergeDeepUsingKey(always(null)).
Or if it makes sense that fnObj returns another types, and not only objects, the result can be compared with a constant (like R.T) to continue the deep merge.

authors: Bradcomp CrossEye jmdsg


text:
What is the purpose of performing exponential calculations with e here?

https://github.com/ramdrop/stun/blob/d4fd9b0fe8f34a6f5c6a797c80467b0c72c7e88f/trainer.py#L507
What is the purpose of exponentiating the extracted image features by e?
What is the mathematical basis for doing this?

Today I re-read your paper "STUN: Self-Teaching Uncertainty Estimation for Place Recognition" and the code for stun. I was once again impressed by the great design of the program.Thank you for sharing such an excellent article. I have a few questions about numerical calculations. Could I ask for your help?
https://github.com/ramdrop/stun/blob/d4fd9b0fe8f34a6f5c6a797c80467b0c72c7e88f/trainer.py#L390
https://github.com/ramdrop/stun/blob/d4fd9b0fe8f34a6f5c6a797c80467b0c72c7e88f/trainer.py#L392
https://github.com/ramdrop/stun/blob/d4fd9b0fe8f34a6f5c6a797c80467b0c72c7e88f/trainer.py#L394
https://github.com/ramdrop/stun/blob/d4fd9b0fe8f34a6f5c6a797c80467b0c72c7e88f/trainer.py#L396
# ---------------------- shift sigma_sq ---------------------- #
                if self.opt.loss in ['tri', 'quad']:    # empically found shifting distribution to be helpful for these losses
                    log_sigma_sq = torch.clamp(10 * log_sigma_sq + 0.2, 0, 1)
                # == numerator
                mu_delta = torch.norm((mu_stu - mu_tea), p=2, dim=-1, keepdim=True)                # L2 norm -> ([B, D])
                # == denominator
                sigma_sq = torch.exp(log_sigma_sq)
                # == regulizer
                loss = (mu_delta / sigma_sq + log_sigma_sq).mean()                                 # ([B, D])

How can I understand the mathematical logic behind these lines of code?

log_sigma_sq = torch.clamp(10 * log_sigma_sq + 0.2, 0, 1) shifts the distribution of log_sigma_sq to enable a comparable scale between mu_delta / sigma_sq and log_sigma_sq. This operation has been found to make the training easier.

Thank you for your guidance. May I ask you two more questions?
How are the values 10 and  0.2  chosen in this line of code?
  log_sigma_sq = torch.clamp(10 * log_sigma_sq + 0.2, 0, 1)
How to understand that the output values after the feature values in the student network are processed by the mean_head layer represent the logarithmic variance of the student network feature values?

10 and 0.2 are found to make the distribution of log_sigma_sq to enable a comparable scale between mu_delta / sigma_sq and log_sigma_sq.

How to understand that the output values after the feature values in the student network are processed by the mean_head layer represent the logarithmic variance of the student network feature values?
I didn’t quite catch your question. Can you break it down a bit more?

If I want to apply your paper's method to other areas, can you tell me the method to find these values of 10 and 0.2, so I can migrate it to other projects? Because in different situations, these values should be different, right?
   log_sigma_sq = torch.clamp(10 * log_sigma_sq + 0.2, 0, 1)

Why do the output values after the feature values in the student network are processed by the mean_head layer represent the logarithmic variance of the student network feature values?

If I want to apply your paper's method to other areas, can you tell me the method to find these values of 10 and 0.2, so I can migrate it to other projects? Because in different situations, these values should be different, right?
 log_sigma_sq = torch.clamp(10 * log_sigma_sq + 0.2, 0, 1)

I recommend you search for the two parameters.

Think  you for your help
Because it refers to the logarithmic variance by design.
Is there a mathematical basis for this?
I recommend you search for the two parameters.
Could you please give me some guidance on how to conduct the search specifically? Thank you.

Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." Advances in neural information processing systems 30 (2017).
You can regard it as hyperparameters of the model and use common hyperparameter search methods.

Think you for your help

source: gharchive/issue

authors: ramdrop wpumain

repo: ramdrop/stun

license_type: permissive

license_source: github-api


text:
Verbesserungen

Hi,
konnte dich bei Discord nicht finden. Schreibe es deswegen hier.
Das Programm würde ich gern auf einem Rechner 24/7 laufen lassen. Bin auf der Suche nach einer Wohnung.
Telegram Bot integrieren, sobald eine neue Anzeige veröffentlicht ist, dass man eine Telegram Nachricht bekommt.
Intervalle für die "Checks" setzen können, alle 10 Sek, 15 Sek, 30 Sek..usw
Würde dir einen grünen Schein spenden. Später wahrscheinlich noch mehr. :)
LG

Hallo,
ja mein Discord Tag hat sich geändert.
Die von dir gewünschten Änderungen kann ich gerne umsetzen. Schreib mir gerne auf Disord:Ramo#3735
Lg Ramo

authors: clariion ramy-zemo

repo: ramy-zemo/Willhaben-Grabber

license: MIT


text:
Datasets with vastly varying edge counts

Some of the larger datasets out there (such as Shapenet and ABC) have edge counts ranging from 100's to 100000's.
Do you have any advice on selecting a pool_res and ncf for datasets with this sort of edge count variation? And/or any thoughts on the effect this would have on your research?

Hi @mrmoss ,
I responded to a similar question in this issue. Basically, I suggest processing the data first. I just posted a very basic script to do this in blender here.

authors: mrmoss ranahanocka

repo: ranahanocka/MeshCNN


text:
Running Convoy EBS on RancherOS

[rancher@ip-10-0-0-12 ~]$ ros -v
ros version v0.4.5
[rancher@ip-10-0-0-12 ~]$ sudo convoy -v
convoy version 0.3
I've followed the instructions provided on this git to install convoy on my AWS instance running RancherOS.
But, when I try to create a volume, I've the following error:
[rancher@ip-10-0-0-12 ~]$ sudo convoy create my_vol
ERRO[0010] Error response from server, Failed to execute: mkfs [-t ext4 /dev/xvdk], output , error exec: "mkfs": executable file not found in $PATH

{
        "Error": "Error response from server, Failed to execute: mkfs [-t ext4 /dev/xvdk], output , error exec: \"mkfs\": executable file not found in $PATH\n"
}

I've a workaround which consist to alias mkfs:
[rancher@ip-10-0-0-12 ~]$ sudo ln -s /sbin/mkfs.ext4 /sbin/mkfs
[rancher@ip-10-0-0-12 ~]$ sudo convoy create my_vol3
721a9fb3-5f9d-4c89-b58e-da02f64b8f1b

But it's not much pretty.

Furthermore, sudo convoy list always return an empty list, even if some volume were created.

I've just started with Convoy and EBS on RancherOS.  I use the symlink you mentioned (thank you!) but I get the right answer when running sudo convoy list.
[rancher@awshost1 ~]$ ros -v
ros version v0.4.5
[rancher@awshost1 ~]$ sudo convoy -v
convoy version 0.4.3

My convoy's newer, so I bet they fixed the list issue you mentioned.

@mathuin You means there is some improvement, great! I'll upgrade to the new version and check what's new :)

authors: mathuin saphoooo

repo: rancher/convoy

license_source: bigquery


text:
Shell fails to open in a pod

Setup

Rancher version: v2.9-c9be13b09329bbee60a5f6419d500198f83c44d1-head
Browser type & version: Google Chrome 123.0.6312.107 (Official Build) (arm64) and Safari 16.6

Describe the bug
Shell fails to open in a pod due to an unknown path /bin/sh: no such file or directory: unknown. According to the console logs, the error occurs in file ContainerShell.vue:319.
To Reproduce

Create a downstream cluster (in this example, it's a GKE cluster)
When the cluster is available, via UI, go to Workloads > Pods and click on the 3 dot button of any pod, then click on Execute Shell
Observe the error message when the shell is open

Result
Shell fails to open in a pod due to an unknown path /bin/sh: no such file or directory: unknown
Expected Result
Shell should be open in a pod with no errors
Screenshots

I wonder if this is related to changing the underlying image to BCI?
We should confirm if this is a bug in 2.7 and 2.8 as well

Will file something for Mapps to figure out what we can do around new BCI behaviours. Ideally, the UI could detect that there is no shell and elegantly state there's nothing to shell into.

@IsaSih when you get a moment, can you check the behaviour in 2.8.x? I don't know if we have to go back into 2.7.x at all but that may help if the problem existed somewhat earlier than 2.9.0

This issue does not occur in version 2.8-head. Tested on v2.8-94108186cf35028bb5f7db2c34c5ae99450dad9b-head

Internal reference with Mapps: SURE-8318
May have to monitor here as that team works on that in case it comes back to UI or QA

We should re-validate that this is still an issue - I believe this is down to a change in base image which does not contain sh - I think this may have been fixed - we should validate.
Should also add an automated test for this.

Worked through this with Neil.
The description shows pods with images not from Rancher, it looks like those just don't contain a shell so will never work.
I've brought up a new v2.9-head with the BCI change and can confirm some work

alertmanager-rancher-monitoring-alertmanager-0, cattle-monitoring-system, docker.io/rancher/mirrored-prometheus-alertmanager:v0.26.0
gitjob-7f85cd5774-4r429, cattle-fleet-system, rancher/fleet:v0.10.0-rc.13

and some don't

capi-controller-manager-64b8d54977-c54w6, cattle-provisioning-capi-system, rancher/mirrored-cluster-api-controller:v1.4.4
coredns-6799fbcd5-dzggt, kube-system, rancher/mirrored-coredns-coredns:1.10.1

I don't think this is a regression, but we need to make the error clearer. One is the root failure and one is from the UI. We can do this by adding a return after each error and maybe some before our error.

Holding on backport for now until someone picks this up to see what a solution looks like.

I updated to logging so that it's easier to parse when an error occurs, we'll also get some additional context when /bin/sh is missing from the container.
@gaktive this will be a fairly easy change to backport, what should our targets be at this point?

@gaktive this will be a fairly easy change to backport, what should our targets be at this point?

Following up from my previous comment: we discussed backporting OOB and decided that there is no need to do so at this point.

authors: IsaSih gaktive nwmac rak-phillip richard-cox

repo: rancher/dashboard


text:
OPA- Gatekeeper is inaccessible upon Rancher upgrade

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):
Provision a cluster in Rancher v2.4.8
Navigate to dashboard and install OPA

Create a constraint and validate it is being enforced:

Upgrade Rancher to v2.5.0-alpha1
In Rancher, try to access OPA Gatekeeper
Result:
OPA Gatekeeper install page is seen.
Under installed apps, gatekeeper is seen:

Trying to install gatekeeper again will fail because there is no CRD:

User is also unable to delete the old version of OPA Gatekeeper
Other details that may be helpful:
The constraint created in Rancher v2.4.8 is still enforced after Rancher upgrade. OPA cannot be accessed to delete or edit old constraints created previously.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.5.0-alpha1
Installation option (single install/HA): HA

Upgrading Rancher to master-head version 8fd81a9 makes OPA available again. Closing issue.


text:
As a cluster member attempting to add a member to the cluster results in no error message just stuck in creating

Rancher Server Setup
Rancher version: v2.6.0-head on commit-id: 72986b2
Installation option (Docker install/Helm Chart): Helm Chart
If Helm Chart, Kubernetes Info:
Cluster Type (RKE1, RKE2, k3s, EKS, etc): RKE1
Information about the Cluster
Kubernetes version: v1.21.3-rc6+rke2r2
Cluster Type (Local/Downstream): RKE2 Digital Ocean Cluster
Describe the bug
With a cluster member, when attempting to add a member to the cluster, it gets stuck in the creating phase, with no error message ever returned.
This is the API response:
{baseType: "error", code: "Forbidden",…}
baseType: "error"
code: "Forbidden"
message: "clusterroletemplatebindings.management.cattle.io is forbidden: User \"u-jsrv7\" cannot create resource \"clusterroletemplatebindings\" in API group \"management.cattle.io\" in the namespace \"c-d4zkz\""
status: 403
type: "error"

To Reproduce

With an Admin user created a cluster (either RKE1 or RKE2)
Create a 2 standard users; user1 and user2
Add that the user1 as a cluster member for the aforementioned cluster.
Login as user1, and attempt to add user2 as a cluster member.

Result

UI gets stuck in the creating state and no error message is returned.

Expected Result
A failure with a descriptive error message, like in the API call. OR removed Add option for CLuster Members.

On Rancher v2.6-head version c10ef4b:

v2.6-head 1a57856 pulled 9/17/21 ~9:30am Pacific
@igomez06 I have a question about the steps mentioned to reproduce - I cannot reproduce this. It is not possible for a cluster member to add another cluster member as they do not have rights to modify the cluster.
Was there anything else you might have done during user1 creation? suer1 is just a standard user right? (that's it?)

No option to edit cluster (as expected) therefore not possible to add user2

hmm - or maybe I am confused? How did you add another user to the cluster as a cluster member?

My checks PASSED
Reproduction Steps:

Not required. The issue was filed by a peer and I understand the reproduction steps.

Validation Environment:

Rancher version: v2.6-head 1a57856 pulled 9/17/21 ~9:30am Pacific
Rancher cluster type: single-node docker install
Docker version: 20.10
Downstream cluster type: rke2 (could also be rke1, does not matter)
Downstream K8s version: v1.21.4+rke2r2
Validation steps:

With an Admin user create a cluster (RKE2 in my case, 1 node with DO provider but this does not matter)
Create a 2 standard users; user1 and user2
Add that the user1 as a cluster member for the aforementioned cluster.
Login as user1, and attempt to add user2 as a cluster member (do this by exploring the cluster and on the left pane, go to RBAC>Cluster Members)
The backend will respond with a 403 error (as expected) but now the UI will keep us on the form and show an error message in the UI. Therefore this now works correctly and the issue is resolved.

Additional Info:

You cannot as a cluster member edit the cluster to add a member, but you can see this in the UI under the cluster explorer - why? If the user is not authorized I would expect to see this hidden in the UI -- but I think this is because they might be able to add custom roles of some kind if their user had the correct rights.

authors: aaronRancher davidnuzik igomez06


text:
Add zube migration script (WIP)

This PR adds a script to migrate the swimline from the Zube label on a GH issue to the GitHub Project.
It also adds a new workflow to move linked issues to the appropriate column in the GH Project.
Note:

This is currently not complete, but needs to be merged, so I can test the permissions that the GH_TOKEN has, to understand if it can read the GH project.

@aalves08 The workflow contains debug info - need this merged so I can observe the output to check what permissions we need to set up.

source: gharchive/pull-request


text:
[Automation] - Adjust cypress execution to generate html report on Jenkins

Summary
Fixes #

Occurred changes and/or fixed issues

Technical notes summary

Areas or cases that should be tested

Areas which could experience regressions

Related
https://github.com/rancherlabs/corral-packages/pull/65
Checklist

[ ] The PR is linked to an issue and the linked issue has a Milestone, or no issue is needed
[ ] The PR has a Milestone
[ ] The PR template has been filled out
[ ] The PR has been self reviewed
[ ] The PR has a reviewer assigned
[ ] The PR has automated tests or clear instructions for manual tests and the linked issue has appropriate QA labels, or tests are not needed
[ ] The PR has reviewed with UX and tested in light and dark mode, or there are no UX changes

@yonasberhe23 @rancher-max I moved the json configuration that supplemented the cypress.config.ts in the repo root to a dedicated cypress config. I've tested and it's working on branches other than master (release-2.7 ..). Although I had to make changes to make it work on the others, like the testDirs set.
This will need backports of cypress/support/e2e.ts, package.json and yarn.lock to include cypress-mochawesome-reporter as devDependencies. Which should be an innocuous change for dashboard.

Related PRs:
DevDependency 2.9: https://github.com/rancher/dashboard/pull/12163
DevDependency 2.8: https://github.com/rancher/dashboard/pull/12164
DevDependency 2.7: https://github.com/rancher/dashboard/pull/12165

This depends on the corral-package PR in the description. I'm still working on getting that merged


text:
Add helper messaging to opa constraint parameters

rancher/dashboard#670

@deniseschannon This edit as yaml within a form which also has an outer edit as yaml/form is very weird.  And the yaml option has no example of what you're supposed to put in there.

I'm closing this as we may implement a different solution.


text:
#279: add short flags

Closes: #279
This PR updates urfave/cli to v2 and adds short flags.

This looks good. I assume this removes the ability to use flags like this: -fork-path?

This looks good. I assume this removes the ability to use flags like this: -fork-path?

No, but there's an option to do so

repo: rancher/ecm-distro-tools


text:
k3s crashes on thread exhaustion.  Can't be restarted immediately due to db locking

Version:
k3s version v1.17.2+k3s1 (cdab19b0)
K3s arguments:
/sbin/k3s server --no-deploy traefik --node-label galileosuite.com/allow-galileo-re
dis-0=true --node-label galileosuite.com/allow-galileo-pxc-0=true --node-label galileosuite.com/allow-galileo-clickhouse-default-0-0=true --node-lab
el galileosuite.com/allow-galileo-clickhouse-default-1-0=true --node-label galileosuite.com/allow-galileo-zookeeper-0=true --node-label galileosuite
.com/allow-galileo-zookeeper-1=true --node-label galileosuite.com/allow-galileo-zookeeper-2=true --node-label galileosuite.com/allow-prometheus=true
--node-label galileosuite.com/allow-alertmanager=true --node-label galileosuite.com/allow-grafana=true --node-label k3os.io/mode=local --node-label
k3os.io/version=v0.9.1
Describe the bug

we ran into an issue (twice now) where k3s crashes on a message regarding thread exhaustion, then after many failed restarts with exceptions about the sqlite db, supervise-daemon eventually gives up and stops restarting it. After this, it is possible to start k3s manually without issue.
To Reproduce

Simply run k3s with above options.  We have 3 helm charts deployed, running about 60 pods on a server with 8 cores and 32GB of RAM.   Nothing terribly interesting.  No signs of CPU,Memory, or Disk i/o constraints.
Expected behavior

Normal operation
Actual behavior

k3s crashes with the following output.  Then repeatedly fails to start with go exceptions regarding the sqlite db:
runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion
runtime stack:
runtime.throw(0x3e409c2, 0x11)
/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.checkmcount()
/usr/local/go/src/runtime/proc.go:600 +0xac
runtime.mcommoninit(0xc020642a80)
/usr/local/go/src/runtime/proc.go:618 +0x73
runtime.allocm(0xc000080000, 0x3ff0508, 0xc000000001)
/usr/local/go/src/runtime/proc.go:1489 +0x150
runtime.newm(0x3ff0508, 0xc000080000)
/usr/local/go/src/runtime/proc.go:1804 +0x39
runtime.startm(0xc000080000, 0x6d86f01)
/usr/local/go/src/runtime/proc.go:1969 +0x12a
runtime.handoffp(0xc000080000)
/usr/local/go/src/runtime/proc.go:2007 +0x29a
runtime.retake(0x24f4a2d8b343, 0x1)
/usr/local/go/src/runtime/proc.go:4444 +0x157
runtime.sysmon()
/usr/local/go/src/runtime/proc.go:4353 +0x120
runtime.mstart1()
/usr/local/go/src/runtime/proc.go:1201 +0xc3
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1167 +0x6e
Additional context / logs
Full log from kubelet start:
shorter.log

Looks to be a kine issue, I see a ton of goroutines all with a similar stack:
goroutine 4132 [select]:
github.com/rancher/k3s/vendor/github.com/mattn/go-sqlite3.(*SQLiteRows).Next(0xc01582ede0, 0xc01b131ad0, 0xb, 0xb, 0x0, 0x0)
	/go/src/github.com/rancher/k3s/vendor/github.com/mattn/go-sqlite3/sqlite3.go:1994 +0x1e3
database/sql.(*Rows).nextLocked(0xc00ea1b200, 0x420000)
	/usr/local/go/src/database/sql/sql.go:2767 +0xd5
database/sql.(*Rows).Next.func1()
	/usr/local/go/src/database/sql/sql.go:2745 +0x3c
database/sql.withLock(0x4769740, 0xc00ea1b230, 0xc01852fba0)
	/usr/local/go/src/database/sql/sql.go:3184 +0x6d
database/sql.(*Rows).Next(0xc00ea1b200, 0xc012662c78)
	/usr/local/go/src/database/sql/sql.go:2744 +0x87
github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/sqllog.RowsToEvents(0xc00ea1b200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/sqllog/sql.go:233 +0xfa
github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/sqllog.(*SQLLog).List(0xc000ca93c0, 0x47bb880, 0xc00091bc40, 0xc00364be60, 0x52, 0x0, 0x0, 0x1, 0x0, 0xc004580501, ...)
	/go/src/github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/sqllog/sql.go:200 +0x16e
github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured.(*LogStructured).get(0xc001099410, 0x47bb880, 0xc00091bc40, 0xc00364be60, 0x52, 0x0, 0x1, 0x5, 0x1, 0x7f9dfd9c36d0, ...)
	/go/src/github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/logstructured.go:55 +0xa6
github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured.(*LogStructured).Delete(0xc001099410, 0x47bb880, 0xc00091bc40, 0xc00364be60, 0x52, 0x1b8fe8, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/logstructured.go:120 +0x177
github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured.(*LogStructured).ttl.func1(0x47bb880, 0xc00091bc40, 0xc001099410, 0xc003ec5f60)
	/go/src/github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/logstructured.go:305 +0x148
created by github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured.(*LogStructured).ttl
	/go/src/github.com/rancher/k3s/vendor/github.com/rancher/kine/pkg/logstructured/logstructured.go:299 +0xa1

Does your workload perhaps rely heavily on leases? There's this issue in upstream, but since kine sort of does its own thing (and the code even has a note about how it's particularly naieve) this is probably a k3s/kine thing.

Just for giggles, I added log statements to the kine ttl code, and raised the stock kine log messages from debug to info. The current ttl approach is pretty rough - not only do you get a goroutine per event (as expected), you also get a goroutine per non-compacted revision for keys that are replaced (such as leases). All of theses stacked goroutines then expire at once and all try to delete the already-deleted kv store entries.
This is all just against a bog-standard install with nothing running; if there was an actual workload I imagine things could get a lot worse.
time="2020-04-09T16:12:07.844838667-07:00" level=info msg="Starting k3s v1.17.4+k3s-937b13b3 (937b13b3)"
time="2020-04-09T16:12:07.845030968-07:00" level=info msg="Cluster bootstrap already complete"
time="2020-04-09T16:12:07.856245067-07:00" level=info msg="CREATE /registry/health, size=17, lease=0 => rev=774, err=rpc error: code = InvalidArgument desc = etcdserver: duplicate key given in txn request"
time="2020-04-09T16:12:07.856302465-07:00" level=info msg="Kine listening on unix://kine.sock"
time="2020-04-09T16:12:07.856951342-07:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Flag --basic-auth-file has been deprecated, Basic authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments.
I0409 16:12:07.857508    1051 server.go:622] external host was not specified, using 10.0.1.248
I0409 16:12:07.857829    1051 server.go:163] Version: v1.17.4+k3s-937b13b3
I0409 16:12:07.862080    1051 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass.
I0409 16:12:07.862319    1051 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota.
I0409 16:12:07.864429    1051 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass.
I0409 16:12:07.864452    1051 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota.
time="2020-04-09T16:12:07.867028766-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1a8aade29, revision=186, duration=3660"
time="2020-04-09T16:12:07.867162243-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=289, duration=15"
time="2020-04-09T16:12:07.867206365-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1a8aade29, revision=186, duration=3660"
time="2020-04-09T16:12:07.867225501-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=295, duration=15"
time="2020-04-09T16:12:07.867163461-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=287, duration=15"
time="2020-04-09T16:12:07.867262435-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1e70349f8, revision=214, duration=3660"
time="2020-04-09T16:12:07.867284642-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa4502dc, revision=194, duration=3660"
time="2020-04-09T16:12:07.867297731-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b414fda18f, revision=217, duration=3660"
time="2020-04-09T16:12:07.867309358-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf61879b4, revision=359, duration=3660"
time="2020-04-09T16:12:07.867227870-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa44c706, revision=193, duration=3660"
time="2020-04-09T16:12:07.867407221-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik.160448cf6b9df45b, revision=368, duration=3660"
time="2020-04-09T16:12:07.867428302-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner.160448cf6ced26d2, revision=375, duration=3660"
time="2020-04-09T16:12:07.867441312-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448cf6bcd5b68, revision=371, duration=3660"
time="2020-04-09T16:12:07.867342531-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccface4491, revision=360, duration=3660"
time="2020-04-09T16:12:07.867819845-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d3e4254b19, revision=487, duration=3660"
time="2020-04-09T16:12:07.867843300-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c09a5b58, revision=500, duration=3660"
time="2020-04-09T16:12:07.867875039-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4b92a32d3, revision=498, duration=3660"
time="2020-04-09T16:12:07.867254409-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa44f779, revision=192, duration=3660"
time="2020-04-09T16:12:07.867500476-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server.160448cf6ceea0e9, revision=380, duration=3660"
time="2020-04-09T16:12:07.867932698-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4bd5bd9da, revision=499, duration=3660"
time="2020-04-09T16:12:07.867298935-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf49f2efa, revision=356, duration=3660"
time="2020-04-09T16:12:07.867958275-07:00" level=info msg="TTL START   /registry/events/default/dev01.1604497776fcd0d1, revision=679, duration=3660"
time="2020-04-09T16:12:07.867408340-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf4b3d14e, revision=357, duration=3660"
time="2020-04-09T16:12:07.867984776-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777d828b50, revision=685, duration=3660"
time="2020-04-09T16:12:07.867062882-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1a8be6dc2, revision=187, duration=3660"
time="2020-04-09T16:12:07.867069515-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa4502dc, revision=190, duration=3660"
time="2020-04-09T16:12:07.867088400-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa44f779, revision=189, duration=3660"
time="2020-04-09T16:12:07.867106444-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa44f779, revision=192, duration=3660"
time="2020-04-09T16:12:07.867113277-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aba34c66, revision=191, duration=3660"
time="2020-04-09T16:12:07.867120069-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa4502dc, revision=194, duration=3660"
time="2020-04-09T16:12:07.867124678-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa44c706, revision=193, duration=3660"
time="2020-04-09T16:12:07.867131150-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=215, duration=15"
time="2020-04-09T16:12:07.867136079-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1e70349f8, revision=214, duration=3660"
time="2020-04-09T16:12:07.867199001-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1a8be6dc2, revision=187, duration=3660"
time="2020-04-09T16:12:07.867209976-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=291, duration=15"
time="2020-04-09T16:12:07.867218829-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=293, duration=15"
time="2020-04-09T16:12:07.867220716-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aba34c66, revision=191, duration=3660"
time="2020-04-09T16:12:07.867272289-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=299, duration=15"
time="2020-04-09T16:12:07.867276898-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=297, duration=15"
time="2020-04-09T16:12:07.867284458-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=301, duration=15"
time="2020-04-09T16:12:07.867291992-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=300, duration=15"
time="2020-04-09T16:12:07.867292922-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b4702d193e, revision=266, duration=3660"
time="2020-04-09T16:12:07.867304966-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf46e536a, revision=355, duration=3660"
time="2020-04-09T16:12:07.867315019-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf4b3d14e, revision=357, duration=3660"
time="2020-04-09T16:12:07.867318660-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccface17cb, revision=361, duration=3660"
time="2020-04-09T16:12:07.867351307-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccface4f90, revision=362, duration=3660"
time="2020-04-09T16:12:07.867393485-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf46e536a, revision=355, duration=3660"
time="2020-04-09T16:12:07.867395486-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=365, duration=15"
time="2020-04-09T16:12:07.867402479-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf49f2efa, revision=356, duration=3660"
time="2020-04-09T16:12:07.867417762-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccf61879b4, revision=359, duration=3660"
time="2020-04-09T16:12:07.867434750-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccface4491, revision=360, duration=3660"
time="2020-04-09T16:12:07.867444291-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccface4f90, revision=362, duration=3660"
time="2020-04-09T16:12:07.867449040-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns.160448cf6cf85887, revision=381, duration=3660"
time="2020-04-09T16:12:07.867452085-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448ccface17cb, revision=361, duration=3660"
time="2020-04-09T16:12:07.867454681-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server.160448cf6ceea0e9, revision=380, duration=3660"
time="2020-04-09T16:12:07.867460963-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44.160448cf6db51a4a, revision=389, duration=3660"
time="2020-04-09T16:12:07.867461743-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448cf6bcd5b68, revision=371, duration=3660"
time="2020-04-09T16:12:07.867465692-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448cf6d6a469f, revision=383, duration=3660"
time="2020-04-09T16:12:07.867467685-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik.160448cf6b9df45b, revision=368, duration=3660"
time="2020-04-09T16:12:07.867471833-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448cf6e116bdd, revision=397, duration=3660"
time="2020-04-09T16:12:07.867507800-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner.160448cf6ced26d2, revision=375, duration=3660"
time="2020-04-09T16:12:07.867516427-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c.160448cf6db5c945, revision=393, duration=3660"
time="2020-04-09T16:12:07.867516967-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448cf6d6a469f, revision=383, duration=3660"
time="2020-04-09T16:12:07.867522838-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns.160448cf6cf85887, revision=381, duration=3660"
time="2020-04-09T16:12:07.867525604-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448cf6e49e287, revision=399, duration=3660"
time="2020-04-09T16:12:07.867532447-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=417, duration=15"
time="2020-04-09T16:12:07.867532433-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4.160448cf6dd5bad6, revision=398, duration=3660"
time="2020-04-09T16:12:07.867536865-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448cf6fe3a7d0, revision=407, duration=3660"
time="2020-04-09T16:12:07.867542191-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d131e733be, revision=419, duration=3660"
time="2020-04-09T16:12:07.867561748-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d1318a0a03, revision=418, duration=3660"
time="2020-04-09T16:12:07.867562763-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c.160448cf6db5c945, revision=393, duration=3660"
time="2020-04-09T16:12:07.867567990-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d1323a3e1e, revision=421, duration=3660"
time="2020-04-09T16:12:07.867573149-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d132202e4e, revision=420, duration=3660"
time="2020-04-09T16:12:07.867574184-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4.160448cf6dd5bad6, revision=398, duration=3660"
time="2020-04-09T16:12:07.867579141-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d2868ebadd, revision=424, duration=3660"
time="2020-04-09T16:12:07.867581087-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448cf6e116bdd, revision=397, duration=3660"
time="2020-04-09T16:12:07.867583533-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44.160448cf6db51a4a, revision=389, duration=3660"
time="2020-04-09T16:12:07.867584150-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d27374532e, revision=423, duration=3660"
time="2020-04-09T16:12:07.867590786-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448cf6fe3a7d0, revision=407, duration=3660"
time="2020-04-09T16:12:07.867592359-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d2e052e181, revision=431, duration=3660"
time="2020-04-09T16:12:07.867596758-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d28afbb19b, revision=425, duration=3660"
time="2020-04-09T16:12:07.867602408-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d2f1e754ea, revision=433, duration=3660"
time="2020-04-09T16:12:07.867615001-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d131e733be, revision=419, duration=3660"
time="2020-04-09T16:12:07.867619640-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d1318a0a03, revision=418, duration=3660"
time="2020-04-09T16:12:07.867624480-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d2ee5670ae, revision=432, duration=3660"
time="2020-04-09T16:12:07.867625541-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d1323a3e1e, revision=421, duration=3660"
time="2020-04-09T16:12:07.867630220-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d132202e4e, revision=420, duration=3660"
time="2020-04-09T16:12:07.867632204-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d31718aa90, revision=437, duration=3660"
time="2020-04-09T16:12:07.867637113-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d30b4ffbba, revision=436, duration=3660"
time="2020-04-09T16:12:07.867643896-07:00" level=info msg="TTL START   /registry/events/kube-system/rancher.io-local-path.160448d31c4e4789, revision=440, duration=3660"
time="2020-04-09T16:12:07.867645939-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d2868ebadd, revision=424, duration=3660"
time="2020-04-09T16:12:07.867590442-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448cf6e49e287, revision=399, duration=3660"
time="2020-04-09T16:12:07.867649016-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d31b357cb4, revision=438, duration=3660"
time="2020-04-09T16:12:07.867650508-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d27374532e, revision=423, duration=3660"
time="2020-04-09T16:12:07.867655117-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d3abc1f6bb, revision=446, duration=3660"
time="2020-04-09T16:12:07.867656759-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d2e052e181, revision=431, duration=3660"
time="2020-04-09T16:12:07.867659866-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=444, duration=15"
time="2020-04-09T16:12:07.867661218-07:00" level=info msg="TTL START   /registry/events/kube-system/metrics-server-59b5bcbc44-89mqn.160448d28afbb19b, revision=425, duration=3660"
time="2020-04-09T16:12:07.867665978-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d3be6a4c3e, revision=448, duration=3660"
time="2020-04-09T16:12:07.867691301-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff.160448d3e25ae1b4, revision=471, duration=3660"
time="2020-04-09T16:12:07.867695602-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d2f1e754ea, revision=433, duration=3660"
time="2020-04-09T16:12:07.867697502-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik.160448d3e24082bd, revision=469, duration=3660"
time="2020-04-09T16:12:07.867700441-07:00" level=info msg="TTL START   /registry/events/kube-system/coredns-7b8c5b77b4-lmv8v.160448d2ee5670ae, revision=432, duration=3660"
time="2020-04-09T16:12:07.867703137-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d3bb4b4e43, revision=447, duration=3660"
time="2020-04-09T16:12:07.867705962-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d31718aa90, revision=437, duration=3660"
time="2020-04-09T16:12:07.867710681-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d30b4ffbba, revision=436, duration=3660"
time="2020-04-09T16:12:07.867716491-07:00" level=info msg="TTL START   /registry/events/kube-system/rancher.io-local-path.160448d31c4e4789, revision=440, duration=3660"
time="2020-04-09T16:12:07.867718957-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d3e2c2e9b3, revision=477, duration=3660"
time="2020-04-09T16:12:07.867720365-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik.160448d3e3cc1fd8, revision=484, duration=3660"
time="2020-04-09T16:12:07.867720759-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.160448d31b357cb4, revision=438, duration=3660"
time="2020-04-09T16:12:07.867726811-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d3e4254b19, revision=487, duration=3660"
time="2020-04-09T16:12:07.867727038-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d3abc1f6bb, revision=446, duration=3660"
time="2020-04-09T16:12:07.867742600-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik.160448d3e24082bd, revision=469, duration=3660"
time="2020-04-09T16:12:07.867760514-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d3bb4b4e43, revision=447, duration=3660"
time="2020-04-09T16:12:07.867765894-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4007cb193, revision=494, duration=3660"
time="2020-04-09T16:12:07.867770423-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d3be6a4c3e, revision=448, duration=3660"
time="2020-04-09T16:12:07.867775612-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d3e2c2e9b3, revision=477, duration=3660"
time="2020-04-09T16:12:07.867790621-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d3fed8e93c, revision=493, duration=3660"
time="2020-04-09T16:12:07.867796341-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4b92a32d3, revision=498, duration=3660"
time="2020-04-09T16:12:07.867813834-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff.160448d3e25ae1b4, revision=471, duration=3660"
time="2020-04-09T16:12:07.869431035-07:00" level=info msg="LIST /registry/apiextensions.k8s.io/customresourcedefinitions/, start=/registry/apiextensions.k8s.io/customresourcedefinitions/, limit=10001, rev=0 => rev=774, kvs=2, err=<nil>"
time="2020-04-09T16:12:07.867849150-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik.160448d3e3cc1fd8, revision=484, duration=3660"
time="2020-04-09T16:12:07.867055979-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b1aa44c706, revision=188, duration=3660"
time="2020-04-09T16:12:07.867149585-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=285, duration=15"
time="2020-04-09T16:12:07.867151342-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b414fda18f, revision=217, duration=3660"
time="2020-04-09T16:12:07.867154153-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=283, duration=15"
time="2020-04-09T16:12:07.867046291-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=147, duration=15"
time="2020-04-09T16:12:07.867865902-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4007cb193, revision=494, duration=3660"
time="2020-04-09T16:12:07.867882574-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d4305c1f4d, revision=496, duration=3660"
time="2020-04-09T16:12:07.867906198-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4bd5bd9da, revision=499, duration=3660"
time="2020-04-09T16:12:07.867928140-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c09a5b58, revision=500, duration=3660"
time="2020-04-09T16:12:07.867927729-07:00" level=info msg="TTL START   /registry/events/kube-system/helm-install-traefik-tvnn6.160448d4305c1f4d, revision=496, duration=3660"
time="2020-04-09T16:12:07.867937938-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c2d38194, revision=502, duration=3660"
time="2020-04-09T16:12:07.867940963-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c0a0de51, revision=501, duration=3660"
time="2020-04-09T16:12:07.867944781-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d5398fb01f, revision=508, duration=3660"
time="2020-04-09T16:12:07.867946804-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c7286c34, revision=503, duration=3660"
time="2020-04-09T16:12:07.867951072-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d549101bd3, revision=510, duration=3660"
time="2020-04-09T16:12:07.867952394-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d545718c3d, revision=509, duration=3660"
time="2020-04-09T16:12:07.867969647-07:00" level=info msg="TTL START   /registry/events/default/dev01.16044977774ef0d4, revision=680, duration=3660"
time="2020-04-09T16:12:07.867977071-07:00" level=info msg="TTL START   /registry/events/default/dev01.16044977775c18ca, revision=681, duration=3660"
time="2020-04-09T16:12:07.867989725-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777d824f3d, revision=684, duration=3660"
time="2020-04-09T16:12:07.867992870-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777af8de54, revision=683, duration=3660"
time="2020-04-09T16:12:07.867998211-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.1604497790002744, revision=688, duration=3660"
time="2020-04-09T16:12:07.867999302-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777d829d64, revision=687, duration=3660"
time="2020-04-09T16:12:07.868004423-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.1604497796c10200, revision=690, duration=3660"
time="2020-04-09T16:12:07.868004842-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.16044977929b28ec, revision=689, duration=3660"
time="2020-04-09T16:12:07.868010855-07:00" level=info msg="TTL START   /registry/events/default/dev01.16044979feb67da8, revision=704, duration=3660"
time="2020-04-09T16:12:07.868012707-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.16044977cb54b038, revision=692, duration=3660"
time="2020-04-09T16:12:07.868017527-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=769, duration=15"
time="2020-04-09T16:12:07.868028397-07:00" level=info msg="TTL START   /registry/events/kube-system/rancher.io-local-path.1604497bc62326a1, revision=707, duration=3660"
time="2020-04-09T16:12:07.869664067-07:00" level=info msg="COUNT /registry/apiextensions.k8s.io/customresourcedefinitions/ => rev=774, count=2, err=<nil>"
time="2020-04-09T16:12:07.867142501-07:00" level=info msg="TTL START   /registry/events/default/dev01.160448b4702d193e, revision=266, duration=3660"
time="2020-04-09T16:12:07.867869989-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d3fed8e93c, revision=493, duration=3660"
time="2020-04-09T16:12:07.869851403-07:00" level=info msg="LIST /registry/apiextensions.k8s.io/customresourcedefinitions/, start=/registry/apiextensions.k8s.io/customresourcedefinitions/, limit=10001, rev=0 => rev=774, kvs=2, err=<nil>"
time="2020-04-09T16:12:07.870337420-07:00" level=info msg="COUNT /registry/apiextensions.k8s.io/customresourcedefinitions/ => rev=774, count=2, err=<nil>"
time="2020-04-09T16:12:07.870419103-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c2d38194, revision=502, duration=3660"
time="2020-04-09T16:12:07.870437297-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c0a0de51, revision=501, duration=3660"
time="2020-04-09T16:12:07.870475629-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d5398fb01f, revision=508, duration=3660"
time="2020-04-09T16:12:07.870504132-07:00" level=info msg="TTL START   /registry/events/kube-system/svclb-traefik-72qdf.160448d4c7286c34, revision=503, duration=3660"
time="2020-04-09T16:12:07.870517253-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d549101bd3, revision=510, duration=3660"
time="2020-04-09T16:12:07.870535050-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=515, duration=15"
time="2020-04-09T16:12:07.870521605-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.160448d545718c3d, revision=509, duration=3660"
time="2020-04-09T16:12:07.870560724-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=560, duration=15"
time="2020-04-09T16:12:07.870565824-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=553, duration=15"
time="2020-04-09T16:12:07.870578397-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=588, duration=15"
time="2020-04-09T16:12:07.870584559-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=581, duration=15"
time="2020-04-09T16:12:07.870588456-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=602, duration=15"
time="2020-04-09T16:12:07.870620567-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=651, duration=15"
time="2020-04-09T16:12:07.870661528-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777af8de54, revision=683, duration=3660"
time="2020-04-09T16:12:07.870612632-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=644, duration=15"
time="2020-04-09T16:12:07.870677463-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777d828b50, revision=685, duration=3660"
time="2020-04-09T16:12:07.870688965-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.1604497790002744, revision=688, duration=3660"
time="2020-04-09T16:12:07.870699485-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.1604497796c10200, revision=690, duration=3660"
time="2020-04-09T16:12:07.870737746-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=698, duration=15"
time="2020-04-09T16:12:07.870620977-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=658, duration=15"
time="2020-04-09T16:12:07.870766976-07:00" level=info msg="TTL START   /registry/events/kube-system/rancher.io-local-path.1604497bc62326a1, revision=707, duration=3660"
time="2020-04-09T16:12:07.870782059-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=720, duration=15"
time="2020-04-09T16:12:07.870816218-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=755, duration=15"
time="2020-04-09T16:12:07.870828261-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=748, duration=15"
time="2020-04-09T16:12:07.870730904-07:00" level=info msg="TTL START   /registry/events/kube-system/local-path-provisioner-58dbf4c86c-c89mx.16044977929b28ec, revision=689, duration=3660"
time="2020-04-09T16:12:07.870571855-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=574, duration=15"
time="2020-04-09T16:12:07.870578648-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=567, duration=15"
time="2020-04-09T16:12:07.870605037-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=630, duration=15"
time="2020-04-09T16:12:07.870608324-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=623, duration=15"
time="2020-04-09T16:12:07.870614555-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=637, duration=15"
time="2020-04-09T16:12:07.870590961-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=595, duration=15"
time="2020-04-09T16:12:07.870627219-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=671, duration=15"
time="2020-04-09T16:12:07.870627664-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=665, duration=15"
time="2020-04-09T16:12:07.870647427-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=677, duration=15"
time="2020-04-09T16:12:07.870647772-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=672, duration=15"
time="2020-04-09T16:12:07.870654535-07:00" level=info msg="TTL START   /registry/events/default/dev01.1604497776fcd0d1, revision=679, duration=3660"
time="2020-04-09T16:12:07.870655472-07:00" level=info msg="TTL START   /registry/events/default/dev01.16044977774ef0d4, revision=680, duration=3660"
time="2020-04-09T16:12:07.870669398-07:00" level=info msg="TTL START   /registry/events/default/dev01.16044977775c18ca, revision=681, duration=3660"
time="2020-04-09T16:12:07.870682923-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777d824f3d, revision=684, duration=3660"
time="2020-04-09T16:12:07.870693614-07:00" level=info msg="TTL START   /registry/events/default/dev01.160449777d829d64, revision=687, duration=3660"
time="2020-04-09T16:12:07.870595600-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=616, duration=15"
time="2020-04-09T16:12:07.870550866-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=538, duration=15"
time="2020-04-09T16:12:07.870529931-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=526, duration=15"
time="2020-04-09T16:12:07.870597283-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=609, duration=15"
time="2020-04-09T16:12:07.870745946-07:00" level=info msg="TTL START   /registry/events/kube-system/traefik-5f7b78ff-gkvkw.16044977cb54b038, revision=692, duration=3660"
time="2020-04-09T16:12:07.870752539-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=705, duration=15"
time="2020-04-09T16:12:07.870756496-07:00" level=info msg="TTL START   /registry/events/default/dev01.16044979feb67da8, revision=704, duration=3660"
time="2020-04-09T16:12:07.870763058-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=713, duration=15"
time="2020-04-09T16:12:07.870570693-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=546, duration=15"
time="2020-04-09T16:12:07.870777882-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=727, duration=15"
time="2020-04-09T16:12:07.870806560-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=741, duration=15"
time="2020-04-09T16:12:07.870811108-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=734, duration=15"
time="2020-04-09T16:12:07.870825792-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=769, duration=15"
time="2020-04-09T16:12:07.870830671-07:00" level=info msg="TTL START   /registry/masterleases/10.0.1.248, revision=762, duration=15"

Seems like there are a couple issues here:

kine is not compacted on startup, leading to re-creation of ttl goroutines for superceded entry revisions.
ttl value is tracked as kv.Lease, but the last update time is not stored, so the effective TTL is reset when the node restarts.
single goroutine per store entry ttl is inefficient.

@brandond this is excellent discovery work, thank you!

@brandond Thanks! The TTL eviction in kine is an extremely naive approach that (up until now) just seemed to work.  What I guess is happening is that in general a goroutine does not create a thread, but in this situation we are creating a lot of short lived goroutines that are making a CGO call, which then all block.  So I imagine there is a large number of records that are getting purged at once which is then creating way too many OS threads due to the CGO.  Seem like we just need to implement a less naive approach to TTL in kine.

The stupidest solution would just be to put a mutex in the goroutine around the Delete call so that they are sequential (which they are reqardless because of sqlite).  But adding the mutex would allow the goroutines to not block on CGO, but instead a mutex.  We can have millions of goroutines with no issue.

@ibuildthecloud yeah, having it block on a mutex instead of somewhere deep in the bowels of sqlite is probably an easy immediate fix.
I'm looking for stuff to do in Go, I thought it might be fun to try to write a dedicated goroutine to handle TTL expiration. Fixing the TTL resetting on startup would probably require a schema change to track key change timestamps. I'll put together a PR and let you see if it looks too ugly.

the is fixed on @pdisciascio setup when using the patched version of kine that includes the mutex in the ttl event handling, please reopen if you still seeing the problem.

authors: brandond dweomer galal-hussein ibuildthecloud pdisciascio


text:
KillPodSandboxError: failed to stop container: container does not exist

Version:
k3s version v1.17.4+k3s1 (3eee8ac3)
K3s arguments:
server --disable=servicelb --disable=local-storage --disable=traefik
Describe the bug
I have a couple of Pods that remain in Terminating for a long time, and errors in the logs indicate that the containers fail to be terminated because according to runc, they do not exist.
To Reproduce
Unknown.
Expected behavior
If a Pod is Terminating, it should eventually cease to exist.
Actual behavior
I have four pods stuck in Terminating on a node called kathleen (Raspberry Pi 4). One is in Terminating for weeks, one since a few days, two more are stuck in Terminating since today:
$ kubectl get pods -A -o wide | grep Terminating
openebs          openebs-provisioner-c4c58475c-ffqzh                               1/1     Terminating        21         44d     10.42.0.209       kathleen   <none>           <none>
default          registry-5d89d66c56-9zpvt                                         0/1     Terminating        0          44d     10.42.0.221       kathleen   <none>           <none>
openebs          openebs-snapshot-operator-84fb87cf4c-q4f4v                        1/2     Terminating        46         44d     10.42.0.208       kathleen   <none>           <none>
openebs          openebs-localpv-provisioner-848db54f69-98452                      1/1     Terminating        75         44d     10.42.0.205       kathleen   <none>           <none>

The last 200 lines of the log are here: https://pastebin.com/DS3AZzux - it is visible that the k3s server attempts deletion of the containers, but fails. These messages, which occur a lot, show that the most clearly:
E0613 19:43:07.848664   18509 remote_runtime.go:128] StopPodSandbox "0e7524296d0370cd7a9cf35a3e2d952b4064f98bc2556a09279e126a246e3e22" from runtime service failed: rpc error: code = Unknown desc = failed to stop container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e": failed to kill container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e": unknown error after kill: runc did not terminate sucessfully: container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e" does not exist
: unknown
E0613 19:43:07.849136   18509 kuberuntime_manager.go:898] Failed to stop sandbox {"containerd" "0e7524296d0370cd7a9cf35a3e2d952b4064f98bc2556a09279e126a246e3e22"}
[...]
E0613 19:43:07.849807   18509 pod_workers.go:191] Error syncing pod c4efbb3b-3c1a-47fd-8cdb-a742571b5eeb ("openebs-localpv-provisioner-848db54f69-98452_openebs(c4efbb3b-3c1a-47fd-8cdb-a742571b5eeb)"), skipping: error killing pod: [failed to "KillContainer" for "openebs-provisioner-hostpath" with KillContainerError: "rpc error: code = Unknown desc = failed to stop container \"d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e\": unknown error after kill: runc did not terminate sucessfully: container \"d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e\" does not exist\n: unknown", failed to "KillPodSandbox" for "c4efbb3b-3c1a-47fd-8cdb-a742571b5eeb" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container \"d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e\": failed to kill container \"d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e\": unknown error after kill: runc did not terminate sucessfully: container \"d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e\" does not exist\n: unknown"]

The container does show in crictl:
kathleen [~]$ sudo k3s crictl ps --id d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e --all --verbose
ID: d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e
PodID: 0e7524296d0370cd7a9cf35a3e2d952b4064f98bc2556a09279e126a246e3e22
Name: openebs-provisioner-hostpath
Attempt: 75
State: Running
Image: sha256:eb07d8dcd675112cf429395e5c556dc74489b83cd4f3bef701b1234b1cd06d5f
Created: 4 weeks ago
Labels:
	 ->
	 ->
	 ->
	 ->
	io.kubernetes.container.name -> openebs-provisioner-hostpath
	io.kubernetes.pod.name -> openebs-localpv-provisioner-848db54f69-98452
	io.kubernetes.pod.namespace -> openebs
	io.kubernetes.pod.uid -> c4efbb3b-3c1a-47fd-8cdb-a742571b5eeb
Annotations:
	 ->
	 ->
	 ->
	 ->
	 ->
	io.kubernetes.container.hash -> 1a91b36d
	io.kubernetes.container.restartCount -> 75
	io.kubernetes.container.terminationMessagePath -> /dev/termination-log
	io.kubernetes.container.terminationMessagePolicy -> File
	io.kubernetes.pod.terminationGracePeriod -> 30

Indeed, when I try to stop it manually:
kathleen [~]$ sudo k3s crictl stop d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e
FATA[2020-06-13T19:51:30.961591215Z] Stopping the container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e" failed: rpc error: code = Unknown desc = failed to stop container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e": unknown error after kill: runc did not terminate sucessfully: container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e" does not exist
: unknown
kathleen [~]$ sudo k3s crictl rm -f d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e
ERRO[2020-06-13T19:52:05.463324646Z] stopping the container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e" failed: rpc error: code = Unknown desc = failed to kill container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e": unknown error after kill: runc did not terminate sucessfully: container "d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e" does not exist
: unknown
FATA[2020-06-13T19:52:05.463526088Z] unable to remove container(s)

Logs do work, though, so the container clearly ran at some point:
kathleen [~]$ sudo k3s crictl logs d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e
I0512 07:56:53.973044       1 start.go:65] Starting Provisioner...
I0512 07:56:54.133147       1 leaderelection.go:242] attempting to acquire leader lease  openebs/openebs.io-local...
I0512 07:57:11.832701       1 leaderelection.go:252] successfully acquired lease openebs/openebs.io-local
I0512 07:57:11.833729       1 controller.go:769] Starting provisioner controller openebs.io/local_openebs-localpv-provisioner-848db54f69-98452_093a1cbf-491d-4f56-9122-57dc14c8b527!
I0512 07:57:11.833912       1 event.go:281] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"openebs", Name:"openebs.io-local", UID:"97f3bdbb-3991-4e50-85c6-80eb78612b08", APIVersion:"v1", ResourceVersion:"57828734", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' openebs-localpv-provisioner-848db54f69-98452_093a1cbf-491d-4f56-9122-57dc14c8b527 became leader
I0512 07:57:12.034417       1 controller.go:818] Started provisioner controller openebs.io/local_openebs-localpv-provisioner-848db54f69-98452_093a1cbf-491d-4f56-9122-57dc14c8b527!
I0512 08:26:19.146971       1 leaderelection.go:288] failed to renew lease openebs/openebs.io-local: failed to tryAcquireOrRenew context deadline exceeded
F0512 08:26:19.148250       1 controller.go:847] leaderelection lost

I'm getting the feeling I should report this upstream to runc, but I'm not sure where to do that or what information to include. If you agree, could you give me some pointers for that?

I just noticed that I could remove the container using k3s ctr containers rm:
kathleen [~]$ sudo k3s ctr containers ls | grep d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e
d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e    quay.io/openebs/provisioner-localpv-arm64:1.9.0                                                         io.containerd.runc.v2
kathleen [~]$ sudo k3s ctr containers rm d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e
kathleen [~]$ sudo k3s ctr containers ls | grep d99085b0dbfde2a40ed79423dd6b8f111bd3f7f8e4e44821790124c4e4e0a95e
kathleen [~]$

For the interest of investigating this issue, I will keep two Pods in Terminating (not stopping their containers using k3s ctr containers rm for now), so we can investigate their state a bit more if you'd like.

Actually, this is exactly this issue: https://github.com/containerd/containerd/issues/4016

In https://github.com/containerd/containerd/issues/4016 it is reported that containerd 1.4.0 fixes this issue, but I don't know how to confirm, since I don't know how to reproduce this.


text:
Changed SELinux/semanage error to fatal

If the install errors out on semanage not found, an error is
thrown as a 'file not found' for error. Updating to 'fatal' resolves
this as the script then exits as intended and throws an 'error'.
Actual output (before):
sh: line 347: error: command not found
Output (after):
[ERROR] SELinux is enabled but semanage is not found

Thanks @ibuildthecloud :)
(I wasn't sure what to do with that particular CI failure 😨 )

how to solve

@13567436138 You would need to have policycoreutils-python or similar installed on your machine (CentOS/RedHat)
try yum install policycoreutils-python
:)


text:
Config file is not updating properly

Hi,
I was trying to create an bridge using network-config.yml

rancher:
network:
interfaces:
wanbr:
bridge: true
address: 192.168.15.15/24
gateway: 192.168.15.1
etho:
bridge: wanbr
eth1:
bridge: wanbr

The configuration was updated and applied on the ROS using this comment

ros config merge -i network-config.yml

The issue is i can't delete the wanbr from the os by updating this configuration

rancher:
network:
interfaces:
etho:
dhcp: true
eth1:
dhcp: false

But the bridge is not removed from the ROS
Please help to update the configuration dynamically
Regards
Tamil

hi
@niusmallnan  is there is any answer available for me to proceed
Regards
Tamil

@itstamil
The ros config merge command will only superimpose or replace the item in the cloud-config file, and will not remove the original item. You can edit the cloud-config.yml file to remove unwanted items.

@kingsd041
Even i removed an interface from config file in path /var/lib/rancher/conf/cloud-config.yml
Then i run command system-docker restart network
even its not removing the bridge

@itstamil system-docker restart network will not remove the bridge, you can try to restart RancherOS

repo: rancher/os


text:
#77 Addressing backlog issue about registered AKS clusters

While going through backlog, I did a review of #77. Some, but not all, of the problems mentioned in the issue have been addressed. Rather than creating a PR for the entire issue, I'm going to pick through it to make sure that the problems are still relevant to the current state of the docs.
In https://github.com/rancher/rancher-docs/issues/77#issuecomment-1245965075 from the thread:

Please also look at https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/registered-clusters/#additional-features-for-registered-eks-and-gke-clusters where it does not mention AKS clusters (my understanding is that AKS clusters can be registered too):

Additional Features for Registered EKS and GKE Clusters
Registering an Amazon EKS cluster or GKE cluster allows Rancher to treat it as though it were created in Rancher.
Amazon EKS clusters and GKE clusters can now be registered in Rancher. For the most part, these registered clusters are treated the same way as clusters created in the Rancher UI, except for deletion.
When you delete an EKS cluster or GKE cluster that was created in Rancher, the cluster is destroyed. When you delete a cluster that was registered in Rancher, it is disconnected from the Rancher server, but it still exists and you can still access it in the same way you did before it was registered in Rancher.
The capabilities for registered clusters are listed in the table on this page.

This was answered with https://github.com/rancher/rancher-docs/issues/77#issuecomment-1245965081:

https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/

Thanks for your contribution. AKS came along a little later and can be registered as well, as you noted.
[...] add "AKS" to the verbiage, tables, and sections wherein we only mention EKS and GKE cluster registration in the links  provided. Thanks!

At the time, this related to Rancher v2.6, but the v2.6 page wasn't updated. v2.7 does mention AKS, but referred to it as "Azure AKS," which is inaccurate. The section also needed a style refresh.
I updated v2.6 and 2.7 to match. v2.5 was left alone as I'm not sure if it was intended to be updated in the original thread.

While going through the repo, I also found an instance of "EKS and GKE" verbiage left on CIS Scans. Since the intro to that page states,

The CIS scans can run on any Kubernetes cluster, including hosted Kubernetes providers such as EKS, AKS, and GKE

are we safe in updating the following to include AKS?

The EKS and GKE cluster scan profiles are based on CIS Benchmark versions that are specific to those types of clusters.

The rancher-cis-benchmark supports the CIS 1.6 Benchmark version.

For RKE Kubernetes clusters, the RKE Permissive 1.6 profile is the default.
EKS and GKE have their own CIS Benchmarks published by kube-bench. The corresponding test profiles are used by default for those clusters.
For RKE2 Kubernetes clusters, the RKE2 Permissive 1.6 profile is the default.
For cluster types other than RKE, RKE2, EKS and GKE, the Generic CIS 1.5 profile will be used by default.

I think this might be a question for @jiaqiluo.

@martyav, I do not have an answer because I am not the maintainer of the rancher-cis-benchmark app and have not tired/validated it on AKS cluster.  I would like to bring @macedogm  into the loop.

@martyav / @jiaqiluo thanks for flagging this. The CIS chart is actually maintained by team/area3 (but is transitioning internally to another team). @mitulshah-suse and @rayandas can you help us with this update, please?
If we are going to update the CIS page, we also need to update the profiles to add the 1.20 and 1.23 versions to each distro.

@macedogm We have an open PR related to 1.20 and 1.23 (#209). We can address that issue there -- there's already been a lot of work put towards it, but it's been awaiting QA.

authors: jiaqiluo macedogm martyav

repo: rancher/rancher-docs


text:
Add secrets of type "Certificate" to workload

Rancher versions:
rancher/server or rancher/rancher: 2.0.7
rancher/agent or rancher/rancher-agent: 2.0.7
Infrastructure Stack versions:
healthcheck:
ipsec:
network-services:
scheduler:
kubernetes (if applicable): 1.11.1
Docker version: (docker version,docker info preferred) 17.3.2
Operating system and kernel: (cat /etc/os-release, uname -r preferred) RancherOS 1.4.0
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) OpenStack
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB) Single Node internal DB
Environment Template: (Cattle/Kubernetes/Swarm/Mesos) Kubernetes
Rancher 2 offers to add secrets as volumes when deploying a workload.
In our case, I used cert-manager to generate a TLS certificate and want to use it inside our ApacheDS container.
The secret has been created in the same namespace:
$ kubectl get secret
NAME                        TYPE                                  DATA      AGE
apacheds-tls                kubernetes.io/tls                     2         16m

but I can't select it in Rancher UI. I assume secrets are filtered - would it be reasonable to allow selecting certificates as well?

We need to add use a certificate options when we add volume.

Version - 2.1 master 9/4
@loganhz - It's not in list?

authors: loganhz stieler-it tfiduccia

repo: rancher/rancher


text:
Error while getting agent config: invalid response 500: Failed to validate cluster: Hostname_override [xxx-xxx-KUB02] for host

What kind of request is this (question/bug/enhancement/feature request):
question/bug
Steps to reproduce (least amount of steps as possible):
Spin up rancher 2.1.1 server and add a cluster with etcd, controlpanel and worker and run it on an server with hostname in xxx-xxx-xxx format.
Result:
ime="2018-10-25T14:49:16Z" level=info msg="Connecting to wss://testrancher.xxx.org/v3/connect/register with token 2gprbmc9lc5wmbbc8d7cmcj8qtq62ckx2n229dmlqlbdp8wxl9t6rz"
time="2018-10-25T14:49:16Z" level=info msg="Connecting to proxy" url="wss://testrancher.xxx.org/v3/connect/register"
time="2018-10-25T14:49:16Z" level=info msg="Error while getting agent config: invalid response 500: Failed to validate cluster: Hostname_override [xxx-xxx-KUB02] for host (1) is not valid: [a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(\.a-z0-9?)')]"
time="2018-10-25T14:49:21Z" level=info msg="Error while getting agent config: invalid response 500: Failed to validate cluster: Hostname_override [xxx-xxx-KUB02] for host (1) is not valid: [a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(\.a-z0-9?)')]"
Other details that may be helpful:
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.1.1
Installation option (single install/HA): single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
Machine type (cloud/VM/metal) and specifications (CPU/memory): VM, CPU 2, Memory - 4GB
Kubernetes version (use kubectl version): v1.11.3 and also tried v1.10.5

Before I used v2.0.0 of rancher and it worked without any issue, seeing this issue in v2.1.1 of rancher.
All our hostnames are in this format xxx-xxx-xxx. Could you let me know, how to fix this issue?

The validation was added in https://github.com/rancher/rancher/issues/14571, the error describes the validation used which is lower case. Can you try with hostnames not containing uppercase characters?

@superseb  All our hostnames have uppercase characters, Is there a workaround for it. If not could you please add support to allow uppercase characters?

Current workaround is adding --node-name lowercase-hostname at the end of the generated docker run command. We can discuss if we want to lowercase hostname by default (currently it's hostname -s)

Thank you for your quick response. I think it should either support upper case hostnames or change them to lowercase by default instead of adding --node-name.

@superseb, What is the workaround for #16508 and #16541 when the issue rise during an upgrade ?
Thanks

Reproduction
Rancher version: 2.1.1
Installation option: single
Step: Added a cluster from an existing node which hostname is uppercased, put all roles on one node
Result:
get the following error:
Failed to validate cluster: Hostname_override [xxx-xxx-KUB02] for host (1) is not valid: [a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(\.a-z0-9?)')]"

Validation
Rancher version: 2.1.2-rc11
Installation option: single
Step: Added a cluster from an existing node which hostname is uppercased, put all roles on one node
Result:
Successfully added the cluster.

Conclusion
Bug fixed verified.

Is this will also fix #16541and #16508 when upgrading an old cluster create with v2.0.x on UpperCase named node ?

@david-maumenee Hi, the issue #16541 and #16508 are fixed in the coming Rancher v2.1.2.
Please see my comment here for testing result in v2.1.2-rc12: https://github.com/rancher/rancher/issues/16541#issuecomment-439117764

authors: david-maumenee jiaqiluo rammaram06 superseb


text:
WorkloadLabels error causes app details to not show workloads

What kind of request is this (question/bug/enhancement/feature request):
bug
Steps to reproduce (least amount of steps as possible):

Launch longhorn, drone or other app with default parameters, I have not done any other operations.
The pods of the workloads list are all running

Result:
The app active number is 0, and there is nothing in the workload of the details page.
WorkloadLabels appId tag is incorrect.
Other details that may be helpful:
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
rancher/rancher:master 1/30

Installation option (single install/HA):
single , HA

Version: master 2.2 (1/31/19)
Confirmed fixed.


text:
Custom cluster - system tool - cert rotate- error while running cert rotate command

What kind of request is this (question/bug/enhancement/feature request): bug
Steps to reproduce (least amount of steps as possible):

On rancher v2.1.8. have a custom cluster. (Node - Amazon EC2 instance)
Run the command - ./system-tools config --url https://x.x.x.x/v3 --token "token-xxxxxxxxxxxxxx" --cluster c-lx67x (system tools from : https://github.com/rancher/system-tools/releases/tag/v0.1.1-rc7)
config file will be generated. Enter the access file name and file path.
Run the command - ./system-tools cert rotate --url https://x.x.x.x/v3 --token "token-xxxxxxxxxxxxxx" --cluster c-lx67x --config c-lx67x

Expected Result:
certs are rotated successfully.
Actual Result:
Error is seen - WARN[0002] Failed to set up SSH tunneling for host [y.y.y.y]: Can't retrieve Docker Info: error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info: Unable to access node with address [y.y.y.y] using SSH. Please check if the configured key or specified key file is a valid SSH Private Key. Error: Error configuring SSH: ssh: no key found
Command and error:
INFO[0000] Setup rkeconfig for cluster [c-lx67x]
INFO[0000] Get kubeconfig for cluster [c-lx67x]
INFO[0000] Write temporary kubeconfig for cluster [c-lx67x]
INFO[0000] Successfully wrote temporary kubeconfig file for cluster [c-lx67x] at [./kube_config_c-lx67x]
INFO[0000] Fetching cluster [c-lx67x] full state from kubernetes
INFO[0000] Initiating Kubernetes cluster
INFO[0000] Rotating Kubernetes cluster certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0001] [certificates] Generating etcd-x.y.z.x certificate and key
INFO[0001] [certificates] Generating Kube Proxy certificates
INFO[0001] [certificates] Generating Kube Controller certificates
INFO[0002] [certificates] Generating Kube Scheduler certificates
INFO[0002] [certificates] Generating Node certificate
INFO[0002] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0002] Successfully Deployed state file at [./c-lx67x.rkestate]
INFO[0002] Rebuilding Kubernetes cluster with rotated certificates
INFO[0002] [dialer] Setup tunnel for host [y.y.y.y]
WARN[0002] Failed to set up SSH tunneling for host [y.y.y.y]: Can't retrieve Docker Info: error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info: Unable to access node with address [y.y.y.y] using SSH. Please check if the configured key or specified key file is a valid SSH Private Key. Error: Error configuring SSH: ssh: no key found
WARN[0002] Removing host [y.y.y.y] from node lists
FATA[0002] Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) [y.y.y.y] ```

**Other details that may be helpful:**

**Environment information**
- Rancher version (`rancher/rancher`/`rancher/server` image tag or shown bottom left in the UI): rancher:v2.1.8
- Installation option (single install/HA): single

**Cluster information**
- Cluster type (Hosted/Infrastructure Provider/Custom/Imported): custom cluster

Error Failed to set up SSH tunneling for host [x.x.x.x]: Can't retrieve Docker Info: error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info: Unable to access the service on /var/run/docker.sock. The service might be still starting up. Error: ssh: rejected: connect failed (open failed) seen while doing it for a custom cluster, is related to this issue - https://github.com/rancher/rancher/issues/19555

Instead of system-tools, cert rotation was backported to rancher rke/rancher. The feature can be tested with rancher v2.1.9-rc2. Closing this issue.


text:
Schedule CIS Scan Enable value is not set on the UI

What kind of request is this (question/bug/enhancement/feature request):
Steps to reproduce (least amount of steps as possible):

Enable Schedule CIS scan on a cluster.
Save the changes made.
Schedule scan will run in the time specified in the cron time.
Go to Edit Cluster --> Advanced Options --> Scheduled CIS scan --> The values "Yes" is not saved.

Expected Result:
Yes/No for Scheduled CIS scan should be saved for a cluster
Other details that may be helpful:
Related to: https://github.com/rancher/rancher/issues/26202#issuecomment-602879342
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): master-head - ui tag: latest2
Installation option (single install/HA): single

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): rke do
Kubernetes version (use kubectl version):

1.17

I also added a spelling fix to this issue that @sowmyav27 found.
I corrected Maxiumum Worker Nodes Unavailable

Verified on latest master-head commit id: 29e223653 - ui tag: latest2

Reopening the issue as when the user selects Schedule CIS scan enabled: Yes, the fields - Profile, retention count and Cron time, are NOT enabled on the UI

Verified on latest master- ui tag: latest2

Enabled Schedule scan - Yes, Given in a cron time: 6 * * * *, default value for retention: 24 and Profile: Permissive CIS 1.4
Error is seen on clicking on Save cluster.

scheduledClusterScan is sent as null
In the network tab dev tools:

scheduledClusterScan: {enabled: true, scanConfig: null, scheduleConfig: null, type: "/v3/schemas/scheduledClusterScan"}
enabled: true
scanConfig: null
scheduleConfig: null
type: "/v3/schemas/scheduledClusterScan"

Verified on latest master-head - commit id: 44f73ad06 ui tag: latest2

User is able to enable Schedule CIS scan on the UI.
Values set for the params is saved in the cluster
Schedule scan is successfully run on the cluster. and report is generated