text:
Pod creation too slow

In addition to pod creation not working reliably, pod creation is far too slow. At first pods are created fairly quickly, but after a while the pod creation rate significantly decreases.
See  #5750
The log below is for an experiment that tries to create 50 * 5 pods (50 nodes, 5 pods per node).
I0322 05:22:21.546217    1886 serve_hostnames.go:53] Starting serve_hostnames soak test with queries=10000 and perNode=5 upTo=100
I0322 05:22:21.912549    1886 serve_hostnames.go:78] Nodes found on this cluster:
I0322 05:22:21.912580    1886 serve_hostnames.go:80] 0: kubernetes-minion-08vl.c.kubernetes-scale.internal
I0322 05:22:21.912591    1886 serve_hostnames.go:80] 1: kubernetes-minion-14sw.c.kubernetes-scale.internal
I0322 05:22:21.912598    1886 serve_hostnames.go:80] 2: kubernetes-minion-16y7.c.kubernetes-scale.internal
I0322 05:22:21.912606    1886 serve_hostnames.go:80] 3: kubernetes-minion-1xeu.c.kubernetes-scale.internal
I0322 05:22:21.912613    1886 serve_hostnames.go:80] 4: kubernetes-minion-2y1m.c.kubernetes-scale.internal
I0322 05:22:21.912628    1886 serve_hostnames.go:80] 5: kubernetes-minion-37uc.c.kubernetes-scale.internal
I0322 05:22:21.912634    1886 serve_hostnames.go:80] 6: kubernetes-minion-3gif.c.kubernetes-scale.internal
I0322 05:22:21.912641    1886 serve_hostnames.go:80] 7: kubernetes-minion-3p70.c.kubernetes-scale.internal
I0322 05:22:21.912648    1886 serve_hostnames.go:80] 8: kubernetes-minion-3sbf.c.kubernetes-scale.internal
I0322 05:22:21.912654    1886 serve_hostnames.go:80] 9: kubernetes-minion-4aej.c.kubernetes-scale.internal
I0322 05:22:21.912661    1886 serve_hostnames.go:80] 10: kubernetes-minion-4l4h.c.kubernetes-scale.internal
I0322 05:22:21.912668    1886 serve_hostnames.go:80] 11: kubernetes-minion-59d2.c.kubernetes-scale.internal
I0322 05:22:21.912675    1886 serve_hostnames.go:80] 12: kubernetes-minion-6q3q.c.kubernetes-scale.internal
I0322 05:22:21.912682    1886 serve_hostnames.go:80] 13: kubernetes-minion-7rlr.c.kubernetes-scale.internal
I0322 05:22:21.912703    1886 serve_hostnames.go:80] 14: kubernetes-minion-87a6.c.kubernetes-scale.internal
I0322 05:22:21.912711    1886 serve_hostnames.go:80] 15: kubernetes-minion-8kpq.c.kubernetes-scale.internal
I0322 05:22:21.912732    1886 serve_hostnames.go:80] 16: kubernetes-minion-95vc.c.kubernetes-scale.internal
I0322 05:22:21.912741    1886 serve_hostnames.go:80] 17: kubernetes-minion-a5zj.c.kubernetes-scale.internal
I0322 05:22:21.912749    1886 serve_hostnames.go:80] 18: kubernetes-minion-aap6.c.kubernetes-scale.internal
I0322 05:22:21.912756    1886 serve_hostnames.go:80] 19: kubernetes-minion-adgb.c.kubernetes-scale.internal
I0322 05:22:21.912760    1886 serve_hostnames.go:80] 20: kubernetes-minion-af1x.c.kubernetes-scale.internal
I0322 05:22:21.912764    1886 serve_hostnames.go:80] 21: kubernetes-minion-ay8o.c.kubernetes-scale.internal
I0322 05:22:21.912768    1886 serve_hostnames.go:80] 22: kubernetes-minion-bmy8.c.kubernetes-scale.internal
I0322 05:22:21.912772    1886 serve_hostnames.go:80] 23: kubernetes-minion-c4tz.c.kubernetes-scale.internal
I0322 05:22:21.912776    1886 serve_hostnames.go:80] 24: kubernetes-minion-d7xa.c.kubernetes-scale.internal
I0322 05:22:21.912794    1886 serve_hostnames.go:80] 25: kubernetes-minion-gvgc.c.kubernetes-scale.internal
I0322 05:22:21.912798    1886 serve_hostnames.go:80] 26: kubernetes-minion-he22.c.kubernetes-scale.internal
I0322 05:22:21.912801    1886 serve_hostnames.go:80] 27: kubernetes-minion-in2a.c.kubernetes-scale.internal
I0322 05:22:21.912805    1886 serve_hostnames.go:80] 28: kubernetes-minion-jxsn.c.kubernetes-scale.internal
I0322 05:22:21.912809    1886 serve_hostnames.go:80] 29: kubernetes-minion-kcki.c.kubernetes-scale.internal
I0322 05:22:21.912812    1886 serve_hostnames.go:80] 30: kubernetes-minion-lmfs.c.kubernetes-scale.internal
I0322 05:22:21.912816    1886 serve_hostnames.go:80] 31: kubernetes-minion-lv0e.c.kubernetes-scale.internal
I0322 05:22:21.912820    1886 serve_hostnames.go:80] 32: kubernetes-minion-mm0s.c.kubernetes-scale.internal
I0322 05:22:21.912824    1886 serve_hostnames.go:80] 33: kubernetes-minion-ng41.c.kubernetes-scale.internal
I0322 05:22:21.912834    1886 serve_hostnames.go:80] 34: kubernetes-minion-ny0n.c.kubernetes-scale.internal
I0322 05:22:21.912838    1886 serve_hostnames.go:80] 35: kubernetes-minion-om04.c.kubernetes-scale.internal
I0322 05:22:21.912842    1886 serve_hostnames.go:80] 36: kubernetes-minion-q82l.c.kubernetes-scale.internal
I0322 05:22:21.912846    1886 serve_hostnames.go:80] 37: kubernetes-minion-rfyr.c.kubernetes-scale.internal
I0322 05:22:21.912850    1886 serve_hostnames.go:80] 38: kubernetes-minion-skvs.c.kubernetes-scale.internal
I0322 05:22:21.912853    1886 serve_hostnames.go:80] 39: kubernetes-minion-svl0.c.kubernetes-scale.internal
I0322 05:22:21.912857    1886 serve_hostnames.go:80] 40: kubernetes-minion-t4qd.c.kubernetes-scale.internal
I0322 05:22:21.912861    1886 serve_hostnames.go:80] 41: kubernetes-minion-tqwt.c.kubernetes-scale.internal
I0322 05:22:21.912864    1886 serve_hostnames.go:80] 42: kubernetes-minion-ujsc.c.kubernetes-scale.internal
I0322 05:22:21.912868    1886 serve_hostnames.go:80] 43: kubernetes-minion-v086.c.kubernetes-scale.internal
I0322 05:22:21.912872    1886 serve_hostnames.go:80] 44: kubernetes-minion-vxvq.c.kubernetes-scale.internal
I0322 05:22:21.912875    1886 serve_hostnames.go:80] 45: kubernetes-minion-xe3v.c.kubernetes-scale.internal
I0322 05:22:21.912879    1886 serve_hostnames.go:80] 46: kubernetes-minion-yk7x.c.kubernetes-scale.internal
I0322 05:22:21.912883    1886 serve_hostnames.go:80] 47: kubernetes-minion-ymbd.c.kubernetes-scale.internal
I0322 05:22:21.912886    1886 serve_hostnames.go:80] 48: kubernetes-minion-zb2x.c.kubernetes-scale.internal
I0322 05:22:21.912891    1886 serve_hostnames.go:80] 49: kubernetes-minion-zmy2.c.kubernetes-scale.internal
I0322 05:22:21.912920    1886 serve_hostnames.go:86] Using namespace serve-hostnames-7439 for this test.
I0322 05:22:21.912972    1886 serve_hostnames.go:89] Creating service serve-hostnames
I0322 05:22:21.996305    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-0-0 on node kubernetes-minion-08vl.c.kubernetes-scale.internal
I0322 05:22:22.117997    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-0-1 on node kubernetes-minion-08vl.c.kubernetes-scale.internal
I0322 05:22:22.545970    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-0-2 on node kubernetes-minion-08vl.c.kubernetes-scale.internal
I0322 05:22:22.582324    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-0-3 on node kubernetes-minion-08vl.c.kubernetes-scale.internal
I0322 05:22:22.617486    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-0-4 on node kubernetes-minion-08vl.c.kubernetes-scale.internal
I0322 05:22:22.695350    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-1-0 on node kubernetes-minion-14sw.c.kubernetes-scale.internal
I0322 05:22:22.724771    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-1-1 on node kubernetes-minion-14sw.c.kubernetes-scale.internal
I0322 05:22:23.214797    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-1-2 on node kubernetes-minion-14sw.c.kubernetes-scale.internal
I0322 05:22:23.589729    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-1-3 on node kubernetes-minion-14sw.c.kubernetes-scale.internal
I0322 05:22:23.633423    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-1-4 on node kubernetes-minion-14sw.c.kubernetes-scale.internal
I0322 05:22:23.665842    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-2-0 on node kubernetes-minion-16y7.c.kubernetes-scale.internal
I0322 05:22:23.693280    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-2-1 on node kubernetes-minion-16y7.c.kubernetes-scale.internal
I0322 05:22:24.175879    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-2-2 on node kubernetes-minion-16y7.c.kubernetes-scale.internal
I0322 05:22:24.276307    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-2-3 on node kubernetes-minion-16y7.c.kubernetes-scale.internal
I0322 05:22:24.308142    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-2-4 on node kubernetes-minion-16y7.c.kubernetes-scale.internal
I0322 05:22:24.340032    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-3-0 on node kubernetes-minion-1xeu.c.kubernetes-scale.internal
I0322 05:22:24.369683    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-3-1 on node kubernetes-minion-1xeu.c.kubernetes-scale.internal
I0322 05:22:25.122989    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-3-2 on node kubernetes-minion-1xeu.c.kubernetes-scale.internal
I0322 05:22:25.156950    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-3-3 on node kubernetes-minion-1xeu.c.kubernetes-scale.internal
I0322 05:22:25.196893    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-3-4 on node kubernetes-minion-1xeu.c.kubernetes-scale.internal
I0322 05:22:25.481616    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-4-0 on node kubernetes-minion-2y1m.c.kubernetes-scale.internal
I0322 05:22:26.057240    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-4-1 on node kubernetes-minion-2y1m.c.kubernetes-scale.internal
I0322 05:22:26.616187    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-4-2 on node kubernetes-minion-2y1m.c.kubernetes-scale.internal
I0322 05:22:26.825010    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-4-3 on node kubernetes-minion-2y1m.c.kubernetes-scale.internal
I0322 05:22:27.059459    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-4-4 on node kubernetes-minion-2y1m.c.kubernetes-scale.internal
I0322 05:22:27.115107    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-5-0 on node kubernetes-minion-37uc.c.kubernetes-scale.internal
I0322 05:22:27.151505    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-5-1 on node kubernetes-minion-37uc.c.kubernetes-scale.internal
I0322 05:22:27.674820    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-5-2 on node kubernetes-minion-37uc.c.kubernetes-scale.internal
I0322 05:22:27.708916    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-5-3 on node kubernetes-minion-37uc.c.kubernetes-scale.internal
I0322 05:22:27.748777    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-5-4 on node kubernetes-minion-37uc.c.kubernetes-scale.internal
I0322 05:22:27.781448    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-6-0 on node kubernetes-minion-3gif.c.kubernetes-scale.internal
I0322 05:22:27.812185    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-6-1 on node kubernetes-minion-3gif.c.kubernetes-scale.internal
I0322 05:22:28.424761    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-6-2 on node kubernetes-minion-3gif.c.kubernetes-scale.internal
I0322 05:22:28.610764    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-6-3 on node kubernetes-minion-3gif.c.kubernetes-scale.internal
I0322 05:22:29.097744    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-6-4 on node kubernetes-minion-3gif.c.kubernetes-scale.internal
I0322 05:22:29.157710    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-7-0 on node kubernetes-minion-3p70.c.kubernetes-scale.internal
I0322 05:22:29.189646    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-7-1 on node kubernetes-minion-3p70.c.kubernetes-scale.internal
I0322 05:22:29.619296    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-7-2 on node kubernetes-minion-3p70.c.kubernetes-scale.internal
I0322 05:22:29.732808    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-7-3 on node kubernetes-minion-3p70.c.kubernetes-scale.internal
I0322 05:22:29.777044    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-7-4 on node kubernetes-minion-3p70.c.kubernetes-scale.internal
I0322 05:22:29.829220    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-8-0 on node kubernetes-minion-3sbf.c.kubernetes-scale.internal
I0322 05:22:29.859899    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-8-1 on node kubernetes-minion-3sbf.c.kubernetes-scale.internal
I0322 05:22:30.631038    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-8-2 on node kubernetes-minion-3sbf.c.kubernetes-scale.internal
I0322 05:22:30.664945    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-8-3 on node kubernetes-minion-3sbf.c.kubernetes-scale.internal
I0322 05:22:30.766531    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-8-4 on node kubernetes-minion-3sbf.c.kubernetes-scale.internal
I0322 05:22:30.801802    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-9-0 on node kubernetes-minion-4aej.c.kubernetes-scale.internal
I0322 05:22:30.838323    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-9-1 on node kubernetes-minion-4aej.c.kubernetes-scale.internal
I0322 05:22:31.306949    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-9-2 on node kubernetes-minion-4aej.c.kubernetes-scale.internal
I0322 05:22:31.363579    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-9-3 on node kubernetes-minion-4aej.c.kubernetes-scale.internal
I0322 05:22:31.468343    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-9-4 on node kubernetes-minion-4aej.c.kubernetes-scale.internal
I0322 05:22:31.517287    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-10-0 on node kubernetes-minion-4l4h.c.kubernetes-scale.internal
I0322 05:22:31.547323    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-10-1 on node kubernetes-minion-4l4h.c.kubernetes-scale.internal
I0322 05:22:31.969761    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-10-2 on node kubernetes-minion-4l4h.c.kubernetes-scale.internal
I0322 05:22:32.406346    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-10-3 on node kubernetes-minion-4l4h.c.kubernetes-scale.internal
I0322 05:22:32.777266    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-10-4 on node kubernetes-minion-4l4h.c.kubernetes-scale.internal
I0322 05:22:32.949970    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-11-0 on node kubernetes-minion-59d2.c.kubernetes-scale.internal
I0322 05:22:32.990210    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-11-1 on node kubernetes-minion-59d2.c.kubernetes-scale.internal
I0322 05:22:33.217663    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-11-2 on node kubernetes-minion-59d2.c.kubernetes-scale.internal
I0322 05:22:33.261745    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-11-3 on node kubernetes-minion-59d2.c.kubernetes-scale.internal
I0322 05:22:33.303186    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-11-4 on node kubernetes-minion-59d2.c.kubernetes-scale.internal
I0322 05:22:33.354908    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-12-0 on node kubernetes-minion-6q3q.c.kubernetes-scale.internal
I0322 05:22:33.428808    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-12-1 on node kubernetes-minion-6q3q.c.kubernetes-scale.internal
I0322 05:22:33.478255    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-12-2 on node kubernetes-minion-6q3q.c.kubernetes-scale.internal
I0322 05:22:33.940551    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-12-3 on node kubernetes-minion-6q3q.c.kubernetes-scale.internal
I0322 05:22:33.973761    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-12-4 on node kubernetes-minion-6q3q.c.kubernetes-scale.internal
I0322 05:22:34.006591    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-13-0 on node kubernetes-minion-7rlr.c.kubernetes-scale.internal
I0322 05:22:34.062838    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-13-1 on node kubernetes-minion-7rlr.c.kubernetes-scale.internal
I0322 05:22:36.064766    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-13-2 on node kubernetes-minion-7rlr.c.kubernetes-scale.internal
I0322 05:22:36.416096    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-13-3 on node kubernetes-minion-7rlr.c.kubernetes-scale.internal
I0322 05:22:36.897155    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-13-4 on node kubernetes-minion-7rlr.c.kubernetes-scale.internal
I0322 05:22:36.998734    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-14-0 on node kubernetes-minion-87a6.c.kubernetes-scale.internal
I0322 05:22:38.250576    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-14-1 on node kubernetes-minion-87a6.c.kubernetes-scale.internal
I0322 05:22:38.294923    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-14-2 on node kubernetes-minion-87a6.c.kubernetes-scale.internal
I0322 05:22:38.343140    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-14-3 on node kubernetes-minion-87a6.c.kubernetes-scale.internal
I0322 05:22:38.380256    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-14-4 on node kubernetes-minion-87a6.c.kubernetes-scale.internal
I0322 05:22:38.415173    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-15-0 on node kubernetes-minion-8kpq.c.kubernetes-scale.internal
I0322 05:22:38.517129    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-15-1 on node kubernetes-minion-8kpq.c.kubernetes-scale.internal
I0322 05:22:38.691481    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-15-2 on node kubernetes-minion-8kpq.c.kubernetes-scale.internal
I0322 05:22:39.653454    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-15-3 on node kubernetes-minion-8kpq.c.kubernetes-scale.internal
I0322 05:22:39.811871    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-15-4 on node kubernetes-minion-8kpq.c.kubernetes-scale.internal
I0322 05:22:40.029403    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-16-0 on node kubernetes-minion-95vc.c.kubernetes-scale.internal
I0322 05:22:40.150463    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-16-1 on node kubernetes-minion-95vc.c.kubernetes-scale.internal
I0322 05:22:40.287944    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-16-2 on node kubernetes-minion-95vc.c.kubernetes-scale.internal
I0322 05:22:41.183437    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-16-3 on node kubernetes-minion-95vc.c.kubernetes-scale.internal
I0322 05:22:41.939556    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-16-4 on node kubernetes-minion-95vc.c.kubernetes-scale.internal
I0322 05:22:42.127595    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-17-0 on node kubernetes-minion-a5zj.c.kubernetes-scale.internal
I0322 05:22:42.247769    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-17-1 on node kubernetes-minion-a5zj.c.kubernetes-scale.internal
I0322 05:22:42.518890    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-17-2 on node kubernetes-minion-a5zj.c.kubernetes-scale.internal
I0322 05:22:42.596369    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-17-3 on node kubernetes-minion-a5zj.c.kubernetes-scale.internal
I0322 05:22:43.386358    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-17-4 on node kubernetes-minion-a5zj.c.kubernetes-scale.internal
I0322 05:22:43.639490    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-18-0 on node kubernetes-minion-aap6.c.kubernetes-scale.internal
I0322 05:22:43.696959    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-18-1 on node kubernetes-minion-aap6.c.kubernetes-scale.internal
I0322 05:22:44.333237    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-18-2 on node kubernetes-minion-aap6.c.kubernetes-scale.internal
I0322 05:22:44.380173    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-18-3 on node kubernetes-minion-aap6.c.kubernetes-scale.internal
I0322 05:22:44.423442    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-18-4 on node kubernetes-minion-aap6.c.kubernetes-scale.internal
I0322 05:22:44.471621    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-19-0 on node kubernetes-minion-adgb.c.kubernetes-scale.internal
I0322 05:22:44.685509    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-19-1 on node kubernetes-minion-adgb.c.kubernetes-scale.internal
I0322 05:22:46.203908    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-19-2 on node kubernetes-minion-adgb.c.kubernetes-scale.internal
I0322 05:22:46.514693    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-19-3 on node kubernetes-minion-adgb.c.kubernetes-scale.internal
I0322 05:22:46.644134    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-19-4 on node kubernetes-minion-adgb.c.kubernetes-scale.internal
I0322 05:22:46.689195    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-20-0 on node kubernetes-minion-af1x.c.kubernetes-scale.internal
I0322 05:22:46.724809    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-20-1 on node kubernetes-minion-af1x.c.kubernetes-scale.internal
I0322 05:22:47.574324    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-20-2 on node kubernetes-minion-af1x.c.kubernetes-scale.internal
I0322 05:22:48.215347    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-20-3 on node kubernetes-minion-af1x.c.kubernetes-scale.internal
I0322 05:22:48.330718    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-20-4 on node kubernetes-minion-af1x.c.kubernetes-scale.internal
I0322 05:22:48.385289    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-21-0 on node kubernetes-minion-ay8o.c.kubernetes-scale.internal
I0322 05:22:48.490541    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-21-1 on node kubernetes-minion-ay8o.c.kubernetes-scale.internal
I0322 05:22:49.457217    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-21-2 on node kubernetes-minion-ay8o.c.kubernetes-scale.internal
I0322 05:22:50.035142    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-21-3 on node kubernetes-minion-ay8o.c.kubernetes-scale.internal
I0322 05:22:50.258755    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-21-4 on node kubernetes-minion-ay8o.c.kubernetes-scale.internal
I0322 05:22:50.349118    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-22-0 on node kubernetes-minion-bmy8.c.kubernetes-scale.internal
I0322 05:22:50.393374    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-22-1 on node kubernetes-minion-bmy8.c.kubernetes-scale.internal
I0322 05:22:51.180895    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-22-2 on node kubernetes-minion-bmy8.c.kubernetes-scale.internal
I0322 05:22:51.243888    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-22-3 on node kubernetes-minion-bmy8.c.kubernetes-scale.internal
I0322 05:22:51.336810    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-22-4 on node kubernetes-minion-bmy8.c.kubernetes-scale.internal
I0322 05:22:52.292412    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-23-0 on node kubernetes-minion-c4tz.c.kubernetes-scale.internal
I0322 05:22:52.441863    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-23-1 on node kubernetes-minion-c4tz.c.kubernetes-scale.internal
I0322 05:22:52.822973    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-23-2 on node kubernetes-minion-c4tz.c.kubernetes-scale.internal
I0322 05:22:52.875391    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-23-3 on node kubernetes-minion-c4tz.c.kubernetes-scale.internal
I0322 05:22:53.330336    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-23-4 on node kubernetes-minion-c4tz.c.kubernetes-scale.internal
I0322 05:22:53.450916    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-24-0 on node kubernetes-minion-d7xa.c.kubernetes-scale.internal
I0322 05:22:53.501680    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-24-1 on node kubernetes-minion-d7xa.c.kubernetes-scale.internal
I0322 05:22:54.537498    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-24-2 on node kubernetes-minion-d7xa.c.kubernetes-scale.internal
I0322 05:22:54.693946    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-24-3 on node kubernetes-minion-d7xa.c.kubernetes-scale.internal
I0322 05:22:55.547279    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-24-4 on node kubernetes-minion-d7xa.c.kubernetes-scale.internal
I0322 05:22:56.449017    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-25-0 on node kubernetes-minion-gvgc.c.kubernetes-scale.internal
I0322 05:22:56.515764    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-25-1 on node kubernetes-minion-gvgc.c.kubernetes-scale.internal
I0322 05:22:56.913610    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-25-2 on node kubernetes-minion-gvgc.c.kubernetes-scale.internal
I0322 05:22:56.985363    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-25-3 on node kubernetes-minion-gvgc.c.kubernetes-scale.internal
I0322 05:22:57.993207    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-25-4 on node kubernetes-minion-gvgc.c.kubernetes-scale.internal
I0322 05:22:58.233818    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-26-0 on node kubernetes-minion-he22.c.kubernetes-scale.internal
I0322 05:22:58.357318    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-26-1 on node kubernetes-minion-he22.c.kubernetes-scale.internal
I0322 05:22:58.613130    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-26-2 on node kubernetes-minion-he22.c.kubernetes-scale.internal
I0322 05:22:58.667134    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-26-3 on node kubernetes-minion-he22.c.kubernetes-scale.internal
I0322 05:22:58.785130    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-26-4 on node kubernetes-minion-he22.c.kubernetes-scale.internal
I0322 05:22:58.849330    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-27-0 on node kubernetes-minion-in2a.c.kubernetes-scale.internal
I0322 05:22:58.896885    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-27-1 on node kubernetes-minion-in2a.c.kubernetes-scale.internal
I0322 05:22:59.100458    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-27-2 on node kubernetes-minion-in2a.c.kubernetes-scale.internal
I0322 05:22:59.960103    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-27-3 on node kubernetes-minion-in2a.c.kubernetes-scale.internal
I0322 05:23:01.365544    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-27-4 on node kubernetes-minion-in2a.c.kubernetes-scale.internal
I0322 05:23:02.522763    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-28-0 on node kubernetes-minion-jxsn.c.kubernetes-scale.internal
I0322 05:23:03.292278    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-28-1 on node kubernetes-minion-jxsn.c.kubernetes-scale.internal
I0322 05:23:03.377111    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-28-2 on node kubernetes-minion-jxsn.c.kubernetes-scale.internal
I0322 05:23:03.519283    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-28-3 on node kubernetes-minion-jxsn.c.kubernetes-scale.internal
I0322 05:23:03.589100    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-28-4 on node kubernetes-minion-jxsn.c.kubernetes-scale.internal
I0322 05:23:03.661003    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-29-0 on node kubernetes-minion-kcki.c.kubernetes-scale.internal
I0322 05:23:04.954675    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-29-1 on node kubernetes-minion-kcki.c.kubernetes-scale.internal
I0322 05:23:05.011557    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-29-2 on node kubernetes-minion-kcki.c.kubernetes-scale.internal
I0322 05:23:05.085917    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-29-3 on node kubernetes-minion-kcki.c.kubernetes-scale.internal
I0322 05:23:05.264257    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-29-4 on node kubernetes-minion-kcki.c.kubernetes-scale.internal
I0322 05:23:05.515528    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-30-0 on node kubernetes-minion-lmfs.c.kubernetes-scale.internal
I0322 05:23:06.160838    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-30-1 on node kubernetes-minion-lmfs.c.kubernetes-scale.internal
I0322 05:23:06.972605    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-30-2 on node kubernetes-minion-lmfs.c.kubernetes-scale.internal
I0322 05:23:08.261051    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-30-3 on node kubernetes-minion-lmfs.c.kubernetes-scale.internal
I0322 05:23:08.642217    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-30-4 on node kubernetes-minion-lmfs.c.kubernetes-scale.internal
I0322 05:23:08.729053    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-31-0 on node kubernetes-minion-lv0e.c.kubernetes-scale.internal
I0322 05:23:08.749364    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-31-1 on node kubernetes-minion-lv0e.c.kubernetes-scale.internal
I0322 05:23:10.070614    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-31-2 on node kubernetes-minion-lv0e.c.kubernetes-scale.internal
I0322 05:23:10.186772    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-31-3 on node kubernetes-minion-lv0e.c.kubernetes-scale.internal
I0322 05:23:10.283464    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-31-4 on node kubernetes-minion-lv0e.c.kubernetes-scale.internal
I0322 05:23:10.332581    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-32-0 on node kubernetes-minion-mm0s.c.kubernetes-scale.internal
I0322 05:23:10.356737    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-32-1 on node kubernetes-minion-mm0s.c.kubernetes-scale.internal
I0322 05:23:10.771131    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-32-2 on node kubernetes-minion-mm0s.c.kubernetes-scale.internal
I0322 05:23:10.848039    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-32-3 on node kubernetes-minion-mm0s.c.kubernetes-scale.internal
I0322 05:23:10.953242    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-32-4 on node kubernetes-minion-mm0s.c.kubernetes-scale.internal
I0322 05:23:10.999823    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-33-0 on node kubernetes-minion-ng41.c.kubernetes-scale.internal
I0322 05:23:12.138257    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-33-1 on node kubernetes-minion-ng41.c.kubernetes-scale.internal
I0322 05:23:12.225604    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-33-2 on node kubernetes-minion-ng41.c.kubernetes-scale.internal
I0322 05:23:12.247832    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-33-3 on node kubernetes-minion-ng41.c.kubernetes-scale.internal
I0322 05:23:12.279203    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-33-4 on node kubernetes-minion-ng41.c.kubernetes-scale.internal
I0322 05:23:12.315322    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-34-0 on node kubernetes-minion-ny0n.c.kubernetes-scale.internal
I0322 05:23:12.331898    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-34-1 on node kubernetes-minion-ny0n.c.kubernetes-scale.internal
I0322 05:23:12.785829    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-34-2 on node kubernetes-minion-ny0n.c.kubernetes-scale.internal
I0322 05:23:12.812949    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-34-3 on node kubernetes-minion-ny0n.c.kubernetes-scale.internal
I0322 05:23:12.833245    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-34-4 on node kubernetes-minion-ny0n.c.kubernetes-scale.internal
I0322 05:23:12.853259    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-35-0 on node kubernetes-minion-om04.c.kubernetes-scale.internal
I0322 05:23:12.870637    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-35-1 on node kubernetes-minion-om04.c.kubernetes-scale.internal
I0322 05:23:13.627546    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-35-2 on node kubernetes-minion-om04.c.kubernetes-scale.internal
I0322 05:23:14.435362    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-35-3 on node kubernetes-minion-om04.c.kubernetes-scale.internal
I0322 05:23:14.634512    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-35-4 on node kubernetes-minion-om04.c.kubernetes-scale.internal
I0322 05:23:14.685235    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-36-0 on node kubernetes-minion-q82l.c.kubernetes-scale.internal
I0322 05:23:14.706083    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-36-1 on node kubernetes-minion-q82l.c.kubernetes-scale.internal
I0322 05:23:15.230734    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-36-2 on node kubernetes-minion-q82l.c.kubernetes-scale.internal
I0322 05:23:15.250109    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-36-3 on node kubernetes-minion-q82l.c.kubernetes-scale.internal
I0322 05:23:15.286150    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-36-4 on node kubernetes-minion-q82l.c.kubernetes-scale.internal
I0322 05:23:15.763658    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-37-0 on node kubernetes-minion-rfyr.c.kubernetes-scale.internal
I0322 05:23:16.057726    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-37-1 on node kubernetes-minion-rfyr.c.kubernetes-scale.internal
I0322 05:23:16.581140    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-37-2 on node kubernetes-minion-rfyr.c.kubernetes-scale.internal
I0322 05:23:16.601507    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-37-3 on node kubernetes-minion-rfyr.c.kubernetes-scale.internal
I0322 05:23:16.714759    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-37-4 on node kubernetes-minion-rfyr.c.kubernetes-scale.internal
I0322 05:23:16.764386    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-38-0 on node kubernetes-minion-skvs.c.kubernetes-scale.internal
I0322 05:23:16.780750    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-38-1 on node kubernetes-minion-skvs.c.kubernetes-scale.internal
I0322 05:23:16.796477    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-38-2 on node kubernetes-minion-skvs.c.kubernetes-scale.internal
I0322 05:23:17.441746    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-38-3 on node kubernetes-minion-skvs.c.kubernetes-scale.internal
I0322 05:23:17.463000    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-38-4 on node kubernetes-minion-skvs.c.kubernetes-scale.internal
I0322 05:23:17.483388    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-39-0 on node kubernetes-minion-svl0.c.kubernetes-scale.internal
I0322 05:23:17.504558    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-39-1 on node kubernetes-minion-svl0.c.kubernetes-scale.internal
I0322 05:23:18.003219    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-39-2 on node kubernetes-minion-svl0.c.kubernetes-scale.internal
I0322 05:23:18.023594    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-39-3 on node kubernetes-minion-svl0.c.kubernetes-scale.internal
I0322 05:23:18.054086    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-39-4 on node kubernetes-minion-svl0.c.kubernetes-scale.internal
I0322 05:23:18.073404    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-40-0 on node kubernetes-minion-t4qd.c.kubernetes-scale.internal
I0322 05:23:18.089289    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-40-1 on node kubernetes-minion-t4qd.c.kubernetes-scale.internal
I0322 05:23:18.104255    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-40-2 on node kubernetes-minion-t4qd.c.kubernetes-scale.internal
I0322 05:23:18.595583    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-40-3 on node kubernetes-minion-t4qd.c.kubernetes-scale.internal
I0322 05:23:18.615560    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-40-4 on node kubernetes-minion-t4qd.c.kubernetes-scale.internal
I0322 05:23:18.634497    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-41-0 on node kubernetes-minion-tqwt.c.kubernetes-scale.internal
I0322 05:23:18.658746    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-41-1 on node kubernetes-minion-tqwt.c.kubernetes-scale.internal
I0322 05:23:19.040595    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-41-2 on node kubernetes-minion-tqwt.c.kubernetes-scale.internal
I0322 05:23:19.060866    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-41-3 on node kubernetes-minion-tqwt.c.kubernetes-scale.internal
I0322 05:23:19.081838    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-41-4 on node kubernetes-minion-tqwt.c.kubernetes-scale.internal
I0322 05:23:19.102109    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-42-0 on node kubernetes-minion-ujsc.c.kubernetes-scale.internal
I0322 05:23:19.118436    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-42-1 on node kubernetes-minion-ujsc.c.kubernetes-scale.internal
I0322 05:23:19.134808    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-42-2 on node kubernetes-minion-ujsc.c.kubernetes-scale.internal
I0322 05:23:19.605758    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-42-3 on node kubernetes-minion-ujsc.c.kubernetes-scale.internal
I0322 05:23:19.813634    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-42-4 on node kubernetes-minion-ujsc.c.kubernetes-scale.internal
I0322 05:23:20.906963    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-43-0 on node kubernetes-minion-v086.c.kubernetes-scale.internal
I0322 05:23:21.233551    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-43-1 on node kubernetes-minion-v086.c.kubernetes-scale.internal
I0322 05:23:21.869792    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-43-2 on node kubernetes-minion-v086.c.kubernetes-scale.internal
I0322 05:23:22.094022    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-43-3 on node kubernetes-minion-v086.c.kubernetes-scale.internal
I0322 05:23:22.554471    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-43-4 on node kubernetes-minion-v086.c.kubernetes-scale.internal
I0322 05:23:22.806294    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-44-0 on node kubernetes-minion-vxvq.c.kubernetes-scale.internal
I0322 05:23:22.992304    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-44-1 on node kubernetes-minion-vxvq.c.kubernetes-scale.internal
I0322 05:23:23.499607    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-44-2 on node kubernetes-minion-vxvq.c.kubernetes-scale.internal
I0322 05:23:23.522929    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-44-3 on node kubernetes-minion-vxvq.c.kubernetes-scale.internal
I0322 05:23:23.548255    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-44-4 on node kubernetes-minion-vxvq.c.kubernetes-scale.internal
I0322 05:23:23.575344    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-45-0 on node kubernetes-minion-xe3v.c.kubernetes-scale.internal
I0322 05:23:23.596418    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-45-1 on node kubernetes-minion-xe3v.c.kubernetes-scale.internal
I0322 05:23:23.922830    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-45-2 on node kubernetes-minion-xe3v.c.kubernetes-scale.internal
I0322 05:23:23.947755    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-45-3 on node kubernetes-minion-xe3v.c.kubernetes-scale.internal
I0322 05:23:24.084489    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-45-4 on node kubernetes-minion-xe3v.c.kubernetes-scale.internal
I0322 05:23:24.110400    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-46-0 on node kubernetes-minion-yk7x.c.kubernetes-scale.internal
I0322 05:23:24.134706    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-46-1 on node kubernetes-minion-yk7x.c.kubernetes-scale.internal
I0322 05:23:24.599161    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-46-2 on node kubernetes-minion-yk7x.c.kubernetes-scale.internal
I0322 05:23:24.624189    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-46-3 on node kubernetes-minion-yk7x.c.kubernetes-scale.internal
I0322 05:23:24.645166    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-46-4 on node kubernetes-minion-yk7x.c.kubernetes-scale.internal
I0322 05:23:24.671056    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-47-0 on node kubernetes-minion-ymbd.c.kubernetes-scale.internal
I0322 05:23:24.691168    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-47-1 on node kubernetes-minion-ymbd.c.kubernetes-scale.internal
I0322 05:23:26.032335    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-47-2 on node kubernetes-minion-ymbd.c.kubernetes-scale.internal
I0322 05:23:27.533269    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-47-3 on node kubernetes-minion-ymbd.c.kubernetes-scale.internal
I0322 05:23:28.802208    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-47-4 on node kubernetes-minion-ymbd.c.kubernetes-scale.internal
I0322 05:23:29.073507    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-48-0 on node kubernetes-minion-zb2x.c.kubernetes-scale.internal
I0322 05:23:29.100987    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-48-1 on node kubernetes-minion-zb2x.c.kubernetes-scale.internal
I0322 05:23:29.717936    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-48-2 on node kubernetes-minion-zb2x.c.kubernetes-scale.internal
I0322 05:23:29.742580    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-48-3 on node kubernetes-minion-zb2x.c.kubernetes-scale.internal
I0322 05:23:29.764884    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-48-4 on node kubernetes-minion-zb2x.c.kubernetes-scale.internal
I0322 05:23:29.788289    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-49-0 on node kubernetes-minion-zmy2.c.kubernetes-scale.internal
I0322 05:23:29.807325    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-49-1 on node kubernetes-minion-zmy2.c.kubernetes-scale.internal
I0322 05:23:29.828921    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-49-2 on node kubernetes-minion-zmy2.c.kubernetes-scale.internal
I0322 05:23:30.149308    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-49-3 on node kubernetes-minion-zmy2.c.kubernetes-scale.internal
I0322 05:23:30.174374    1886 serve_hostnames.go:123] Creating pod serve-hostnames-7439/serve-hostname-49-4 on node kubernetes-minion-zmy2.c.kubernetes-scale.internal
I0322 05:23:30.198543    1886 serve_hostnames.go:158] Waiting for the serve-hostname pods to be ready
I0322 05:23:30.204981    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-0-0 is running
I0322 05:23:30.211353    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-0-1 is running
I0322 05:23:30.217960    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-0-2 is running
I0322 05:23:30.224566    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-0-3 is running
I0322 05:23:30.230489    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-0-4 is running
I0322 05:23:30.236681    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-1-0 is running
I0322 05:23:30.244146    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-1-1 is running
I0322 05:23:30.249179    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-1-2 is running
I0322 05:23:30.255217    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-1-3 is running
I0322 05:23:30.261380    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-1-4 is running
I0322 05:24:14.592412    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-2-0 is running
I0322 05:24:14.598917    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-2-1 is running
I0322 05:24:14.605601    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-2-2 is running
I0322 05:24:14.730251    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-2-3 is running
I0322 05:24:14.809217    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-2-4 is running
I0322 05:24:26.852350    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-3-0 is running
I0322 05:24:26.861153    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-3-1 is running
I0322 05:24:26.870947    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-3-2 is running
I0322 05:24:26.879465    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-3-3 is running
I0322 05:24:26.889280    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-3-4 is running
I0322 05:24:26.902217    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-4-0 is running
I0322 05:24:26.911720    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-4-1 is running
I0322 05:24:26.922184    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-4-2 is running
I0322 05:24:26.931839    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-4-3 is running
I0322 05:24:26.941384    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-4-4 is running
I0322 05:24:26.948369    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-5-0 is running
I0322 05:24:26.954724    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-5-1 is running
I0322 05:24:26.982014    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-5-2 is running
I0322 05:24:26.989021    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-5-3 is running
I0322 05:24:26.995736    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-5-4 is running
I0322 05:24:27.002071    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-6-0 is running
I0322 05:24:27.007262    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-6-1 is running
I0322 05:24:27.014383    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-6-2 is running
I0322 05:24:27.021261    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-6-3 is running
I0322 05:24:27.028100    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-6-4 is running
I0322 05:24:27.035031    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-7-0 is running
I0322 05:24:27.041695    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-7-1 is running
I0322 05:24:27.048171    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-7-2 is running
I0322 05:24:27.064478    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-7-3 is running
I0322 05:24:27.071044    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-7-4 is running
I0322 05:24:27.078141    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-8-0 is running
I0322 05:24:27.085299    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-8-1 is running
I0322 05:24:27.091110    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-8-2 is running
I0322 05:24:27.097797    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-8-3 is running
I0322 05:24:27.104263    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-8-4 is running
I0322 05:24:27.110191    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-9-0 is running
I0322 05:24:27.117008    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-9-1 is running
I0322 05:24:27.123832    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-9-2 is running
I0322 05:24:27.129643    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-9-3 is running
I0322 05:24:27.135946    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-9-4 is running
I0322 05:26:00.806251    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-10-0 is running
I0322 05:26:01.476447    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-10-1 is running
I0322 05:26:01.808893    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-10-2 is running
I0322 05:26:02.095323    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-10-3 is running
I0322 05:26:02.370236    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-10-4 is running
I0322 05:26:02.519892    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-11-0 is running
I0322 05:26:02.633418    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-11-1 is running
I0322 05:26:02.760401    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-11-2 is running
I0322 05:26:02.861615    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-11-3 is running
I0322 05:26:03.001410    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-11-4 is running
I0322 05:26:03.081528    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-12-0 is running
I0322 05:26:03.168789    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-12-1 is running
I0322 05:26:03.287628    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-12-2 is running
I0322 05:26:03.844625    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-12-3 is running
I0322 05:26:04.039909    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-12-4 is running
I0322 05:26:04.340961    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-13-0 is running
I0322 05:26:04.473156    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-13-1 is running
I0322 05:26:04.581629    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-13-2 is running
I0322 05:26:04.627103    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-13-3 is running
I0322 05:26:04.653634    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-13-4 is running
I0322 05:26:04.683583    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-14-0 is running
I0322 05:26:04.722204    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-14-1 is running
I0322 05:26:04.742324    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-14-2 is running
I0322 05:26:04.764654    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-14-3 is running
I0322 05:26:04.780809    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-14-4 is running
I0322 05:26:04.795943    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-15-0 is running
I0322 05:26:04.810281    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-15-1 is running
I0322 05:26:04.843990    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-15-2 is running
I0322 05:26:04.872066    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-15-3 is running
I0322 05:26:04.952489    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-15-4 is running
I0322 05:26:04.973052    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-16-0 is running
I0322 05:26:05.000455    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-16-1 is running
I0322 05:26:05.015853    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-16-2 is running
I0322 05:26:05.029361    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-16-3 is running
I0322 05:26:05.049779    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-16-4 is running
I0322 05:26:05.063322    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-17-0 is running
I0322 05:26:05.077648    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-17-1 is running
I0322 05:26:05.097129    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-17-2 is running
I0322 05:26:05.116230    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-17-3 is running
I0322 05:26:05.129339    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-17-4 is running
I0322 05:26:05.137652    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-18-0 is running
I0322 05:26:05.143680    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-18-1 is running
I0322 05:26:05.151605    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-18-2 is running
I0322 05:26:05.160457    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-18-3 is running
I0322 05:26:05.168875    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-18-4 is running
I0322 05:26:05.177315    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-19-0 is running
I0322 05:26:05.185399    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-19-1 is running
I0322 05:26:05.192080    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-19-2 is running
I0322 05:26:05.197893    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-19-3 is running
I0322 05:26:05.204562    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-19-4 is running
I0322 05:26:05.210865    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-20-0 is running
I0322 05:26:05.220278    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-20-1 is running
I0322 05:26:05.231762    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-20-2 is running
I0322 05:26:05.237643    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-20-3 is running
I0322 05:26:05.246930    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-20-4 is running
I0322 05:26:05.255298    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-21-0 is running
I0322 05:26:05.261026    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-21-1 is running
I0322 05:26:05.273186    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-21-2 is running
I0322 05:26:05.370418    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-21-3 is running
I0322 05:26:05.408833    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-21-4 is running
I0322 05:26:05.429486    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-22-0 is running
I0322 05:26:05.448865    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-22-1 is running
I0322 05:26:05.454720    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-22-2 is running
I0322 05:26:05.460603    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-22-3 is running
I0322 05:26:05.466928    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-22-4 is running
I0322 05:26:05.473505    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-23-0 is running
I0322 05:26:05.479564    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-23-1 is running
I0322 05:26:05.485590    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-23-2 is running
I0322 05:26:05.491485    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-23-3 is running
I0322 05:26:05.571741    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-23-4 is running
I0322 05:26:06.063423    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-24-0 is running
I0322 05:26:06.264424    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-24-1 is running
I0322 05:26:06.280575    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-24-2 is running
I0322 05:26:06.286665    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-24-3 is running
I0322 05:26:06.292310    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-24-4 is running
I0322 05:26:06.298724    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-25-0 is running
I0322 05:26:06.305198    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-25-1 is running
I0322 05:26:06.310910    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-25-2 is running
I0322 05:26:06.319795    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-25-3 is running
I0322 05:26:06.326108    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-25-4 is running
I0322 05:26:06.331814    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-26-0 is running
I0322 05:26:06.337583    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-26-1 is running
I0322 05:26:06.343668    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-26-2 is running
I0322 05:26:06.350151    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-26-3 is running
I0322 05:26:06.356232    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-26-4 is running
I0322 05:26:06.362388    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-27-0 is running
I0322 05:26:06.368571    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-27-1 is running
I0322 05:26:06.375442    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-27-2 is running
I0322 05:26:06.380877    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-27-3 is running
I0322 05:26:06.387163    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-27-4 is running
I0322 05:26:06.393336    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-28-0 is running
I0322 05:26:06.399293    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-28-1 is running
I0322 05:26:06.405323    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-28-2 is running
I0322 05:26:06.411080    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-28-3 is running
I0322 05:26:06.417348    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-28-4 is running
I0322 05:26:06.424108    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-29-0 is running
I0322 05:26:06.430007    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-29-1 is running
I0322 05:26:06.437464    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-29-2 is running
I0322 05:26:06.443634    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-29-3 is running
I0322 05:26:06.450772    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-29-4 is running
I0322 05:26:06.457712    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-30-0 is running
I0322 05:28:08.773645    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-30-1 is running
I0322 05:28:11.549408    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-30-2 is running
I0322 05:28:12.293811    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-30-3 is running
I0322 05:28:12.850401    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-30-4 is running
I0322 05:28:13.190399    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-31-0 is running
I0322 05:28:13.410294    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-31-1 is running
I0322 05:28:13.605235    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-31-2 is running
I0322 05:28:13.882662    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-31-3 is running
I0322 05:28:14.109717    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-31-4 is running
I0322 05:28:14.333386    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-32-0 is running
I0322 05:28:14.572399    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-32-1 is running
I0322 05:28:14.779834    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-32-2 is running
I0322 05:28:15.665221    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-32-3 is running
I0322 05:28:16.302788    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-32-4 is running
I0322 05:28:17.029830    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-33-0 is running
I0322 05:28:17.598826    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-33-1 is running
I0322 05:28:17.706727    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-33-2 is running
I0322 05:28:17.792178    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-33-3 is running
I0322 05:28:17.870510    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-33-4 is running
I0322 05:28:17.921481    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-34-0 is running
I0322 05:28:17.973809    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-34-1 is running
I0322 05:28:18.034131    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-34-2 is running
I0322 05:28:18.053514    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-34-3 is running
I0322 05:28:18.077664    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-34-4 is running
I0322 05:28:18.094010    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-35-0 is running
I0322 05:28:18.114837    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-35-1 is running
I0322 05:28:18.143633    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-35-2 is running
I0322 05:28:18.179197    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-35-3 is running
I0322 05:28:18.190705    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-35-4 is running
I0322 05:28:18.197097    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-36-0 is running
I0322 05:28:18.203000    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-36-1 is running
I0322 05:28:18.209741    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-36-2 is running
I0322 05:28:18.215699    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-36-3 is running
I0322 05:28:18.221643    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-36-4 is running
I0322 05:28:18.228139    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-37-0 is running
I0322 05:28:18.233908    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-37-1 is running
I0322 05:28:18.240513    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-37-2 is running
I0322 05:28:18.287481    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-37-3 is running
I0322 05:28:18.319106    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-37-4 is running
I0322 05:28:18.325116    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-38-0 is running
I0322 05:28:18.331924    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-38-1 is running
I0322 05:28:18.337728    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-38-2 is running
I0322 05:28:18.344256    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-38-3 is running
I0322 05:28:18.351182    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-38-4 is running
I0322 05:28:18.357146    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-39-0 is running
I0322 05:28:18.362495    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-39-1 is running
I0322 05:28:18.368258    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-39-2 is running
I0322 05:28:18.374966    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-39-3 is running
I0322 05:28:18.381215    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-39-4 is running
I0322 05:28:18.387863    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-40-0 is running
I0322 05:28:18.429666    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-40-1 is running
I0322 05:28:18.435298    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-40-2 is running
I0322 05:28:18.464910    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-40-3 is running
I0322 05:28:18.471802    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-40-4 is running
I0322 05:28:18.477823    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-41-0 is running
I0322 05:28:18.484531    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-41-1 is running
I0322 05:28:18.490574    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-41-2 is running
I0322 05:28:18.497016    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-41-3 is running
I0322 05:28:18.502864    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-41-4 is running
I0322 05:28:18.509423    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-42-0 is running
I0322 05:28:18.521025    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-42-1 is running
I0322 05:28:18.547922    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-42-2 is running
I0322 05:28:18.554423    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-42-3 is running
I0322 05:28:18.560049    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-42-4 is running
I0322 05:28:18.566280    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-43-0 is running
I0322 05:28:18.572734    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-43-1 is running
I0322 05:28:18.577583    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-43-2 is running
I0322 05:28:18.583649    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-43-3 is running
I0322 05:28:18.589962    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-43-4 is running
I0322 05:28:18.596070    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-44-0 is running
I0322 05:28:18.602344    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-44-1 is running
I0322 05:28:18.608679    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-44-2 is running
I0322 05:28:18.615241    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-44-3 is running
I0322 05:28:18.621667    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-44-4 is running
I0322 05:28:18.627877    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-45-0 is running
I0322 05:28:18.634042    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-45-1 is running
I0322 05:28:18.639880    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-45-2 is running
I0322 05:28:18.646876    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-45-3 is running
I0322 05:28:18.653790    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-45-4 is running
I0322 05:28:18.659124    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-46-0 is running
I0322 05:28:18.665042    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-46-1 is running
I0322 05:28:18.671449    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-46-2 is running
I0322 05:28:18.677618    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-46-3 is running
I0322 05:28:18.726345    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-46-4 is running
I0322 05:28:18.756259    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-47-0 is running
I0322 05:28:18.797083    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-47-1 is running
I0322 05:28:18.842308    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-47-2 is running
I0322 05:28:18.849160    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-47-3 is running
I0322 05:28:18.855153    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-47-4 is running
I0322 05:28:18.860809    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-48-0 is running
I0322 05:28:18.867405    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-48-1 is running
I0322 05:28:18.874161    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-48-2 is running
I0322 05:28:18.880089    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-48-3 is running
I0322 05:28:18.886300    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-48-4 is running
I0322 05:28:18.892621    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-49-0 is running
I0322 05:28:18.897995    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-49-1 is running
I0322 05:28:18.903932    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-49-2 is running
I0322 05:28:18.910522    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-49-3 is running
I0322 05:28:18.917045    1886 serve_hostnames.go:175] serve-hostnames-7439/serve-hostname-49-4 is running
satnam@asia-east1-b:~/gocode/src/github.com/GoogleCloudPlatform/kubernetes/test/soak/serve_hostnames$

CCing @gmarek and @wojtek-t who were looking into performance stats.

Yeah this is because we synchronously bind pods. I have a fix baking. @satnam6502 is there a bug for the pod creation reliability issues I can look at?

@bprashanth : sorry, can you be a bit more specific? I intended this to be the "GitHub bug" and there is a link to a "GitHub Issue" where I am reporting the results of running various experiments. Perhaps that's not what you mean?

From your initial report:

In addition to pod creation not working reliably

I was asking if you had details/repro steps/another bug I can look at.

To repo you could use the program that I have a PR out for: https://github.com/GoogleCloudPlatform/kubernetes/pull/5750

@satnam6502 This issue is > 2 weeks old, so I assume it has improved/been fixed, given the various performance improvements we made. Can you please close if it is fixed?

Now definitely fixed, pods are created at a blistering speed.

In realted news, I think we decided we care less about tail latency, which is what you'd get hit by if you're measuring end to end creation time of O(1000) pods. Pods per second is more important and as long as we're doing something like 10pps, we could take 100s to create them all and that would be withing our perf goals.
For more context on this see discussion on https://github.com/GoogleCloudPlatform/kubernetes/issues/3954

authors: bprashanth davidopp fgrzadkowski satnam6502

repo: GoogleCloudPlatform/kubernetes


text:
Update volumes.md

nfs remark

ok to test
———
Jüri

On 06.07.2015, at 13:19, Kubernetes Bot notifications@github.com wrote:
Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")
If this message is too spammy, please complain to ixdy.
—
Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/kubernetes/pull/10741#issuecomment-118799552.

@jyrip thanks for the note - I need you to sort the CLA out before we can merge, though.

I belive I sgined it already.

LGTM

Looks like this will need a rebase btw

@jyrip - please rebase - I'm not able to merge it as is.

removing ok-to-merge as we wait for rebase.

Hi! I tried to, but seems, that I don't  have no rights to upload the result.
Jüri.

On 09.07.2015, at 12:28, Wojciech Tyczynski notifications@github.com wrote:
@jyrip - please rebase - I'm not able to merge it as is.
—
Reply to this email directly or view it on GitHub.

You don't have rights to push to your own fork?

sorry, I am lost here. I understand volumes.md has moved and I don't know what to do with this pull-request.

I think this is a useful comment, but yeah, docs are undergoing some flux right now, so you might need to just close this PR, make the same change again and offer a new one :)

@jyrip can you rebase or open a new PR with this comment? Please reopen when you rebase.

source: gharchive/pull-request

authors: jyrip mikedanese rjnagal thockin vmarmol wojtek-t

license_type: permissive

license_source: github-api


text:
adding shippable.yml

Proposal to add shippable config file to improve build performance
fixes a bug in the previous PR

seems like docker cert expired and the pull is failing because of this error
Get https://index.docker.io/v1/repositories/shipimg/ubuntu1204_go/images: x509: certificate has expired or is not yet vali

I'm closing this for now. Will open another PR once this issue is resolved.

docker seems to have fixed the cert issues.
re-opening PR

@alex-mohr @ixdy @ zmerlynn @brendandburns here is the PR with the shippable config file shippable.yml. You should see a big perf gain with this merge, and we'll continue to iterate and improve further. Thanks again!

I will take a look at this tomorrow (Monday).

@ixdy did you get a chance to look at this yesterday? Let us know if everything looks ok.

Thanks Manisha -- looks reasonable to me, though it's outside my area of expertise.  If Jeff @ixdy doesn't see any red flags, let's merge this tomorrow morning and monitor how it goes?

I'm very sorry that I didn't get to this on Monday as I promised. (Or yesterday.) I will do better in the future.
This generally looks like it's just a copy of our .travis.yml, though it adds a build image and some before_install steps. Is that assumption basically correct? I'm curious which changes help the build performance.
Also, when there's both a .travis.yml and shippable.yml file in the repo, will Shippable only look at the latter?

@ixdy when there is no build_image specified, builds run on our default image. This image (minv2) contains all languages, all tools, and also incurs some overhead that slows down the build. More importantly, it does not have the race package for go and this is causing all the slowness when compared to Travis.
The yml in this PR uses our lightweight go image with the right packages preinstalled. This is cutting our build time to be as fast as Travis. With the added benefit of no community queueing, I expect faster results wrt Travis. Check out this screenshot for this PR build :)

We will continue adding the right things to this image and the yml to optimize it for kubernetes builds, this is an ongoing commitment for us.
And yes, when there is a travis and shippable yml in a repo, Shippable picks up shippable.yml

@ixdy also,we've merged the changes from the other PR as you suggested.

Thanks for the explanation and thanks for the PR! LGTM.

authors: alex-mohr ixdy manishas ric03uec


text:
Improve validation and fix not throwing an error during ns delete

Fixes https://github.com/GoogleCloudPlatform/kubernetes/issues/6633

LGTM

authors: derekwaynecarr lavalamp


text:
This commit fixes "kubectl get nodes" output and makes it more sane. It also fixes related test case resource_printer_test.

In its current form, "kubectl get nodes" output seems misaligned and sometimes incorrect too as it shows for example:
#kubectl get nodes
NAME                LABELS              STATUS
fed-node   Schedulable   name=fed-node-label  NotReady
This patch combines Schedulable/Unschedubale with Ready/NotReady/Unknown in STATUS field. So if unschedulable bit is true, the output will be:
NAME       LABELS                      STATUS
fed-node    name=fed-node-label   Ready,Unschedubale
or STATUS will be (NotReady,Unschedulable) or (Unknown, Unschedulable).
If unschedulable bit is false (by default), it will show:
fed-node    name=fed-node-label   Ready,Schedubale
If Node is in Ready state. But if Node is in NotReady or Unkown state, it will show:
fed-node    name=fed-node-label   NotReady,UnSchedubale  or  (Unknown,Unschedulable)
Currently it shows Schedulable  even with  NotReady or Unknown, which seems incorrect as it does not make any sense to output a Node as Schedulable when in NotReady/Unknown state. As soon as a Node becomes Ready, it will show STATUS as Ready,Schedulable.

Travis Shows:
The command "./hack/verify-gofmt.sh" failed and exited with 1 during .
Please run gofmt on your binaries and make sure hack/verify-gofmt.sh can pass.

starngely it passed in koji: https://kojipkgs.fedoraproject.org/work/tasks/1655/9491655/build.log

@aveshagarwal I don't know why it passed in koji, but it clearly doesn't pass anywhere else.
The issue is that "case ..." is formatted incorrectly.
Can you please run gofmt and update this PR?

@aveshagarwal koji does not run verify-gendocs, verify-gofmt, verify-boilerplate, etc.  It only runs the build tests.

ok thanks for comments, I am working to fix it.

Fixed case formatting error and updated PR.

"Schedulable" and "Ready" are (now) unrelated/orthonganal. An admin would manually set a node "Unschedulable". The system determines "Ready/NotReady/Unknown"
Since "schedulable" will almost always be true, and this is for human consumption, can we just get rid of it entirely?
Since "unschedulable" is really a state set or unset explicitly by the admin, I'm not sure we should show it for NotReady/Unknown.
What do you think Avesh? Can you try to write a test for that interesting case?  Ready but admin set Unschedulable?

I agree with you that Ready/NotReady/Unknown and Unschedulable/Schedulable are kind of orthogonal.
Thats why a node could be Ready,Unschedulable if set by admins,  or Ready,Schedulable by default.
I agree and I would guess that most of the time, in normal cases, it would be Ready,Schedulable.
On the other hand, If a node is set by an admin as unschedulable, and without showing "Unschedubale" in STATUS, it would be very confusing IMHO, to show STATUS just as Ready and not knowing why this node is not doing anything if there are hundreds or thousands of nodes. So it really shows very important piece of information.
One thing that might be done is to add one more column to segregate these two statuses would be like:
Name Label Schedulability Status or something like this. Just a suggestion.
I have just tested it (Ready,Unschedulable) manually by writing above json files for nodes. I will have to look about writing a test case.
Believe me that I understand and agree completely what would you think to eliminate Schedubale/Unschedubale, but as I said my only concern is that what if an admin sets a node Unschedulable, and without showing it that information is lost and might create confusion.
That said, I am fine even if we decide to eliminate Schedulable/Unschedulable as far as the output is consistent, as anyway it was not there before.

I was only suggesting getting rid of "schedulable"   So you would show just "ready"   or "ready,unschedulable"

@eparis I agree that's better that's what I was thinking too. I have pushed updated PR. You might look to see if it does as expected. Regarding the test, I will look into that.

cc @davidopp @gmarek @ddysher

We should document unschedulable in node.md:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/node.md

I totally understand the conundrum, but I don't agree with this solution.
I had originally suggested we call it MarkedUnschedulable rather than Unschedulable -- to make it clearer that this is reporting what a human set, not what the system will do. That's still the best solution I can think of. An administrator may want to know the Unschedulable status, regardless of the NodeReady status. So I don't think suppressing printing it in some cases (as you do in this PR) is the best thing to do. I think we should rename Unschedulable to MarkedUnschedulable.

How about SchedulingDisabled?
Or SchedulingEnabled. @lavalamp said he'd prefer not to make the field negative, which I kind of agree with, though if we're not happy with disabled by default, when we'd need to make the field a pointer in order to be able to detect the lack of specification in the defaulting code.

So we have following options now:

Adding a new coloum: NAME LABELS SCHEDULING STATUS and then under scheduling we could have Enabled/Dislabled.

So we would have following outputs:
a) if unschedulable is set:
NAME LABELS SCHEDULING STATUS
   Disabled   Ready
   Enabled   Ready
   Disabled   NotReady
   Enabled   NotReady
   Disabled   Unknown
   Enabled   Unknown

Keeping the columns same, and as @davidopp suggested, we could have
NAME LABELS STATUS
   Ready,MarkedUnschedulable
   Ready,MarkedSchedulable
   NotReady,MarkedUnschedulable
   NotReady,MarkedSchedulable
  Unknown,MarkedUnschedulable
   Unknown,MarkedSchedulable

As @lavalamp and @bgrant0607 suggested:
NAME LABELS STATUS
   Ready,SchedulingEnabled
   Ready,SchedulingDisabled
   NotReady,SchedulingEnabled
   NotReady,SchedulingDisabled
  Unknown,SchedulingEnabled
   Unknown,SchedulingDisabled

Or as @eparis suggested to eliminate schedulable:
NAME LABELS STATUS
   Ready,UnSchedulable
   Ready
   NotReady,Unschedulable
   NotReady
  Unknown,Unschedulable
   Unknown

or something else. I am fine with anything as far as it conveys what is expected and consistent.

I'm with bgrant, SchedulingDisabled is a good name. But we should suppress ""SchedulingEnabled" altogether. Its the default, it's not something the user needs thrown in his face.
I also think the negative in the api is ok. It is much more intuitive that you need to set Unschedulable to stop scheduling that it is to have to set schedulable to false...

so shall we go with SchedulingDisabled, and leaving out SchedulingEnabled? In that case, we would have:

if unschedulable is true: Ready,SchedulingDisabled or NotReady,SchedulingDisabled or Unknwon,SchedulingDisabled
otherwise: Ready or NotReady or Unknown

So do we have agreement here?

We'd better rename the field in the api as well, but that's out of scope for this PR.
I like ScheduingDisabled.  So "Ready,SchedulingDisabled", and "Ready" sounds good.
Make sure you also update kubectl describe as well.

cc @ppires

I agree: SchedulingDisabled, elide enabled.

if unschedulable is true: Ready,SchedulingDisabled or NotReady,SchedulingDisabled or
Unknwon,SchedulingDisabled
otherwise: Ready or NotReady or Unknown

Yeah that sounds good.

Thanks for feedback. Pushed updated PR.

Thanks avesh, add a unit test and I'll merge.
hint: https://gist.github.com/eparis/183e866b1d85f408db6e

@eparis thanks for the hint. I am trying to understand why does it have api.NodeReady twice? As per my (limited) understanding, it should be :
{
minion: api.Node{
ObjectMeta: api.ObjectMeta{Name: "foo2"},
Spec:       api.NodeSpec{Unschedulable: true},
Status: api.NodeStatus{Conditions: []api.NodeCondition{
{Type: api.NodeReady, Status: api.ConditionTrue}}},
},
status: "Ready,SchedulingDisabled",
},
},

Pushed updated PR with unit test cases.

you are absolutely right.  foo2 should not have double status.  foo3 did it on purpose to specifically test what the apiserver does it if gets 2 copies of status.  So lets put that test back.

ok, somehow I thought foo3 as redundant. Anyway I have put foo3 back and pushed updated PR.

authors: aveshagarwal bgrant0607 davidopp ddysher eparis wojtek-t


text:
Fixing node version warning when deploying with gcloud.

gcloud gives a warning about a missing node version:

WARNING: No node version specified.  Please add your node version, see https://docs.npmjs.com/files/package.json#engines

This change fixes this for 1-hello-world-guided. This should be propagated to the other branches.

I have to rebase the repository to apply this. Closing this PR and created #28. Thanks!

authors: chuckjaz jonparrott

license_source: bigquery


text:
Support Gauge Distributions in Cloud Monitoring Exporter

My team is maintaining an OTel collector. We provide this collector to other teams at Google. Those teams use the collector to gather metrics about their VMs/clusters/... .
One of our customers has a Gauge Distribution metric in Cloud Monitoring. It is a valid thing, according to https://cloud.google.com/monitoring/api/v3/kinds-and-types#kind-type-combos.
However, in OTel, only Delta and Cumulative histograms (distributions) exist. Thus, there is no way to express gauge distributions. We would like to have this functionality.

We are probably planning this eventually, but right now we want to target https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/issues/624 first

I think this would require OTel spec changes.  At minimum, it would need to be added to the OTel data model.

source: gharchive/issue

authors: ItsLastDay damemi dashpole


text:
Make integrationtest public

This is so that users of this exporter can easily write similar GCM integration tests to the ones we have here.

I have no objection to this PR, but I thought we also discussed adding an option to dump out RPC requests. Are we still planning to do that in a separate PR?

I think we can do that at a later point if we still want to.  Other users of the exporter can actually start using the integration tests directly, even while using the opencensus-based exporter.

authors: dashpole punya


text:
Missing data points with monitoring metric log_entry_count and request_count

Describe the bug
In Cloud Monitoring, the values of the metrics agent.googleapis.com/agent/log_entry_count and agent.googleapis.com/agent/request_count do not increase even though the Logging Agent keeps sending logs to Cloud Logging.
Also, by querying the /metrics path of the fluent bit agent on an instance, I can see that the metric fluentbit_stackdriver_proc_records_total keeps increasing.
Other metrics such as uptime or memory_usage seem fine and report legit values.
Expected behavior
The log_entry_count and request_count metrics increase as logs are forwarded to Cloud Logging by the Logging Agent.
Environment:

VM distro / OS:  Debian 10
Ops Agent version: 2.21.0
Ops Agent configuration:

logging:
 processors:
   docker:
     time_format: '%Y-%m-%dT%H:%M:%S.%LZ'
     time_key: time
     type: parse_json
   syslog:
     regex: '^\S+ \d+ \d+:\d+:\d+ (?<vm_name>\S+) (?<process>\S+)\[\d+\]: (?<message>.*)$'
     type: parse_regex
 receivers:
   docker:
     include_paths:
     - /var/lib/docker/containers/*/*-json.log
     type: files
   syslog:
     include_paths:
     - /var/log/messages
     - /var/log/syslog
     type: files
 service:
   pipelines:
     default_pipeline:
       processors:
       - syslog
       receivers:
       - syslog
     docker:
       processors:
       - docker
       receivers:
       - docker
metrics:
 service:
   pipelines:
     default_pipeline:
       receivers: []

Additional information
The Ops Agent is installed with the Ansible playbook https://github.com/GoogleCloudPlatform/google-cloud-ops-agents-ansible.

Closing this issue as this was fixed in https://github.com/GoogleCloudPlatform/ops-agent/releases/tag/2.23.0.

repo: GoogleCloudPlatform/ops-agent


text:
chore: enable boring crypto

Enable boring crypto flag for builds of our fork.

LGTM - can you create PRs against 2.43 and 2.45?

Actually @bernot-dev can you kick-off a prombench too?

/prombench

@pintohutch I don't see anything concerning in the Prombench results. If anything, the boringcrypto version performs very slightly better on some of the benchmarks, but I assume it's within the margin for error.

Awesome - thanks for checking

/prombench cancel

authors: bernot-dev pintohutch

repo: GoogleCloudPlatform/prometheus