text:
Support ClusterSpec Propagation Feature in TF 1.14

Proposal: https://github.com/kubeflow/tf-operator/issues/1141
Test:
go test -v ./pkg/controller.v1/tensorflow

Travis tests have failed
Hey @zhujl1991,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     hack/verify-codegen.sh

Generating deepcopy funcs
Generating clientset for tensorflow:v1 at github.com/kubeflow/tf-operator/pkg/client/clientset
Generating listers for tensorflow:v1 at github.com/kubeflow/tf-operator/pkg/client/listers
Generating informers for tensorflow:v1 at github.com/kubeflow/tf-operator/pkg/client/informers
Generating defaulters for tensorflow/v1
Generating OpenAPI specification for tensorflow/v1
diffing hack/../pkg against freshly generated codegen
diff -Naupr hack/../pkg/apis/tensorflow/v1/openapi_generated.go hack/../_tmp/pkg/apis/tensorflow/v1/openapi_generated.go
--- hack/../pkg/apis/tensorflow/v1/openapi_generated.go	2020-03-23 19:11:53.783089704 +0000
+++ hack/../_tmp/pkg/apis/tensorflow/v1/openapi_generated.go	2020-03-23 19:11:13.000000000 +0000
@@ -164,13 +164,6 @@ func GetOpenAPIDefinitions(ref common.Re
 								},
 							},
 						},
-						"enableDynamicWorker": {
-							SchemaProps: spec.SchemaProps{
-								Description: "A switch to enable dynamic worker",
-								Type:        []string{"boolean"},
-								Format:      "",
-							},
-						},
 					},
 					Required: []string{"tfReplicaSpecs"},
 				},
hack/../pkg is out of date. Please run hack/update-codegen.sh

     golangci-lint run ./...

pkg/controller.v1/tensorflow/pod.go:81:28: Error return value of `tc.PodControl.DeletePod` is not checked (errcheck)
				tc.PodControl.DeletePod(tfjob.Namespace, pods.Name, tfjob)
				                       ^
pkg/controller.v1/tensorflow/pod_test.go:390:10: Error return value of `ctr.Run` is not checked (errcheck)
		ctr.Run(testutil.ThreadCount, stopCh)
		       ^
pkg/controller.v1/tensorflow/pod_test.go:470:10: Error return value of `ctr.Run` is not checked (errcheck)
		ctr.Run(testutil.ThreadCount, stopCh)
		       ^
pkg/controller.v1/tensorflow/service.go:59:36: Error return value of `tc.ServiceControl.DeleteService` is not checked (errcheck)
				tc.ServiceControl.DeleteService(tfjob.Namespace, service.Name, tfjob)
				                               ^

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/*.go,pkg/util/*/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.033s	coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateV1TFJobSpec
time="2020-03-23T19:13:38Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2020-03-23T19:13:38Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.034s	coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.017s	coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2020-03-23T19:14:04Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2020-03-23T19:14:04Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.050s	coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.260663ms)" job=default.test-tfjob
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (527.948µs)" job=default.test-tfjob
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (316.409µs)" job=default.test-tfjob
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (508.971µs)" job=default.test-tfjob
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2020-03-23T19:14:09Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2020-03-23T19:14:09Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2020-03-23T19:14:09Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2020-03-23T19:14:09Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-03-23T19:14:09Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0323 19:14:09.696365   10437 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc00005e380), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc0004f6000), "Worker":(*v1.ReplicaSpec)(0xc000309b80)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0005ee430)}, StartTime:(*v1.Time)(0xc0004c3840), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.271401ms)" job=default.test-tfjob
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (274.797µs)" job=default.test-tfjob
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.075298ms)" job=default.test-tfjob
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (326.12µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
time="2020-03-23T19:14:09Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:09Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:09Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:09Z" level=info msg="Starting 1 workers"
time="2020-03-23T19:14:09Z" level=info msg="Started workers"
time="2020-03-23T19:14:10Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestAddTFJob
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:10Z" level=info msg="Starting 1 workers"
time="2020-03-23T19:14:10Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2020-03-23T19:14:10Z" level=info msg="Shutting down workers"
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:10Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (340.765µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2020-03-23T19:14:10Z" level=info msg="Starting 1 workers"
time="2020-03-23T19:14:10Z" level=info msg="Started workers"
time="2020-03-23T19:14:10Z" level=info msg="Shutting down workers"
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (256.501µs)" job=default.test-tfjob
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (233.341µs)" job=default.test-tfjob
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (252.691µs)" job=default.test-tfjob
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (261.965µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.00s)
=== RUN   TestCleanupTFJob
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (171.696µs)" job=default.test-tfjob
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (184.04µs)" job=default.test-tfjob
time="2020-03-23T19:14:10Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:10Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (499.351µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2020-03-23T19:14:12Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:12Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:12Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:12Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (355.492µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:14Z" level=info msg="Starting 1 workers"
time="2020-03-23T19:14:14Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestIsDistributed
--- PASS: TestIsDistributed (0.00s)
time="2020-03-23T19:14:14Z" level=info msg="Shutting down workers"
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2020-03-23T19:14:14Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0323 19:14:14.424366   10437 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc00038e7d0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc00090fb80)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0004218f0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2020-03-23T19:14:14Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:14Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0323 19:14:14.424679   10437 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc00038e7d0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc00090fb80)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0004218f0)}, StartTime:(*v1.Time)(0xc0004c32e0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2020-03-23T19:14:14Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (812.151µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
time="2020-03-23T19:14:14Z" level=info msg="Starting 1 workers"
time="2020-03-23T19:14:14Z" level=info msg="Started workers"
time="2020-03-23T19:14:14Z" level=info msg="Shutting down workers"
=== RUN   TestScaleDown
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Removing 2 workers" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (327.636µs)" job=default.test-tfjob
--- FAIL: TestScaleDown (0.00s)
    pod_test.go:428: Scale down workers test failed
=== RUN   TestScaleUp
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-23T19:14:14Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:14Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (746.828µs)" job=default.test-tfjob
--- PASS: TestScaleUp (0.00s)
time="2020-03-23T19:14:14Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:14Z" level=info msg="Starting 1 workers"
time="2020-03-23T19:14:14Z" level=info msg="Started workers"
time="2020-03-23T19:14:14Z" level=info msg="Shutting down workers"
time="2020-03-23T19:14:14Z" level=info msg="Starting 1 workers"
=== RUN   TestAddService
time="2020-03-23T19:14:14Z" level=info msg="Started workers"
time="2020-03-23T19:14:14Z" level=info msg="Shutting down workers"
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="Starting TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-23T19:14:14Z" level=info msg="Starting 1 workers"
time="2020-03-23T19:14:14Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2020-03-23T19:14:14Z" level=info msg="Shutting down workers"
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0323 19:14:14.530354   10437 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc0005f9020), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc000498dc0)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0005c12d0)}, StartTime:(*v1.Time)(0xc0005f3120), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2020-03-23T19:14:14Z" level=info msg="Creating TFJob controller"
time="2020-03-23T19:14:14Z" level=info msg="Creating Job controller"
time="2020-03-23T19:14:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-23T19:1Labels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
FAIL
coverage: 53.6% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
FAIL	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.902s
exit status 1: warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util/signals
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/version

TravisBuddy Request Identifier: 7d1e9b30-6d3a-11ea-9dc6-19e34c018b2a

/ok-to-test
Is it ready to review?

/ok-to-test
Is it ready to review?

Yes, it is. I'll fix the test failure.

Coverage remained the same at 96.512% when pulling 4f04cbee2e7538e32a2e7d80633e6b9077e90a2d on zhujl1991:community into 011b3ecf4c8e53511ac954c6e6af8163e56f720f on kubeflow:master.

Ping for the review : ) @gaocegege @Jimexist

/assign @johnugeorge @ChanYiLin @richardsliu

ReplicaStatuses

Our internal use case is for online learning, which is more like a long running service and should never be succeeded. I read the existing code and play around. It turns out that the current status related code has already met our needs. So I don't have to make any change for the status part. With that being said, I do think there might be something needs to be done here if we use it for normal offline training jobs. But I think we can make the code change for that part upon request.

Although my change is mainly for the online learning case, I don't think your concern is valid even for offline training jobs:
The number of Active/Succeeded/Failed pods is calculated every single time https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L126 from scratch https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L71 based on the pods in podSlices https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L74 . But remember, in my code, the pods in podSlices only include the pods that should be kept in the cluster -- the pods that need to be deleted are not in it. Actually, the to-be-deleted pods are in podsToBeRemoved variable.
So in your example, we scale down the replicas from 3 to 2, the controller delete index_3, but didn't update the status and ReplicaStatuses count is not valid. tfjob.Status.ReplicaStatuses["Worker"].Succeeded should be 1 instead of 2.

I didn't see any change related to updateStatus which is also mentioned in the proposal.
Will you open another PR to handle that?
Regarding to status, I have a question
we update the status based on the expected, running, failed calculated beforehand
pkg/controller.v1/tensorflow/status.go#L61
what if initially there are 3 workers
1 succeeded
2 running
3 succeeded

tfjob.Status.ReplicaStatuses["Worker"].Active =1
tfjob.Status.ReplicaStatuses["Worker"].Succeeded =2

then we scale down the replicas from 3 to 2, the controller delete index_3, but didn't update the status and ReplicaStatuses count
1 succeeded
2 running

tfjob.Status.ReplicaStatuses["Worker"].Active =1
tfjob.Status.ReplicaStatuses["Worker"].Succeeded =2 // which we should update it to 1 after scaling down

expected := replicas - int(tfjob.Status.ReplicaStatuses[commonType].Succeeded) // will be 2 - 2 = 0, and is wrong, because index_2 is still running

we need to update the ReplicaStatuses count and update the status, or I think there will be bugs
What do you think?

Our internal use case is for online learning, which is more like a long running service and should never be succeeded. I read the existing code and play around. It turns out that the current status related code has already met our needs. So I don't have to make any change for the status part. With that being said, I do think there might be something needs to be done here if we use it for normal offline training jobs. But I think we can make the code change for that part upon request.

Although my change is mainly for the online learning case, I don't think your concern is valid even for offline training jobs:
The number of Active/Succeeded/Failed pods is calculated every single time https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L126 from scratch https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L71 based on the pods in podSlices https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L74 . But remember, in my code, the pods in podSlices only include the pods that should be kept in the cluster -- the pods that need to be deleted are not in it. Actually, the to-be-deleted pods are in podsToBeRemoved variable.
So in your example, we scale down the replicas from 3 to 2, the controller delete index_3, but didn't update the status and ReplicaStatuses count is not valid. tfjob.Status.ReplicaStatuses["Worker"].Succeeded should be 1 instead of 2.

Our internal use case is for online learning, which is more like a long running service and should never be succeeded. I read the existing code and play around. It turns out that the current status related code has already met our needs. So I don't have to make any change for the status part. With that being said, I do think there might be something needs to be done here if we use it for normal offline training jobs. But I think we can make the code change for that part upon request.
Although my change is mainly for the online learning case, I don't think your concern is valid even for offline training jobs:
The number of Active/Succeeded/Failed pods is calculated every single time https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L126
from scratch https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L71
based on the pods in podSlices https://github.com/kubeflow/tf-operator/blob/011b3ecf4c8e53511ac954c6e6af8163e56f720f/pkg/controller.v1/tensorflow/pod.go#L74
. But remember, in my code, the pods in podSlices only include the pods that should be kept in the cluster -- the pods that need to be deleted are not in it. Actually, the to-be-deleted pods are in podsToBeRemoved variable.
So in your example, we scale down the replicas from 3 to 2, the controller delete index_3, but didn't update the status and ReplicaStatuses count is not valid. tfjob.Status.ReplicaStatuses["Worker"].Succeeded should be 1 instead of 2.

Got it! You are right, the number is calculated every single time. Then I don't have doubt on the status part.

Thanks for the PR, generally LGTM.

Travis tests have failed
Hey @zhujl1991,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/*.go,pkg/util/*/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.035s	coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateV1TFJobSpec
time="2020-03-31T07:23:38Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2020-03-31T07:23:38Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.032s	coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.015s	coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2020-03-31T07:24:02Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2020-03-31T07:24:02Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.048s	coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (858.746µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (635.517µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (444.295µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (759.332µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2020-03-31T07:24:08Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2020-03-31T07:24:08Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2020-03-31T07:24:08Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2020-03-31T07:24:08Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2020-03-31T07:24:08Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0331 07:24:08.143068   10501 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc000254450), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc000434000), "Worker":(*v1.ReplicaSpec)(0xc0002b3b80)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc00001aea0)}, StartTime:(*v1.Time)(0xc00034d7a0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.106698ms)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (436.331µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (7.053708ms)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (490.468µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:08Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:08Z" level=info msg="Started workers"
time="2020-03-31T07:24:08Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestAddTFJob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:08Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:08Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
time="2020-03-31T07:24:08Z" level=info msg="Shutting down workers"
=== RUN   TestCopyLabelsAndAnnotation
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (329.49µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2020-03-31T07:24:08Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:08Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:08Z" level=info msg="Started workers"
time="2020-03-31T07:24:08Z" level=info msg="Shutting down workers"
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (298.305µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (241.986µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (247.376µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (233.606µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.00s)
=== RUN   TestCleanupTFJob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (290.952µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:08Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:08Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (238.514µs)" job=default.test-tfjob
time="2020-03-31T07:24:08Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:08Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:10Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (411.956µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2020-03-31T07:24:10Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:10Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:10Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:10Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:10Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:10Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:10Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (537.695µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:12Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2020-03-31T07:24:12Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestIsDistributed
--- PASS: TestIsDistributed (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" levelmal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2020-03-31T07:24:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:12Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0331 07:24:12.877543   10501 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc000760780), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0001a0580)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0006150c0)}, StartTime:(*v1.Time)(0xc000180700), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2020-03-31T07:24:12Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (888.603µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
time="2020-03-31T07:24:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:12Z" level=info msg="Started workers"
time="2020-03-31T07:24:12Z" level=info msg="Shutting down workers"
=== RUN   TestScaleDown
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Removing 2 workers" job=default.test-tfjob replica-type=worker uid=
E0331 07:24:12.878941   10501 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc0005b30b0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0001a0dc0)}, EnableDynamicWorker:true}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc00053a190)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'PodScaleDown' 'Pod: default.worker-2 is being removed'
E0331 07:24:12.879038   10501 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc0005b30b0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0001a0dc0)}, EnableDynamicWorker:true}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc00053a190)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'PodScaleDown' 'Pod: default.worker-1 is being removed'
time="2020-03-31T07:24:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (580.385µs)" job=default.test-tfjob
--- FAIL: TestScaleDown (0.00s)
time="2020-03-31T07:24:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:12Z" level=info msg="Started workers"
time="2020-03-31T07:24:12Z" level=info msg="Shutting down workers"
    pod_test.go:433: Scale down workers test failed
=== RUN   TestScaleUp
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:12Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T07:24:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.400408ms)" job=default.test-tfjob
--- PASS: TestScaleUp (0.00s)
time="2020-03-31T07:24:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:12Z" level=info msg="Started workers"
time="2020-03-31T07:24:12Z" level=info msg="Shutting down workers"
=== RUN   TestAddService
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T07:24:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T07:24:12Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0331 07:24:12.985172   10501 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc0002fe150), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0004382c0)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0005ab370)}, StartTime:(*v1.Time)(0xc00000efa0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2020-03-31T07:24:12Z" level=info msg="Shutting down workers"
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:12Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:13Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:13Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:13Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:13Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-03-31T07:24:13Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T07:24:13Z" level=info msg="Creating TFJob controller"
time="2020-03-31T07:24:13Z" level=info msg="Creating Job controller"
time="2020-03-31T07:24:13Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-31T07:24:13Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-03-31T07:24:13Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestConvertClusterSpecToSparseClusterSpec
--- PASS: TestConvertClusterSpecToSparseClusterSpec (0.00s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
FAIL
coverage: 53.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
FAIL	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.910s
exit status 1: warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util/signals
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/version

TravisBuddy Request Identifier: 9ffd3260-7320-11ea-8d6e-5f9eb1f9039f

Travis tests have failed
Hey @zhujl1991,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.
1st Build
View build log

     goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/*.go,pkg/util/*/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"

=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.038s	coverage: 27.7% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateV1TFJobSpec
time="2020-03-31T08:19:43Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2020-03-31T08:19:43Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.034s	coverage: 20.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.015s	coverage: 0.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2020-03-31T08:20:06Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2020-03-31T08:20:06Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.060s	coverage: 24.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.008236ms)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (443.608µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (616.336µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (476.854µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (331.621µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (502.146µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2020-03-31T08:20:12Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2020-03-31T08:20:12Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2020-03-31T08:20:12Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2020-03-31T08:20:12Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2020-03-31T08:20:12Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0331 08:20:12.195459   10385 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc00069c3d0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc00069e2c0), "Worker":(*v1.ReplicaSpec)(0xc00069e000)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"PS":(*v1.ReplicaStatus)(0xc0004d1120), "Worker":(*v1.ReplicaStatus)(0xc0004d1140)}, StartTime:(*v1.Time)(0xc000671c20), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.66828ms)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (331.492µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.01s)
=== RUN   TestRun
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T08:20:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T08:20:12Z" level=info msg="Started workers"
time="2020-03-31T08:20:12Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestAddTFJob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T08:20:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T08:20:12Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2020-03-31T08:20:12Z" level=info msg="Shutting down workers"
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Starting TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (814.763µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2020-03-31T08:20:12Z" level=info msg="Starting 1 workers"
time="2020-03-31T08:20:12Z" level=info msg="Started workers"
time="2020-03-31T08:20:12Z" level=info msg="Shutting down workers"
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (367.424µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (224.622µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (334.971µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (220.576µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (228.765µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:12Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:12Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (197.364µs)" job=default.test-tfjob
time="2020-03-31T08:20:12Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:12Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:14Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (343.929µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2020-03-31T08:20:14Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:14Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:14Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:14Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:14Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:14Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:16Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2020-03-31T08:20:16Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:16Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:16Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2020-03-31T08:20:16Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (785.249µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2020-03-31T08:20:16Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:16Z" level=info msg="Starting TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T08:20:16Z" level=info msg="Starting 1 workers"
time="2020-03-31T08:20:16Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestIsDistributed
time="2020-03-31T08:20:16Z" level=info msg="Shutting down workers"
--- PASS: TestIsDistributed (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2020-03-31T08:20:16Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:16Z" level=info msg="Starting TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T08:20:16Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:16Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2020-03-31T08:20:16Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0331 08:20:16.923471   10385 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc00069c690), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0009f8580)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc00046b7e0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2020-03-31T08:20:16Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:16Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0331 08:20:16.923556   10385 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc00069c690), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0009f8580)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc00046b7e0)}, StartTime:(*v1.Time)(0xc0006a62e0), CompletionTime:(*v1.Time)(nil), Last(0.00s)
    pod_test.go:436: Scale down workers test failed
=== RUN   TestScaleUp
time="2020-03-31T08:20:16Z" level=info msg="Starting 1 workers"
time="2020-03-31T08:20:16Z" level=info msg="Started workers"
time="2020-03-31T08:20:16Z" level=info msg="Shutting down workers"
time="2020-03-31T08:20:16Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:16Z" level=info msg="Starting TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T08:20:16Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2020-03-31T08:20:16Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:16Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:16Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:16Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:16Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:16Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2020-03-31T08:20:16Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (901.066µs)" job=default.test-tfjob
--- PASS: TestScaleUp (0.00s)
=== RUN   TestAddService
time="2020-03-31T08:20:16Z" level=info msg="Starting 1 workers"
time="2020-03-31T08:20:16Z" level=info msg="Started workers"
time="2020-03-31T08:20:16Z" level=info msg="Shutting down workers"
time="2020-03-31T08:20:16Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:16Z" level=info msg="Starting TFJob controller"
time="2020-03-31T08:20:16Z" level=info msg="Waiting for informer caches to sync"
time="2020-03-31T08:20:17Z" level=info msg="Starting 1 workers"
time="2020-03-31T08:20:17Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2020-03-31T08:20:17Z" level=info msg="Shutting down workers"
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0331 08:20:17.030094   10385 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc000346570), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc0005aa000)}, EnableDynamicWorker:false}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc0004d0e30)}, StartTime:(*v1.Time)(0xc000356740), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjo0, running=0, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="Creating TFJob controller"
time="2020-03-31T08:20:17Z" level=info msg="Creating Job controller"
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2020-03-31T08:20:17Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestConvertClusterSpecToSparseClusterSpec
--- PASS: TestConvertClusterSpecToSparseClusterSpec (0.00s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
FAIL
coverage: 53.5% of statements in github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
FAIL	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.901s
exit status 1: warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/util/signals
warning: no packages being tested depend on matches for pattern github.com/kubeflow/tf-operator/pkg/version

TravisBuddy Request Identifier: 753e88a0-7328-11ea-8d6e-5f9eb1f9039f

/lgtm
/assign @ChanYiLin

/approve

Thanks for your contribution! :tada: :+1:

authors: ChanYiLin TravisBuddy coveralls gaocegege zhujl1991

repo: kubeflow/tf-operator