text:
oadp-1.3: OADP-4265: Reconcile To Fail: Add backup/restore trackers

Signed-off-by: Tiger Kaovilai tkaovila@redhat.com
Thank you for contributing to Velero!
Please add a summary of your change
Test image build
export REGISTRY=ghcr.io/kaovilai VERSION=$(git branch --show-current) && make container && docker push $REGISTRY/velero:$VERSION

image
ghcr.io/kaovilai/velero:OADP-4265-oadp-1.3_addTrackers

UBI build image: ghcr.io/kaovilai/velero:[OADP-4265](https://issues.redhat.com//browse/OADP-4265)-oadp-1.3_addTrackers-ubi
export TAG=ghcr.io/kaovilai/velero:$(git branch --show-current)-ubi; docker build --platform=linux/amd64 -f Dockerfile.ubi -t $TAG . && docker push $TAG

Does your change fix a particular issue?
Fixes #(issue)
Please indicate you've done the following:

[ ] Accepted the DCO. Commits without the DCO will delay acceptance.
[ ] Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
[ ] Updated the corresponding documentation in site/content/docs/main.

@kaovilai and I are going to test this patch tomorrow on the SNO cluster, let's hold until that time.
/hold

Will run local OADP e2e against this PR.

spec:
  backupLocations:
  - velero:
      config:
        insecureSkipTLSVerify: "true"
        profile: default
        region: us-west-2
      credential:
        key: cloud
        name: cloud-credentials
      default: true
      objectStorage:
        bucket: cvpbucketuswest2
        prefix: velero
      provider: aws
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultPlugins:
      - openshift
      - aws
  snapshotLocations:
  - velero:
      config:
        profile: default
        region: us-west-2
      provider: aws
  unsupportedOverrides:
    veleroImageFqin: ghcr.io/kaovilai/velero:OADP-4265-oadp-1.3_addTrackers
status:

NAME                        STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
westest-vsphere-apidown-1   Failed      0        0          2024-06-25 12:37:28 -0400 EDT   30d       dpa-sample-1       <none>
westest-vsphere-apiup-3     Completed   0        0          2024-06-25 12:38:59 -0400 EDT   30d       dpa-sample-1       <none>

Working as designed
started backpu apidown-1, brought down api server, waited for 30 seconds or so, brought api server back up.  Velero properly moved the back up to failed.
Leaving the api server up:
apiup-3 completed.
woot 0/

@kaovilai is going to paste in results from retesting e2e tests w/ an image from this PR.

So far e2e has some passing and some failures.
Changes needed to run:
https://github.com/openshift/oadp-operator/pull/1441
https://github.com/openshift/oadp-operator/pull/1437

Latest local e2e results, 7 of 10 backup tests passes.
Test results

Starting: /Users/tiger/go/bin/dlv dap --listen=127.0.0.1:56177 --log-dest=3 from /Users/tiger/oadp-operator/tests/e2e
DAP server listening at: 127.0.0.1:56177
Type 'dlv help' for list of commands.
Running Suite: OADP E2E using velero prefix: velero-e2e-0550ebf0-13b9-41df-831a-6decc72e3426 - /Users/tiger/oadp-operator/tests/e2e
===================================================================================================================================
Random Seed: 1719424431
Will run 10 of 32 specs
2024/06/26 13:53:51 Waiting for velero pod to be running
2024/06/26 13:53:51 velero pods not found
2024/06/26 13:53:56 velero pods not found
2024/06/26 13:54:01 velero pods not found
2024/06/26 13:54:06 velero pods not found
2024/06/26 13:54:11 velero pods not found
2024/06/26 13:54:16 velero pods not found
2024/06/26 13:54:21 velero pods not found
2024/06/26 13:54:26 velero pods not found
2024/06/26 13:54:31 velero pods not found
2024/06/26 13:54:36 velero pods not found
2024/06/26 13:54:41 velero pods not found
2024/06/26 13:54:47 velero pods not found
2024/06/26 13:54:52 velero pods not found
2024/06/26 13:54:57 velero pods not found
2024/06/26 13:55:02 velero pods not found
2024/06/26 13:55:07 velero pods not found
2024/06/26 13:55:12 velero pods not found
2024/06/26 13:55:17 velero pods not found
2024/06/26 13:55:22 velero pods not found
2024/06/26 13:55:27 velero pods not found
2024/06/26 13:55:33 velero pods not found
2024/06/26 13:55:38 velero pods not found
2024/06/26 13:55:43 pod: velero-68c4958ddc-g5qzr is not yet running: phase is Pending
2024/06/26 13:55:48 pod: velero-68c4958ddc-g5qzr is not yet running: phase is Pending
2024/06/26 13:55:53 velero pods are running
2024/06/26 13:55:53 Creating VolumeSnapshotClass for CSI backuprestore of mysql-csi-e2e
2024/06/26 13:55:53 Installing application for case mysql-csi-e2e
2024/06/26 13:55:53 Creating pvc for case mysql-csi-e2e
2024/06/26 13:55:53 Pod mysql-74f94448f9-s7mpx not yet succeeded: phase is Pending
2024/06/26 13:55:58 Pod mysql-74f94448f9-s7mpx not yet succeeded: phase is Pending
2024/06/26 13:56:03 Pod mysql-74f94448f9-s7mpx not yet succeeded: phase is Pending
2024/06/26 13:56:08 Pod mysql-74f94448f9-s7mpx not yet succeeded: phase is Pending
2024/06/26 13:56:13 Pod mysql-74f94448f9-s7mpx not yet succeeded: phase is Pending
2024/06/26 13:56:19 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:56:24 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:56:29 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:56:34 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:56:39 Pod todolist-1-zzcqc not yet succeeded: phase is Pending
2024/06/26 13:56:44 Running pre-backup function for case mysql-csi-e2e
2024/06/26 13:56:44 checking for the NAMESPACE: mysql-persistent
2024/06/26 13:56:44 Verifying backup/restore data of todolist
2024/06/26 13:56:44 PrebackState: true
2024/06/26 13:56:44 Data before the curl request:
 [{"Id":1,"Description":"time to make the donuts","Completed":false}]
2024/06/26 13:56:44 Writing data to backupFile (backup-data.txt):
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 13:56:44.484831 -0400 EDT m=+173.506445626","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]
2024/06/26 13:56:44 Sleeping for 0s to allow application to be ready for case mysql-csi-e2e
2024/06/26 13:56:44 Creating backup mysql-csi-e2e-54b89fda-33e5-11ef-a1bc-1acc26a6847d for case mysql-csi-e2e
2024/06/26 13:56:54 backup phase: InProgress
2024/06/26 13:57:04 backup phase: WaitingForPluginOperations
2024/06/26 13:57:14 backup phase: WaitingForPluginOperations
2024/06/26 13:57:24 backup phase: Completed
2024/06/26 13:57:25 Backup for case mysql-csi-e2e succeeded
2024/06/26 13:57:25 waiting for volume snapshot contents snapcontent-60c3c95b-ac85-4864-9bb6-8c49ca1b1175 to be ready
2024/06/26 13:57:25 Uninstalling app for case mysql-csi-e2e
2024/06/26 13:57:41 Creating restore mysql-csi-e2e-54b8a502-33e5-11ef-a1bc-1acc26a6847d for case mysql-csi-e2e
2024/06/26 13:57:41 restore phase: InProgress
2024/06/26 13:57:51 restore phase: InProgress
2024/06/26 13:58:01 restore phase: Completed
2024/06/26 13:58:02 Running dc-post-restore.sh script.
2024/06/26 13:58:02 Running post restore script for mysql-csi-e2e-54b8a502-33e5-11ef-a1bc-1acc26a6847d
2024/06/26 13:58:03 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mysql-csi-e2e-54b8a502-33e5-11ef-a1bc-1acc26a6847d
2024/06/26 13:58:03 stdout:
restore: mysql-csi-e2e-54b8a502-33e5-11ef-a1bc-1acc26a6847d
label:   mysql-csi-e2e-54b8a502-33e5-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 13:58:03 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 13:58:03 err: <nil>
2024/06/26 13:58:03 Pod mysql-74f94448f9-s7mpx not yet succeeded: phase is Pending
2024/06/26 13:58:08 Pod mysql-74f94448f9-s7mpx not yet succeeded: phase is Pending
2024/06/26 13:58:13 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:58:18 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:58:23 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:58:28 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:58:33 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:58:38 Pod mysql-74f94448f9-s7mpx not yet succeeded: condition is false: ContainersReady
2024/06/26 13:58:43 Pod todolist-1-zf7hw not yet succeeded: phase is Pending
2024/06/26 13:58:48 Running post-restore function for case mysql-csi-e2e
2024/06/26 13:58:48 checking for the NAMESPACE: mysql-persistent
2024/06/26 13:58:48 Verifying backup/restore data of todolist
2024/06/26 13:58:48 PrebackState: false
2024/06/26 13:58:49 Data came from backup-file
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 13:56:44.484831 -0400 EDT m=+173.506445626","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]

2024/06/26 13:58:49 Data from the response after restore
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 13:56:44.484831 -0400 EDT m=+173.506445626","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]
2024/06/26 13:58:49 Post backup and restore state:  passed
2024/06/26 13:58:49 FLAKE DETECTION: No known flakes found.
2024/06/26 13:58:49 Deleting VolumeSnapshot for CSI backuprestore of mysql-csi-e2e
•
2024/06/26 13:59:04 Waiting for velero pod to be running
2024/06/26 13:59:04 velero pods not found
2024/06/26 13:59:09 pod: velero-68c4958ddc-tlc92 is not yet running: phase is Pending
2024/06/26 13:59:14 velero pods are running
2024/06/26 13:59:14 Creating VolumeSnapshotClass for CSI backuprestore of mongo-csi-e2e
2024/06/26 13:59:14 Installing application for case mongo-csi-e2e
2024/06/26 13:59:14 Creating pvc for case mongo-csi-e2e
2024/06/26 13:59:14 Pod mongo-7c674cb76c-tf22x not yet succeeded: phase is Pending
2024/06/26 13:59:19 Pod mongo-7c674cb76c-tf22x not yet succeeded: phase is Pending
2024/06/26 13:59:25 Pod mongo-7c674cb76c-tf22x not yet succeeded: phase is Pending
2024/06/26 13:59:30 Pod mongo-7c674cb76c-tf22x not yet succeeded: phase is Pending
2024/06/26 13:59:35 Pod mongo-7c674cb76c-tf22x not yet succeeded: phase is Pending
2024/06/26 13:59:40 Pod mongo-7c674cb76c-tf22x not yet succeeded: phase is Pending
2024/06/26 13:59:45 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 13:59:50 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 13:59:55 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 14:00:00 Pod todolist-1-wmtw4 not yet succeeded: phase is Pending
2024/06/26 14:00:05 Running pre-backup function for case mongo-csi-e2e
2024/06/26 14:00:05 Verifying backup/restore data of todolist
2024/06/26 14:00:05 PrebackState: true
2024/06/26 14:00:06 Data before the curl request:
 [{"Id":"667c5724da6662182ccae3e2","Description":"time","Completed":false}]
2024/06/26 14:00:06 Writing data to backupFile (backup-data.txt):
 [{"Id":"667c5724da6662182ccae3e2","Description":"time","Completed":false},{"Id":"667c5726da6662182ccae3e3","Description":"2024-06-26 14:00:06.710702 -0400 EDT m=+375.731778084","Completed":false},{"Id":"667c5726da6662182ccae3e4","Description":"Wednesday","Completed":false}]
2024/06/26 14:00:06 Sleeping for 0s to allow application to be ready for case mongo-csi-e2e
2024/06/26 14:00:06 Creating backup mongo-csi-e2e-cc9898e8-33e5-11ef-a1bc-1acc26a6847d for case mongo-csi-e2e
2024/06/26 14:00:17 backup phase: InProgress
2024/06/26 14:00:27 backup phase: WaitingForPluginOperations
2024/06/26 14:00:37 backup phase: WaitingForPluginOperations
2024/06/26 14:00:47 backup phase: Completed
2024/06/26 14:00:47 Backup for case mongo-csi-e2e succeeded
2024/06/26 14:00:47 waiting for volume snapshot contents snapcontent-392007ac-e429-4728-bb11-d2bae8cf6421 to be ready
2024/06/26 14:00:47 Uninstalling app for case mongo-csi-e2e
2024/06/26 14:01:03 Creating restore mongo-csi-e2e-cc9898f2-33e5-11ef-a1bc-1acc26a6847d for case mongo-csi-e2e
2024/06/26 14:01:03 restore phase: InProgress
2024/06/26 14:01:13 restore phase: InProgress
2024/06/26 14:01:23 restore phase: InProgress
2024/06/26 14:01:33 restore phase: Completed
2024/06/26 14:01:34 Running dc-post-restore.sh script.
2024/06/26 14:01:34 Running post restore script for mongo-csi-e2e-cc9898f2-33e5-11ef-a1bc-1acc26a6847d
2024/06/26 14:01:35 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mongo-csi-e2e-cc9898f2-33e5-11ef-a1bc-1acc26a6847d
2024/06/26 14:01:35 stdout:
restore: mongo-csi-e2e-cc9898f2-33e5-11ef-a1bc-1acc26a6847d
label:   mongo-csi-e2e-cc9898f2-33e5-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 14:01:35 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:01:35 err: <nil>
2024/06/26 14:01:35 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 14:01:40 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 14:01:45 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 14:01:50 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 14:01:55 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 14:02:00 Pod mongo-7c674cb76c-tf22x not yet succeeded: condition is false: ContainersReady
2024/06/26 14:02:05 Pod todolist-1-5swdz not yet succeeded: phase is Pending
2024/06/26 14:02:10 Running post-restore function for case mongo-csi-e2e
2024/06/26 14:02:10 Verifying backup/restore data of todolist
2024/06/26 14:02:10 PrebackState: false
2024/06/26 14:02:10 Data came from backup-file
 [{"Id":"667c5724da6662182ccae3e2","Description":"time","Completed":false},{"Id":"667c5726da6662182ccae3e3","Description":"2024-06-26 14:00:06.710702 -0400 EDT m=+375.731778084","Completed":false},{"Id":"667c5726da6662182ccae3e4","Description":"Wednesday","Completed":false}]

2024/06/26 14:02:10 Data from the response after restore
 [{"Id":"667c5724da6662182ccae3e2","Description":"time","Completed":false},{"Id":"667c5726da6662182ccae3e3","Description":"2024-06-26 14:00:06.710702 -0400 EDT m=+375.731778084","Completed":false},{"Id":"667c5726da6662182ccae3e4","Description":"Wednesday","Completed":false}]
2024/06/26 14:02:10 Post backup and restore state:  passed
2024/06/26 14:02:10 FLAKE DETECTION: No known flakes found.
2024/06/26 14:02:10 Deleting VolumeSnapshot for CSI backuprestore of mongo-csi-e2e
•
2024/06/26 14:02:20 Waiting for velero pod to be running
2024/06/26 14:02:20 velero pods not found
2024/06/26 14:02:26 pod: velero-68c4958ddc-l995d is not yet running: phase is Pending
2024/06/26 14:02:31 velero pods are running
2024/06/26 14:02:31 Creating VolumeSnapshotClass for CSI backuprestore of mysql-twovol-csi-e2e
2024/06/26 14:02:31 Installing application for case mysql-twovol-csi-e2e
diff found for key: allowHostPorts
diff found for key: groups
diff found for key: metadata
diff found for key: allowHostIPC
diff found for key: readOnlyRootFilesystem
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostPID
diff found for key: volumes
diff found for key: allowHostNetwork
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mysql-persistent-scc
2024/06/26 14:02:31 Creating pvc for case mysql-twovol-csi-e2e
2024/06/26 14:02:31 Pod mysql-655d68c56-9j9fc not yet succeeded: phase is Pending
2024/06/26 14:02:36 Pod mysql-655d68c56-9j9fc not yet succeeded: phase is Pending
2024/06/26 14:02:41 Pod mysql-655d68c56-9j9fc not yet succeeded: phase is Pending
2024/06/26 14:02:46 Pod mysql-655d68c56-9j9fc not yet succeeded: phase is Pending
2024/06/26 14:02:51 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:02:56 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:03:01 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:03:06 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:03:12 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:03:17 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:03:22 Running pre-backup function for case mysql-twovol-csi-e2e
2024/06/26 14:03:22 checking for the NAMESPACE: mysql-persistent
2024/06/26 14:03:22 Verifying backup/restore data of todolist
2024/06/26 14:03:22 PrebackState: true
2024/06/26 14:03:22 Data before the curl request:
 [{"Id":1,"Description":"time to make the donuts","Completed":false}]
2024/06/26 14:03:22 Writing data to backupFile (backup-data.txt):
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 14:03:22.33806 -0400 EDT m=+571.358614543","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]
2024/06/26 14:03:22 Writing data to volumeFile (volume-data.txt):
 time="2024-06-26T18:03:19Z" level=info msg="Success: Attached volume and redirected logs to /tmp/log/todoapp/app.log" func=main.main file="/build/todolist.go:179"
time="2024-06-26T18:03:19Z" level=info msg="A running instance of the db: todolist not found, creating" func=main.main file="/build/todolist.go:185"
time="2024-06-26T18:03:19Z" level=info msg="Prepopulate the db" func=main.prepopulate file="/build/todolist.go:152"
time="2024-06-26T18:03:19Z" level=info msg="Starting Todolist API server" func=main.main file="/build/todolist.go:193"
time="2024-06-26T18:03:22Z" level=info msg="Get Incomplete TodoItems" func=main.GetIncompleteItems file="/build/todolist.go:120"
time="2024-06-26T18:03:22Z" level=info msg="Add new TodoItem. Saving to database." func=main.CreateItem file="/build/todolist.go:52" description="2024-06-26 14:03:22.33806 -0400 EDT m=+571.358614543"
time="2024-06-26T18:03:22Z" level=info msg="Add new TodoItem. Saving to database." func=main.CreateItem file="/build/todolist.go:52" description=Wednesday
time="2024-06-26T18:03:22Z" level=info msg="Get Incomplete TodoItems" func=main.GetIncompleteItems file="/build/todolist.go:120"
2024/06/26 14:03:22 Sleeping for 30s to allow application to be ready for case mysql-twovol-csi-e2e
2024/06/26 14:03:52 Creating backup mysql-twovol-csi-e2e-41cb8e4a-33e6-11ef-a1bc-1acc26a6847d for case mysql-twovol-csi-e2e
2024/06/26 14:04:02 backup phase: InProgress
2024/06/26 14:04:12 backup phase: WaitingForPluginOperations
2024/06/26 14:04:22 backup phase: WaitingForPluginOperations
2024/06/26 14:04:32 backup phase: WaitingForPluginOperations
2024/06/26 14:04:42 backup phase: WaitingForPluginOperations
2024/06/26 14:04:52 backup phase: Completed
2024/06/26 14:04:53 Backup for case mysql-twovol-csi-e2e succeeded
2024/06/26 14:04:53 waiting for volume snapshot contents snapcontent-38414a6b-8d3d-4db3-9471-0202a239f6d0 to be ready
2024/06/26 14:04:53 waiting for volume snapshot contents snapcontent-b0f76da2-4514-4e84-a2c5-0a741e04bfc0 to be ready
2024/06/26 14:04:53 Uninstalling app for case mysql-twovol-csi-e2e
2024/06/26 14:05:09 Creating restore mysql-twovol-csi-e2e-41cb8e68-33e6-11ef-a1bc-1acc26a6847d for case mysql-twovol-csi-e2e
2024/06/26 14:05:09 restore phase: InProgress
2024/06/26 14:05:19 restore phase: InProgress
2024/06/26 14:05:29 restore phase: InProgress
2024/06/26 14:05:39 restore phase: Completed
2024/06/26 14:05:40 Running dc-post-restore.sh script.
2024/06/26 14:05:40 Running post restore script for mysql-twovol-csi-e2e-41cb8e68-33e6-11ef-a1bc-1acc26a6847d
2024/06/26 14:06:12 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mysql-twovol-csi-e2e-41cb8e68-33e6-11ef-a1bc-1acc26a6847d
2024/06/26 14:06:12 stdout:
restore: mysql-twovol-csi-e2e-41cb8e68-33e6-11ef-a1bc-1acc26a6847d
label:   mysql-twovol-csi-e2e-41cb8e68-33e6-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
pod "todolist-1-52scf" deleted
Found deployment mysql-persistent/todolist, setting replicas: 1, paused: false
deploymentconfig.apps.openshift.io/todolist patched
2024/06/26 14:06:12 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:06:12 err: <nil>
2024/06/26 14:06:12 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:06:17 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:06:22 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:06:27 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:06:32 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:06:37 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:06:42 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:06:47 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:06:52 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:06:57 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:07:02 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:07:08 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:07:13 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:07:18 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:07:23 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:07:28 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:07:33 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:07:38 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:07:43 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:07:48 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:07:53 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:07:58 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:08:03 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:08:08 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:08:13 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:08:18 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:08:23 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:08:28 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:08:33 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:08:39 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:08:44 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:08:49 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:08:54 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:08:59 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:09:04 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:09:09 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:09:14 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:09:19 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:09:24 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:09:29 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:09:34 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:09:39 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:09:44 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:09:49 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:09:54 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:09:59 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:10:04 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:10:09 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:10:15 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:10:20 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:10:25 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:10:30 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:10:35 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:10:40 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:10:45 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:10:50 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:10:55 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:11:00 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:11:05 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:11:10 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:11:15 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:11:20 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:11:25 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:11:30 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:11:35 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:11:40 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:11:46 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:11:51 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:11:56 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:01 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:06 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:11 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:16 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:21 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:26 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:31 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:36 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:41 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:46 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:51 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:12:56 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:01 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:06 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:11 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:17 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:22 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:27 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:32 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:37 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:13:42 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:13:47 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:13:52 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:13:57 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:14:02 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:14:07 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:14:12 Pod todolist-1-r8qgh not yet succeeded: phase is Pending
2024/06/26 14:14:17 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:22 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:27 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:32 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:37 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:43 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:48 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:53 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:14:58 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:15:03 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:15:08 Pod mysql-655d68c56-9j9fc not yet succeeded: condition is false: ContainersReady
2024/06/26 14:15:12 Post backup and restore state:  failed
2024/06/26 14:15:12 FLAKE DETECTION: No known flakes found.
2024/06/26 14:15:12 Printing events for namespace:  mysql-persistent
2024/06/26 14:15:12 Printing events for namespace:  openshift-adp
2024/06/26 14:15:12 Deleting VolumeSnapshot for CSI backuprestore of mysql-twovol-csi-e2e
2024/06/26 14:15:53 Post backup and restore state:  failed
2024/06/26 14:15:53 FLAKE DETECTION: No known flakes found.
2024/06/26 14:15:53 Printing events for namespace:  mysql-persistent
2024/06/26 14:15:53 Printing events for namespace:  openshift-adp
2024/06/26 14:15:53 Deleting VolumeSnapshot for CSI backuprestore of mysql-twovol-csi-e2e
2024/06/26 14:15:53 Post backup and restore state:  failed
2024/06/26 14:15:53 FLAKE DETECTION: No known flakes found.
2024/06/26 14:15:53 Printing events for namespace:  mysql-persistent
2024/06/26 14:15:53 Printing events for namespace:  openshift-adp
2024/06/26 14:15:53 Deleting VolumeSnapshot for CSI backuprestore of mysql-twovol-csi-e2e

------------------------------
• [FAILED] [813.077 seconds]
Backup and restore tests Backup and restore applications [It] MySQL application two Vol CSI
/Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:307

  Timeline >>
    any(
  - 	bool(false),
    )

    any(
  - 	[]any{},
    )

    map[string]any{
    	"creationTimestamp": string("2024-06-26T17:57:59Z"),
    	"generation":        int64(1),
    	"labels": map[string]any{
    		"e2e-app":                string("true"),
  - 		"velero.io/backup-name":  string("mysql-csi-e2e-54b89fda-33e5-11ef-a1bc-1acc26a6847d"),
  - 		"velero.io/restore-name": string("mysql-csi-e2e-54b8a502-33e5-11ef-a1bc-1acc26a6847d"),
    	},
    	"managedFields": []any{map[string]any{"apiVersion": string("security.openshift.io/v1"), "fieldsType": string("FieldsV1"), "fieldsV1": map[string]any{"f:allowPrivilegeEscalation": map[string]any{}, "f:allowPrivilegedContainer": map[string]any{}, "f:fsGroup": map[string]any{".": map[string]any{}, "f:type": map[string]any{}}, "f:metadata": map[string]any{"f:labels": map[string]any{".": map[string]any{}, "f:e2e-app": map[string]any{}}}, ...}, "manager": string("__debug_bin2136896783"), ...}},
    	"name":          string("mysql-persistent-scc"),
    	... // 2 identical entries
    }

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    []any{
  - 	string("awsElasticBlockStore"),
  + 	string("*"),
  - 	string("azureDisk"),
  - 	string("azureFile"),
  - 	string("cephFS"),
  - 	string("cinder"),
  - 	string("configMap"),
  - 	string("csi"),
  - 	string("downwardAPI"),
  - 	string("emptyDir"),
  - 	string("ephemeral"),
  - 	string("fc"),
  - 	string("flexVolume"),
  - 	string("flocker"),
  - 	string("gcePersistentDisk"),
  - 	string("gitRepo"),
  - 	string("glusterfs"),
  - 	string("iscsi"),
  - 	string("nfs"),
  - 	string("persistentVolumeClaim"),
  - 	string("photonPersistentDisk"),
  - 	string("portworxVolume"),
  - 	string("projected"),
  - 	string("quobyte"),
  - 	string("rbd"),
  - 	string("scaleIO"),
  - 	string("secret"),
  - 	string("storageOS"),
  - 	string("vsphere"),
    }

    any(
  - 	bool(false),
    )

  Name:         mysql-twovol-csi-e2e-41cb8e4a-33e6-11ef-a1bc-1acc26a6847d
  Namespace:    openshift-adp
  Labels:       velero.io/storage-location=ts-example-velero-1
  Annotations:  velero.io/resource-timeout=10m0s
                velero.io/source-cluster-k8s-gitversion=v1.28.7+f1b5f6c
                velero.io/source-cluster-k8s-major-version=1
                velero.io/source-cluster-k8s-minor-version=28

  Phase:  Completed

  Namespaces:
    Included:  mysql-persistent
    Excluded:  <none>

  Resources:
    Included:        *
    Excluded:        <none>
    Cluster-scoped:  auto

  Label selector:  <none>

  Or label selector:  <none>

  Storage Location:  ts-example-velero-1

  Velero-Native Snapshot PVs:  auto
  Snapshot Move Data:          false
  Data Mover:                  velero

  TTL:  720h0m0s

  CSISnapshotTimeout:    10m0s
  ItemOperationTimeout:  4h0m0s

  Hooks:  <none>

  Backup Format Version:  1.1.0

  Started:    2024-06-26 14:03:52 -0400 EDT
  Completed:  2024-06-26 14:04:50 -0400 EDT

  Expiration:  2024-07-26 14:03:52 -0400 EDT

  Total items to be backed up:  92
  Items backed up:              92

  Backup Item Operations:
    Operation for volumesnapshots.snapshot.storage.k8s.io mysql-persistent/velero-mysql-kcvrb:
      Backup Item Action Plugin:  velero.io/csi-volumesnapshot-backupper
      Operation ID:               mysql-persistent/velero-mysql-kcvrb/2024-06-26T18:04:03Z
      Items to Update:
                volumesnapshots.snapshot.storage.k8s.io mysql-persistent/velero-mysql-kcvrb
      Phase:    Completed
      Created:  2024-06-26 14:04:03 -0400 EDT
      Started:  2024-06-26 14:04:03 -0400 EDT
    Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-b0f76da2-4514-4e84-a2c5-0a741e04bfc0:
      Backup Item Action Plugin:  velero.io/csi-volumesnapshotcontent-backupper
      Operation ID:               snapcontent-b0f76da2-4514-4e84-a2c5-0a741e04bfc0/2024-06-26T18:04:03Z
      Items to Update:
                volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-b0f76da2-4514-4e84-a2c5-0a741e04bfc0
      Phase:    Completed
      Created:  2024-06-26 14:04:03 -0400 EDT
      Started:  2024-06-26 14:04:03 -0400 EDT
    Operation for volumesnapshots.snapshot.storage.k8s.io mysql-persistent/velero-applog-br8n8:
      Backup Item Action Plugin:  velero.io/csi-volumesnapshot-backupper
      Operation ID:               mysql-persistent/velero-applog-br8n8/2024-06-26T18:04:08Z
      Items to Update:
                volumesnapshots.snapshot.storage.k8s.io mysql-persistent/velero-applog-br8n8
      Phase:    Completed
      Created:  2024-06-26 14:04:08 -0400 EDT
      Started:  2024-06-26 14:04:08 -0400 EDT
    Operation for volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-38414a6b-8d3d-4db3-9471-0202a239f6d0:
      Backup Item Action Plugin:  velero.io/csi-volumesnapshotcontent-backupper
      Operation ID:               snapcontent-38414a6b-8d3d-4db3-9471-0202a239f6d0/2024-06-26T18:04:08Z
      Items to Update:
                volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-38414a6b-8d3d-4db3-9471-0202a239f6d0
      Phase:    Completed
      Created:  2024-06-26 14:04:08 -0400 EDT
      Started:  2024-06-26 14:04:08 -0400 EDT
  Resource List:
    apiextensions.k8s.io/v1/CustomResourceDefinition:
      - clusterserviceversions.operators.coreos.com
    apps.openshift.io/v1/DeploymentConfig:
      - mysql-persistent/todolist
    apps/v1/Deployment:
      - mysql-persistent/mysql
    apps/v1/ReplicaSet:
      - mysql-persistent/mysql-655d68c56
    authorization.openshift.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin
      - mysql-persistent/alert-routing-edit-dedicated-admins
      - mysql-persistent/dedicated-admins-project-dedicated-admins
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin
      - mysql-persistent/system:deployers
      - mysql-persistent/system:image-builders
      - mysql-persistent/system:image-pullers
    discovery.k8s.io/v1/EndpointSlice:
      - mysql-persistent/mysql-m76n2
      - mysql-persistent/todolist-9plfz
    operators.coreos.com/v1alpha1/ClusterServiceVersion:
      - mysql-persistent/observability-operator.v0.0.28
      - mysql-persistent/route-monitor-operator.v0.1.648-22d7745
    rbac.authorization.k8s.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin
      - mysql-persistent/alert-routing-edit-dedicated-admins
      - mysql-persistent/dedicated-admins-project-dedicated-admins
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin
      - mysql-persistent/system:deployers
      - mysql-persistent/system:image-builders
      - mysql-persistent/system:image-pullers
    route.openshift.io/v1/Route:
      - mysql-persistent/todolist-route
    security.openshift.io/v1/SecurityContextConstraints:
      - mysql-persistent-scc
    snapshot.storage.k8s.io/v1/VolumeSnapshot:
      - mysql-persistent/velero-applog-br8n8
      - mysql-persistent/velero-mysql-kcvrb
    snapshot.storage.k8s.io/v1/VolumeSnapshotClass:
      - oadp-example-snapclass
    snapshot.storage.k8s.io/v1/VolumeSnapshotContent:
      - snapcontent-38414a6b-8d3d-4db3-9471-0202a239f6d0
      - snapcontent-b0f76da2-4514-4e84-a2c5-0a741e04bfc0
    v1/ConfigMap:
      - mysql-persistent/kube-root-ca.crt
      - mysql-persistent/openshift-service-ca.crt
    v1/Endpoints:
      - mysql-persistent/mysql
      - mysql-persistent/todolist
    v1/Event:
      - mysql-persistent/applog.17dc9fe2334940a0
      - mysql-persistent/applog.17dc9fe25bd98de8
      - mysql-persistent/applog.17dc9fe25be0413b
      - mysql-persistent/applog.17dc9fe364b9df8d
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe229deb6a8
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe3c16e3ff4
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe448ebbda6
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe5ab44645b
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe5acbf7234
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe5c646d1c5
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe5cb896a92
      - mysql-persistent/mysql-655d68c56-9j9fc.17dc9fe5ccb5db4d
      - mysql-persistent/mysql-655d68c56.17dc9fe229d219c4
      - mysql-persistent/mysql.17dc9fe228d79b2c
      - mysql-persistent/mysql.17dc9fe2317737a6
      - mysql-persistent/mysql.17dc9fe29775a2f0
      - mysql-persistent/mysql.17dc9fe2977648cb
      - mysql-persistent/mysql.17dc9fe39f107b60
      - mysql-persistent/todolist-1-52scf.17dc9fe385eb30fc
      - mysql-persistent/todolist-1-52scf.17dc9fe41136a40f
      - mysql-persistent/todolist-1-52scf.17dc9fe4f25c9c7b
      - mysql-persistent/todolist-1-52scf.17dc9fe4f3ca2925
      - mysql-persistent/todolist-1-52scf.17dc9fe4f9d69189
      - mysql-persistent/todolist-1-52scf.17dc9fe4fade62f0
      - mysql-persistent/todolist-1-52scf.17dc9fed4c075b43
      - mysql-persistent/todolist-1-52scf.17dc9fed51c853d1
      - mysql-persistent/todolist-1-52scf.17dc9fed52e3ef48
      - mysql-persistent/todolist-1-deploy.17dc9fe233b9c987
      - mysql-persistent/todolist-1-deploy.17dc9fe24c0b3bd2
      - mysql-persistent/todolist-1-deploy.17dc9fe24d618d94
      - mysql-persistent/todolist-1-deploy.17dc9fe252e838f4
      - mysql-persistent/todolist-1-deploy.17dc9fe25404bccc
      - mysql-persistent/todolist-1.17dc9fe25afdc020
      - mysql-persistent/todolist.17dc9fe2309cdfe5
    v1/Namespace:
      - mysql-persistent
    v1/PersistentVolume:
      - pvc-5713067d-af82-45aa-b0a8-0dc2cad2bdda
      - pvc-62f11222-7725-47fd-803b-28ab642c86f5
    v1/PersistentVolumeClaim:
      - mysql-persistent/applog
      - mysql-persistent/mysql
    v1/Pod:
      - mysql-persistent/mysql-655d68c56-9j9fc
      - mysql-persistent/todolist-1-52scf
      - mysql-persistent/todolist-1-deploy
    v1/ReplicationController:
      - mysql-persistent/todolist-1
    v1/Secret:
      - mysql-persistent/builder-dockercfg-mbrxn
      - mysql-persistent/builder-token-tdnm5
      - mysql-persistent/default-dockercfg-6xlv4
      - mysql-persistent/default-token-jsf9n
      - mysql-persistent/deployer-dockercfg-dfhbs
      - mysql-persistent/deployer-token-txv67
      - mysql-persistent/mysql-persistent-sa-dockercfg-mv4n8
      - mysql-persistent/mysql-persistent-sa-token-t8l7c
    v1/Service:
      - mysql-persistent/mysql
      - mysql-persistent/todolist
    v1/ServiceAccount:
      - mysql-persistent/builder
      - mysql-persistent/default
      - mysql-persistent/deployer
      - mysql-persistent/mysql-persistent-sa

  Velero-Native Snapshots: <none included>

  Name:         mysql-twovol-csi-e2e-41cb8e68-33e6-11ef-a1bc-1acc26a6847d
  Namespace:    openshift-adp
  Labels:       <none>
  Annotations:  <none>

  Phase:                       Completed
  Total items to be restored:  57
  Items restored:              57

  Started:    2024-06-26 14:05:09 -0400 EDT
  Completed:  2024-06-26 14:05:30 -0400 EDT

  Warnings:
    Velero:     <none>
    Cluster:  could not restore, CustomResourceDefinition "clusterserviceversions.operators.coreos.com" already exists. Warning: the in-cluster version is different than the backed-up version
              could not restore, VolumeSnapshotContent "snapcontent-38414a6b-8d3d-4db3-9471-0202a239f6d0" already exists. Warning: the in-cluster version is different than the backed-up version
              could not restore, VolumeSnapshotContent "snapcontent-b0f76da2-4514-4e84-a2c5-0a741e04bfc0" already exists. Warning: the in-cluster version is different than the backed-up version
    Namespaces:
      mysql-persistent:  could not restore, RoleBinding "dedicated-admins-project-system:serviceaccounts:dedicated-admin" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, RoleBinding "system:image-pullers" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, ConfigMap "openshift-service-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/admin-dedicated-admins: rolebindings.authorization.openshift.io "admin-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/admin-system:serviceaccounts:dedicated-admin: rolebindings.authorization.openshift.io "admin-system:serviceaccounts:dedicated-admin" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/alert-routing-edit-dedicated-admins: rolebindings.authorization.openshift.io "alert-routing-edit-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/dedicated-admins-project-dedicated-admins: rolebindings.authorization.openshift.io "dedicated-admins-project-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin: rolebindings.authorization.openshift.io "dedicated-admins-project-system:serviceaccounts:dedicated-admin" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:deployers: rolebindings.authorization.openshift.io "system:deployers" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:image-builders: rolebindings.authorization.openshift.io "system:image-builders" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:image-pullers: rolebindings.authorization.openshift.io "system:image-pullers" not found

  Backup:  mysql-twovol-csi-e2e-41cb8e4a-33e6-11ef-a1bc-1acc26a6847d

  Namespaces:
    Included:  all namespaces found in the backup
    Excluded:  <none>

  Resources:
    Included:        *
    Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
    Cluster-scoped:  auto

  Namespace mappings:  <none>

  Label selector:  <none>

  Or label selector:  <none>

  Restore PVs:  auto

  Existing Resource Policy:   <none>
  ItemOperationTimeout:       4h0m0s

  Preserve Service NodePorts:  auto

  Resource List:
    apiextensions.k8s.io/v1/CustomResourceDefinition:
      - clusterserviceversions.operators.coreos.com(failed)
    apps.openshift.io/v1/DeploymentConfig:
      - mysql-persistent/todolist(created)
    apps/v1/Deployment:
      - mysql-persistent/mysql(created)
    apps/v1/ReplicaSet:
      - mysql-persistent/mysql-655d68c56(created)
    authorization.openshift.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins(failed)
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/alert-routing-edit-dedicated-admins(failed)
      - mysql-persistent/dedicated-admins-project-dedicated-admins(failed)
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/system:deployers(failed)
      - mysql-persistent/system:image-builders(failed)
      - mysql-persistent/system:image-pullers(failed)
    discovery.k8s.io/v1/EndpointSlice:
      - mysql-persistent/mysql-m76n2(created)
      - mysql-persistent/todolist-9plfz(created)
    operators.coreos.com/v1alpha1/ClusterServiceVersion:
      - mysql-persistent/observability-operator.v0.0.28(created)
      - mysql-persistent/route-monitor-operator.v0.1.648-22d7745(created)
    rbac.authorization.k8s.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins(created)
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin(created)
      - mysql-persistent/alert-routing-edit-dedicated-admins(created)
      - mysql-persistent/dedicated-admins-project-dedicated-admins(created)
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/system:deployers(created)
      - mysql-persistent/system:image-builders(created)
      - mysql-persistent/system:image-pullers(failed)
    route.openshift.io/v1/Route:
      - mysql-persistent/todolist-route(created)
    security.openshift.io/v1/SecurityContextConstraints:
      - mysql-persistent-scc(created)
    snapshot.storage.k8s.io/v1/VolumeSnapshot:
      - mysql-persistent/velero-applog-br8n8(created)
      - mysql-persistent/velero-mysql-kcvrb(created)
    snapshot.storage.k8s.io/v1/VolumeSnapshotClass:
      - oadp-example-snapclass(skipped)
    snapshot.storage.k8s.io/v1/VolumeSnapshotContent:
      - snapcontent-38414a6b-8d3d-4db3-9471-0202a239f6d0(failed)
      - snapcontent-b0f76da2-4514-4e84-a2c5-0a741e04bfc0(failed)
    v1/ConfigMap:
      - mysql-persistent/kube-root-ca.crt(failed)
      - mysql-persistent/openshift-service-ca.crt(failed)
    v1/Endpoints:
      - mysql-persistent/mysql(created)
      - mysql-persistent/todolist(created)
    v1/Namespace:
      - mysql-persistent(created)
    v1/PersistentVolume:
      - pvc-5713067d-af82-45aa-b0a8-0dc2cad2bdda(skipped)
      - pvc-62f11222-7725-47fd-803b-28ab642c86f5(skipped)
    v1/PersistentVolumeClaim:
      - mysql-persistent/applog(created)
      - mysql-persistent/mysql(created)
    v1/Pod:
      - mysql-persistent/mysql-655d68c56-9j9fc(created)
      - mysql-persistent/todolist-1-52scf(created)
    v1/ReplicationController:
      - mysql-persistent/todolist-1(skipped)
    v1/Secret:
      - mysql-persistent/builder-dockercfg-mbrxn(created)
      - mysql-persistent/builder-token-tdnm5(skipped)
      - mysql-persistent/default-dockercfg-6xlv4(created)
      - mysql-persistent/default-token-jsf9n(skipped)
      - mysql-persistent/deployer-dockercfg-dfhbs(created)
      - mysql-persistent/deployer-token-txv67(skipped)
      - mysql-persistent/mysql-persistent-sa-dockercfg-mv4n8(created)
      - mysql-persistent/mysql-persistent-sa-token-t8l7c(skipped)
    v1/Service:
      - mysql-persistent/mysql(created)
      - mysql-persistent/todolist(created)
    v1/ServiceAccount:
      - mysql-persistent/builder(skipped)
      - mysql-persistent/default(skipped)
      - mysql-persistent/deployer(skipped)
      - mysql-persistent/mysql-persistent-sa(created)

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:219 @ 06/26/24 14:15:12.372
  Printing app namespace events
  Event: Snapshot mysql-persistent/velero-applog-br8n8 was successfully created by the CSI driver., Type: Normal, Count: 1, Src: {VolumeSnapshot mysql-persistent velero-applog-br8n8 e48fe493-5b9b-47d5-a2db-81b4225da1ec snapshot.storage.k8s.io/v1 108610982 }, Reason: SnapshotCreated
  Event: Snapshot mysql-persistent/velero-applog-br8n8 is ready to use., Type: Normal, Count: 1, Src: {VolumeSnapshot mysql-persistent velero-applog-br8n8 e48fe493-5b9b-47d5-a2db-81b4225da1ec snapshot.storage.k8s.io/v1 108610982 }, Reason: SnapshotReady
  Event: failed to provision volume with StorageClass "gp2-csi": error getting handle for DataSource Type VolumeSnapshot by Name velero-mysql-kcvrb: snapshot velero-mysql-kcvrb is not Ready, Type: Warning, Count: 1, Src: {PersistentVolumeClaim mysql-persistent mysql 6106b95e-095d-4629-80d1-e5c92e920d00 v1 108611032 }, Reason: ProvisioningFailed
  Event: waiting for first consumer to be created before binding, Type: Normal, Count: 2, Src: {PersistentVolumeClaim mysql-persistent mysql 6106b95e-095d-4629-80d1-e5c92e920d00 v1 108611002 }, Reason: WaitForFirstConsumer
  Event: waiting for first consumer to be created before binding, Type: Normal, Count: 2, Src: {PersistentVolumeClaim mysql-persistent applog 9ce0caa1-3281-4b19-b5ad-28222b1c09d3 v1 108610996 }, Reason: WaitForFirstConsumer
  Event: Snapshot mysql-persistent/velero-mysql-kcvrb was successfully created by the CSI driver., Type: Normal, Count: 1, Src: {VolumeSnapshot mysql-persistent velero-mysql-kcvrb 08f7f221-30ab-4c39-8136-a8efda2544d3 snapshot.storage.k8s.io/v1 108610991 }, Reason: SnapshotCreated
  Event: External provisioner is provisioning volume for claim "mysql-persistent/applog", Type: Normal, Count: 1, Src: {PersistentVolumeClaim mysql-persistent applog 9ce0caa1-3281-4b19-b5ad-28222b1c09d3 v1 108611042 }, Reason: Provisioning
  Event: Snapshot mysql-persistent/velero-mysql-kcvrb is ready to use., Type: Normal, Count: 1, Src: {VolumeSnapshot mysql-persistent velero-mysql-kcvrb 08f7f221-30ab-4c39-8136-a8efda2544d3 snapshot.storage.k8s.io/v1 108610991 }, Reason: SnapshotReady
  Event: Created new replication controller "todolist-1" for version 1, Type: Normal, Count: 1, Src: {DeploymentConfig mysql-persistent todolist 2ac23aa6-9c06-4b42-a0c2-b0878ceadd13 apps.openshift.io/v1 108611078 }, Reason: DeploymentCreated
  Event: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered., Type: Normal, Count: 2, Src: {PersistentVolumeClaim mysql-persistent applog 9ce0caa1-3281-4b19-b5ad-28222b1c09d3 v1 108611042 }, Reason: ExternalProvisioning
  Event: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:108cb092eb74afbdcb5733986d429cf591586242481217960e37c20eaba38582" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 20ed6a9b-0cdc-4f51-8dc6-dad7981ebae1 v1 108611089 spec.containers{deployment}}, Reason: Pulled
  Event: Created container deployment, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 20ed6a9b-0cdc-4f51-8dc6-dad7981ebae1 v1 108611089 spec.containers{deployment}}, Reason: Created
  Event: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered., Type: Normal, Count: 3, Src: {PersistentVolumeClaim mysql-persistent mysql 6106b95e-095d-4629-80d1-e5c92e920d00 v1 108611032 }, Reason: ExternalProvisioning
  Event: External provisioner is provisioning volume for claim "mysql-persistent/mysql", Type: Normal, Count: 2, Src: {PersistentVolumeClaim mysql-persistent mysql 6106b95e-095d-4629-80d1-e5c92e920d00 v1 108611032 }, Reason: Provisioning
  Event: Started container deployment, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 20ed6a9b-0cdc-4f51-8dc6-dad7981ebae1 v1 108611089 spec.containers{deployment}}, Reason: Started
  Event: Add eth0 [10.131.0.199/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 20ed6a9b-0cdc-4f51-8dc6-dad7981ebae1 v1 108611105 }, Reason: AddedInterface
  Event: Successfully provisioned volume pvc-9ce0caa1-3281-4b19-b5ad-28222b1c09d3, Type: Normal, Count: 1, Src: {PersistentVolumeClaim mysql-persistent applog 9ce0caa1-3281-4b19-b5ad-28222b1c09d3 v1 108611042 }, Reason: ProvisioningSucceeded
  Event: Successfully provisioned volume pvc-6106b95e-095d-4629-80d1-e5c92e920d00, Type: Normal, Count: 1, Src: {PersistentVolumeClaim mysql-persistent mysql 6106b95e-095d-4629-80d1-e5c92e920d00 v1 108611032 }, Reason: ProvisioningSucceeded
  Event: AttachVolume.Attach succeeded for volume "pvc-9ce0caa1-3281-4b19-b5ad-28222b1c09d3" , Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-52scf ef3a8293-5098-4e81-addb-5b505984d1db v1 108611190 }, Reason: SuccessfulAttachVolume
  Event: AttachVolume.Attach succeeded for volume "pvc-6106b95e-095d-4629-80d1-e5c92e920d00" , Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 }, Reason: SuccessfulAttachVolume
  Event: Created container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-52scf ef3a8293-5098-4e81-addb-5b505984d1db v1 108611190 spec.initContainers{init-myservice}}, Reason: Created
  Event: Started container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-52scf ef3a8293-5098-4e81-addb-5b505984d1db v1 108611190 spec.initContainers{init-myservice}}, Reason: Started
  Event: Container image "docker.io/curlimages/curl:8.5.0" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-52scf ef3a8293-5098-4e81-addb-5b505984d1db v1 108611190 spec.initContainers{init-myservice}}, Reason: Pulled
  Event: Add eth0 [10.131.0.198/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-52scf ef3a8293-5098-4e81-addb-5b505984d1db v1 108611199 }, Reason: AddedInterface
  Event: Add eth0 [10.128.2.20/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611186 }, Reason: AddedInterface
  Event: Stopping container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-52scf ef3a8293-5098-4e81-addb-5b505984d1db v1 108611190 spec.initContainers{init-myservice}}, Reason: Killing
  Event: Successfully pulled image "registry.redhat.io/rhel8/mariadb-105:latest" in 10.16s (10.16s including waiting), Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Pulled
  Event: Created pod: todolist-1-r8qgh, Type: Normal, Count: 1, Src: {ReplicationController mysql-persistent todolist-1 cc4f69e9-1c69-479e-be56-957216609da5 v1 108611705 }, Reason: SuccessfulCreate
  Event: Scaled replication controller "todolist-1" from 0 to 1, Type: Normal, Count: 1, Src: {DeploymentConfig mysql-persistent todolist 2ac23aa6-9c06-4b42-a0c2-b0878ceadd13 apps.openshift.io/v1 108611704 }, Reason: ReplicationControllerScaled
  Event: Add eth0 [10.131.0.200/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-r8qgh 408a39f0-1583-46b0-b452-89504d7b1b92 v1 108611716 }, Reason: AddedInterface
  Event: Container image "docker.io/curlimages/curl:8.5.0" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-r8qgh 408a39f0-1583-46b0-b452-89504d7b1b92 v1 108611711 spec.initContainers{init-myservice}}, Reason: Pulled
  Event: Created container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-r8qgh 408a39f0-1583-46b0-b452-89504d7b1b92 v1 108611711 spec.initContainers{init-myservice}}, Reason: Created
  Event: Started container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-r8qgh 408a39f0-1583-46b0-b452-89504d7b1b92 v1 108611711 spec.initContainers{init-myservice}}, Reason: Started
  Event: Successfully pulled image "registry.redhat.io/rhel8/mariadb-105:latest" in 722ms (722ms including waiting), Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Pulled
  Event: Successfully pulled image "registry.redhat.io/rhel8/mariadb-105:latest" in 744ms (744ms including waiting), Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Pulled
  Event: Started container mysql, Type: Normal, Count: 3, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Started
  Event: Created container mysql, Type: Normal, Count: 3, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Created
  Event: Container mysql failed liveness probe, will be restarted, Type: Normal, Count: 3, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Killing
  Event: Pulling image "registry.redhat.io/rhel8/mariadb-105:latest", Type: Normal, Count: 4, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Pulling
  Event: Liveness probe failed: dial tcp 10.128.2.20:3306: i/o timeout, Type: Warning, Count: 15, Src: {Pod mysql-persistent mysql-655d68c56-9j9fc 6325a876-2e8e-4f5e-8e81-2e14c9531bb3 v1 108611185 spec.containers{mysql}}, Reason: Unhealthy
  Printing oadp namespace events

  Attempt #1 Failed.  Retrying ↺ @ 06/26/24 14:15:53.286

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:15:53.287
  Printing app namespace events
  Printing oadp namespace events
  Event: Stopping container velero, Type: Normal, Count: 1, Src: {Pod openshift-adp velero-68c4958ddc-l995d bc2ffa9d-d2a2-41a0-ba7d-1a0b50c7dca3 v1 108607634 spec.containers{velero}}, Reason: Killing

  Attempt #2 Failed.  Retrying ↺ @ 06/26/24 14:15:53.646

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:15:53.647
  Printing app namespace events
  Printing oadp namespace events
  Event: Stopping container velero, Type: Normal, Count: 1, Src: {Pod openshift-adp velero-68c4958ddc-l995d bc2ffa9d-d2a2-41a0-ba7d-1a0b50c7dca3 v1 108607634 spec.containers{velero}}, Reason: Killing
  << Timeline

  [FAILED] No known FLAKE found in a previous run, marking test as failed.
  In [It] at: /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:15:53.647

  There were additional failures detected.  To view them in detail run ginkgo -vv
------------------------------
2024/06/26 14:15:54 Waiting for velero pod to be running
2024/06/26 14:15:54 velero pods not found
2024/06/26 14:15:59 velero pods are running
2024/06/26 14:15:59 Waiting for Node Agent pods to be running
2024/06/26 14:15:59 Checking for correct number of running Node Agent pods...
2024/06/26 14:15:59 Installing application for case mongo-restic-e2e
diff found for key: metadata
diff found for key: groups
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostIPC
diff found for key: allowHostPID
diff found for key: readOnlyRootFilesystem
diff found for key: allowHostNetwork
diff found for key: volumes
diff found for key: allowHostPorts
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mongo-persistent-scc
2024/06/26 14:15:59 Pod mongo-7c674cb76c-z2966 not yet succeeded: phase is Pending
2024/06/26 14:16:04 Pod mongo-7c674cb76c-z2966 not yet succeeded: phase is Pending
2024/06/26 14:16:09 Pod mongo-7c674cb76c-z2966 not yet succeeded: phase is Pending
2024/06/26 14:16:14 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:16:19 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:16:24 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:16:29 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:16:35 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:16:40 Pod todolist-1-v8zrn not yet succeeded: phase is Pending
2024/06/26 14:16:45 Running pre-backup function for case mongo-restic-e2e
2024/06/26 14:16:45 Verifying backup/restore data of todolist
2024/06/26 14:16:45 PrebackState: true
2024/06/26 14:16:45 Data before the curl request:
 [{"Id":"667c5b093ebe0e57769bf303","Description":"time","Completed":false}]
2024/06/26 14:16:45 Writing data to backupFile (backup-data.txt):
 [{"Id":"667c5b093ebe0e57769bf303","Description":"time","Completed":false},{"Id":"667c5b0d3ebe0e57769bf304","Description":"2024-06-26 14:16:45.382604 -0400 EDT m=+1374.374750334","Completed":false},{"Id":"667c5b0d3ebe0e57769bf305","Description":"Wednesday","Completed":false}]
2024/06/26 14:16:45 Sleeping for 0s to allow application to be ready for case mongo-restic-e2e
2024/06/26 14:16:45 Creating backup mongo-restic-e2e-237252a6-33e8-11ef-a1bc-1acc26a6847d for case mongo-restic-e2e
2024/06/26 14:16:55 backup phase: InProgress
2024/06/26 14:17:05 backup phase: Completed
2024/06/26 14:17:06 Backup for case mongo-restic-e2e succeeded
2024/06/26 14:17:06 Uninstalling app for case mongo-restic-e2e
2024/06/26 14:17:16 Creating restore mongo-restic-e2e-237252ba-33e8-11ef-a1bc-1acc26a6847d for case mongo-restic-e2e
2024/06/26 14:17:16 restore phase: InProgress
2024/06/26 14:17:26 restore phase: InProgress
2024/06/26 14:17:36 restore phase: InProgress
2024/06/26 14:17:46 restore phase: InProgress
2024/06/26 14:17:57 restore phase: Completed
2024/06/26 14:17:57 Running dc-post-restore.sh script.
2024/06/26 14:17:57 Running post restore script for mongo-restic-e2e-237252ba-33e8-11ef-a1bc-1acc26a6847d
2024/06/26 14:17:58 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mongo-restic-e2e-237252ba-33e8-11ef-a1bc-1acc26a6847d
2024/06/26 14:17:58 stdout:
restore: mongo-restic-e2e-237252ba-33e8-11ef-a1bc-1acc26a6847d
label:   mongo-restic-e2e-237252ba-33e8-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 14:17:58 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:17:58 err: <nil>
2024/06/26 14:17:59 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:18:04 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:18:09 Pod mongo-7c674cb76c-z2966 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:18:14 Pod todolist-1-tdb7d not yet succeeded: phase is Pending
2024/06/26 14:18:19 Pod todolist-1-tdb7d not yet succeeded: phase is Pending
2024/06/26 14:18:24 Running post-restore function for case mongo-restic-e2e
2024/06/26 14:18:24 Verifying backup/restore data of todolist
2024/06/26 14:18:24 PrebackState: false
2024/06/26 14:18:24 Data came from backup-file
 [{"Id":"667c5b093ebe0e57769bf303","Description":"time","Completed":false},{"Id":"667c5b0d3ebe0e57769bf304","Description":"2024-06-26 14:16:45.382604 -0400 EDT m=+1374.374750334","Completed":false},{"Id":"667c5b0d3ebe0e57769bf305","Description":"Wednesday","Completed":false}]

2024/06/26 14:18:24 Data from the response after restore
 [{"Id":"667c5b093ebe0e57769bf303","Description":"time","Completed":false},{"Id":"667c5b0d3ebe0e57769bf304","Description":"2024-06-26 14:16:45.382604 -0400 EDT m=+1374.374750334","Completed":false},{"Id":"667c5b0d3ebe0e57769bf305","Description":"Wednesday","Completed":false}]
2024/06/26 14:18:24 Post backup and restore state:  passed
2024/06/26 14:18:24 FLAKE DETECTION: No known flakes found.
•
2024/06/26 14:18:39 Waiting for velero pod to be running
2024/06/26 14:18:39 velero pods not found
2024/06/26 14:18:44 velero pods are running
2024/06/26 14:18:44 Waiting for Node Agent pods to be running
2024/06/26 14:18:44 Checking for correct number of running Node Agent pods...
2024/06/26 14:18:44 Installing application for case mysql-restic-e2e
diff found for key: metadata
diff found for key: readOnlyRootFilesystem
diff found for key: allowHostPorts
diff found for key: allowHostPID
diff found for key: groups
diff found for key: allowHostIPC
diff found for key: allowHostNetwork
diff found for key: volumes
diff found for key: allowHostDirVolumePlugin
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mysql-persistent-scc
2024/06/26 14:18:45 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: phase is Pending
2024/06/26 14:18:50 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: phase is Pending
2024/06/26 14:18:55 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:19:00 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:19:05 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:19:10 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:19:15 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:19:20 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:19:25 Pod todolist-1-p87tl not yet succeeded: phase is Pending
2024/06/26 14:19:30 Running pre-backup function for case mysql-restic-e2e
2024/06/26 14:19:30 checking for the NAMESPACE: mysql-persistent
2024/06/26 14:19:30 Verifying backup/restore data of todolist
2024/06/26 14:19:30 PrebackState: true
2024/06/26 14:19:31 Data before the curl request:
 [{"Id":1,"Description":"time to make the donuts","Completed":false}]
2024/06/26 14:19:31 Writing data to backupFile (backup-data.txt):
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 14:19:31.007715 -0400 EDT m=+1539.998376459","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]
2024/06/26 14:19:31 Sleeping for 0s to allow application to be ready for case mysql-restic-e2e
2024/06/26 14:19:31 Creating backup mysql-restic-e2e-86342ffe-33e8-11ef-a1bc-1acc26a6847d for case mysql-restic-e2e
2024/06/26 14:19:41 backup phase: InProgress
2024/06/26 14:19:51 backup phase: Completed
2024/06/26 14:19:54 Backup for case mysql-restic-e2e succeeded
2024/06/26 14:19:54 Uninstalling app for case mysql-restic-e2e
2024/06/26 14:20:05 Creating restore mysql-restic-e2e-86343012-33e8-11ef-a1bc-1acc26a6847d for case mysql-restic-e2e
2024/06/26 14:20:05 restore phase: InProgress
2024/06/26 14:20:15 restore phase: InProgress
2024/06/26 14:20:25 restore phase: InProgress
2024/06/26 14:20:35 restore phase: InProgress
2024/06/26 14:20:45 restore phase: Completed
2024/06/26 14:20:46 Running dc-post-restore.sh script.
2024/06/26 14:20:46 Running post restore script for mysql-restic-e2e-86343012-33e8-11ef-a1bc-1acc26a6847d
2024/06/26 14:20:47 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mysql-restic-e2e-86343012-33e8-11ef-a1bc-1acc26a6847d
2024/06/26 14:20:47 stdout:
restore: mysql-restic-e2e-86343012-33e8-11ef-a1bc-1acc26a6847d
label:   mysql-restic-e2e-86343012-33e8-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 14:20:47 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:20:47 err: <nil>
2024/06/26 14:20:47 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:20:52 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:20:57 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:21:02 Pod mysql-5d58d79cfd-5gx92 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:21:07 Running post-restore function for case mysql-restic-e2e
2024/06/26 14:21:07 checking for the NAMESPACE: mysql-persistent
2024/06/26 14:21:07 Verifying backup/restore data of todolist
2024/06/26 14:21:07 PrebackState: false
2024/06/26 14:21:07 Data came from backup-file
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 14:19:31.007715 -0400 EDT m=+1539.998376459","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]

2024/06/26 14:21:07 Data from the response after restore
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 14:19:31.007715 -0400 EDT m=+1539.998376459","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]
2024/06/26 14:21:07 Post backup and restore state:  passed
2024/06/26 14:21:07 FLAKE DETECTION: No known flakes found.
•
2024/06/26 14:21:18 Waiting for velero pod to be running
2024/06/26 14:21:18 velero pods not found
2024/06/26 14:21:23 velero pods are running
2024/06/26 14:21:23 Waiting for Node Agent pods to be running
2024/06/26 14:21:23 Checking for correct number of running Node Agent pods...
2024/06/26 14:21:23 Installing application for case mongo-kopia-e2e
diff found for key: allowHostIPC
diff found for key: allowHostPID
diff found for key: groups
diff found for key: allowHostNetwork
diff found for key: metadata
diff found for key: readOnlyRootFilesystem
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostPorts
diff found for key: volumes
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mongo-persistent-scc
2024/06/26 14:21:23 Pod mongo-7c674cb76c-9br4s not yet succeeded: phase is Pending
2024/06/26 14:21:28 Pod mongo-7c674cb76c-9br4s not yet succeeded: phase is Pending
2024/06/26 14:21:33 Pod mongo-7c674cb76c-9br4s not yet succeeded: phase is Pending
2024/06/26 14:21:38 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:21:43 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:21:48 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:21:54 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:21:59 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:22:04 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:22:09 Pod todolist-1-ntxrz not yet succeeded: phase is Pending
2024/06/26 14:22:14 Running pre-backup function for case mongo-kopia-e2e
2024/06/26 14:22:14 Verifying backup/restore data of todolist
2024/06/26 14:22:14 PrebackState: true
2024/06/26 14:22:14 Data before the curl request:
 [{"Id":"667c5c539ad4eceea8dcac30","Description":"time","Completed":false}]
2024/06/26 14:22:14 Writing data to backupFile (backup-data.txt):
 [{"Id":"667c5c539ad4eceea8dcac30","Description":"time","Completed":false},{"Id":"667c5c569ad4eceea8dcac31","Description":"2024-06-26 14:22:14.472525 -0400 EDT m=+1703.461721168","Completed":false},{"Id":"667c5c569ad4eceea8dcac32","Description":"Wednesday","Completed":false}]
2024/06/26 14:22:14 Sleeping for 0s to allow application to be ready for case mongo-kopia-e2e
2024/06/26 14:22:14 Creating backup mongo-kopia-e2e-e49901d2-33e8-11ef-a1bc-1acc26a6847d for case mongo-kopia-e2e
2024/06/26 14:22:24 backup phase: InProgress
2024/06/26 14:22:34 backup phase: Completed
2024/06/26 14:22:35 Backup for case mongo-kopia-e2e succeeded
2024/06/26 14:22:35 Uninstalling app for case mongo-kopia-e2e
2024/06/26 14:22:46 Creating restore mongo-kopia-e2e-e49901f0-33e8-11ef-a1bc-1acc26a6847d for case mongo-kopia-e2e
2024/06/26 14:22:46 restore phase: InProgress
2024/06/26 14:22:56 restore phase: InProgress
2024/06/26 14:23:06 restore phase: InProgress
2024/06/26 14:23:16 restore phase: InProgress
2024/06/26 14:23:26 restore phase: Completed
2024/06/26 14:23:27 Running dc-post-restore.sh script.
2024/06/26 14:23:27 Running post restore script for mongo-kopia-e2e-e49901f0-33e8-11ef-a1bc-1acc26a6847d
2024/06/26 14:23:27 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mongo-kopia-e2e-e49901f0-33e8-11ef-a1bc-1acc26a6847d
2024/06/26 14:23:27 stdout:
restore: mongo-kopia-e2e-e49901f0-33e8-11ef-a1bc-1acc26a6847d
label:   mongo-kopia-e2e-e49901f0-33e8-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 14:23:27 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:23:27 err: <nil>
2024/06/26 14:23:27 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:23:32 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:23:37 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:23:42 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:23:47 Pod mongo-7c674cb76c-9br4s not yet succeeded: condition is false: ContainersReady
2024/06/26 14:23:52 Pod todolist-1-fblsv not yet succeeded: phase is Pending
2024/06/26 14:23:57 Running post-restore function for case mongo-kopia-e2e
2024/06/26 14:23:58 Verifying backup/restore data of todolist
2024/06/26 14:23:58 PrebackState: false
2024/06/26 14:23:58 Data came from backup-file
 [{"Id":"667c5c539ad4eceea8dcac30","Description":"time","Completed":false},{"Id":"667c5c569ad4eceea8dcac31","Description":"2024-06-26 14:22:14.472525 -0400 EDT m=+1703.461721168","Completed":false},{"Id":"667c5c569ad4eceea8dcac32","Description":"Wednesday","Completed":false}]

2024/06/26 14:23:58 Data from the response after restore
 [{"Id":"667c5c539ad4eceea8dcac30","Description":"time","Completed":false},{"Id":"667c5c569ad4eceea8dcac31","Description":"2024-06-26 14:22:14.472525 -0400 EDT m=+1703.461721168","Completed":false},{"Id":"667c5c569ad4eceea8dcac32","Description":"Wednesday","Completed":false}]
2024/06/26 14:23:58 Post backup and restore state:  passed
2024/06/26 14:23:58 FLAKE DETECTION: No known flakes found.
•
2024/06/26 14:24:13 Waiting for velero pod to be running
2024/06/26 14:24:13 velero pods not found
2024/06/26 14:24:18 pod: velero-6565f59f7c-7v4fw is not yet running: phase is Pending
2024/06/26 14:24:23 velero pods are running
2024/06/26 14:24:23 Waiting for Node Agent pods to be running
2024/06/26 14:24:23 Checking for correct number of running Node Agent pods...
2024/06/26 14:24:23 Installing application for case mysql-kopia-e2e
diff found for key: allowHostNetwork
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostIPC
diff found for key: allowHostPID
diff found for key: readOnlyRootFilesystem
diff found for key: metadata
diff found for key: allowHostPorts
diff found for key: groups
diff found for key: volumes
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mysql-persistent-scc
2024/06/26 14:24:24 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: phase is Pending
2024/06/26 14:24:29 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: phase is Pending
2024/06/26 14:24:34 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:24:39 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:24:44 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:24:49 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:24:54 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:24:59 Pod mysql-5d58d79cfd-dwh45 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:25:04 Pod todolist-1-qh2qq not yet succeeded: phase is Pending
2024/06/26 14:25:09 Running pre-backup function for case mysql-kopia-e2e
2024/06/26 14:25:09 checking for the NAMESPACE: mysql-persistent
2024/06/26 14:25:09 Verifying backup/restore data of todolist
2024/06/26 14:25:09 PrebackState: true
2024/06/26 14:25:09 Data before the curl request:
 [{"Id":1,"Description":"time to make the donuts","Completed":false}]
2024/06/26 14:25:09 Writing data to backupFile (backup-data.txt):
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 14:25:09.73837 -0400 EDT m=+1878.725994334","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]
2024/06/26 14:25:09 Sleeping for 0s to allow application to be ready for case mysql-kopia-e2e
2024/06/26 14:25:09 Creating backup mysql-kopia-e2e-501f63ec-33e9-11ef-a1bc-1acc26a6847d for case mysql-kopia-e2e
2024/06/26 14:25:20 backup phase: InProgress
2024/06/26 14:25:30 backup phase: Completed
2024/06/26 14:25:30 Backup for case mysql-kopia-e2e succeeded
2024/06/26 14:25:30 Uninstalling app for case mysql-kopia-e2e
2024/06/26 14:25:41 Creating restore mysql-kopia-e2e-501f640a-33e9-11ef-a1bc-1acc26a6847d for case mysql-kopia-e2e
2024/06/26 14:25:41 restore phase: InProgress
2024/06/26 14:25:51 restore phase: InProgress
2024/06/26 14:26:01 restore phase: InProgress
2024/06/26 14:26:11 restore phase: InProgress
2024/06/26 14:26:21 restore phase: Failed
2024/06/26 14:26:22 Post backup and restore state:  failed
2024/06/26 14:26:22 FLAKE DETECTION: No known flakes found.
2024/06/26 14:26:22 Printing events for namespace:  mysql-persistent
2024/06/26 14:26:22 Printing events for namespace:  openshift-adp
2024/06/26 14:27:03 Post backup and restore state:  failed
2024/06/26 14:27:03 FLAKE DETECTION: No known flakes found.
2024/06/26 14:27:03 Printing events for namespace:  mysql-persistent
2024/06/26 14:27:03 Printing events for namespace:  openshift-adp
2024/06/26 14:27:03 Post backup and restore state:  failed
2024/06/26 14:27:03 FLAKE DETECTION: No known flakes found.
2024/06/26 14:27:03 Printing events for namespace:  mysql-persistent
2024/06/26 14:27:03 Printing events for namespace:  openshift-adp

------------------------------
• [FAILED] [170.604 seconds]
Backup and restore tests Backup and restore applications [It] MySQL application KOPIA
/Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:340

  Timeline >>
    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    map[string]any{
    	"creationTimestamp": string("2024-06-26T18:20:23Z"),
    	"generation":        int64(1),
    	"labels": map[string]any{
    		"e2e-app":                string("true"),
  - 		"velero.io/backup-name":  string("mysql-restic-e2e-86342ffe-33e8-11ef-a1bc-1acc26a6847d"),
  - 		"velero.io/restore-name": string("mysql-restic-e2e-86343012-33e8-11ef-a1bc-1acc26a6847d"),
    	},
    	"managedFields": []any{map[string]any{"apiVersion": string("security.openshift.io/v1"), "fieldsType": string("FieldsV1"), "fieldsV1": map[string]any{"f:allowPrivilegeEscalation": map[string]any{}, "f:allowPrivilegedContainer": map[string]any{}, "f:fsGroup": map[string]any{".": map[string]any{}, "f:type": map[string]any{}}, "f:groups": map[string]any{}, ...}, "manager": string("__debug_bin2136896783"), ...}},
    	"name":          string("mysql-persistent-scc"),
    	... // 2 identical entries
    }

    any(
  - 	bool(false),
    )

    any(
  - 	[]any{},
    )

    []any{
  - 	string("awsElasticBlockStore"),
  + 	string("*"),
  - 	string("azureDisk"),
  - 	string("azureFile"),
  - 	string("cephFS"),
  - 	string("cinder"),
  - 	string("configMap"),
  - 	string("csi"),
  - 	string("downwardAPI"),
  - 	string("emptyDir"),
  - 	string("ephemeral"),
  - 	string("fc"),
  - 	string("flexVolume"),
  - 	string("flocker"),
  - 	string("gcePersistentDisk"),
  - 	string("gitRepo"),
  - 	string("glusterfs"),
  - 	string("iscsi"),
  - 	string("nfs"),
  - 	string("persistentVolumeClaim"),
  - 	string("photonPersistentDisk"),
  - 	string("portworxVolume"),
  - 	string("projected"),
  - 	string("quobyte"),
  - 	string("rbd"),
  - 	string("scaleIO"),
  - 	string("secret"),
  - 	string("storageOS"),
  - 	string("vsphere"),
    }

  Name:         mysql-kopia-e2e-501f63ec-33e9-11ef-a1bc-1acc26a6847d
  Namespace:    openshift-adp
  Labels:       velero.io/storage-location=ts-example-velero-1
  Annotations:  velero.io/resource-timeout=10m0s
                velero.io/source-cluster-k8s-gitversion=v1.28.7+f1b5f6c
                velero.io/source-cluster-k8s-major-version=1
                velero.io/source-cluster-k8s-minor-version=28

  Phase:  Completed

  Namespaces:
    Included:  mysql-persistent
    Excluded:  <none>

  Resources:
    Included:        *
    Excluded:        <none>
    Cluster-scoped:  auto

  Label selector:  <none>

  Or label selector:  <none>

  Storage Location:  ts-example-velero-1

  Velero-Native Snapshot PVs:  auto
  Snapshot Move Data:          false
  Data Mover:                  velero

  TTL:  720h0m0s

  CSISnapshotTimeout:    10m0s
  ItemOperationTimeout:  4h0m0s

  Hooks:  <none>

  Backup Format Version:  1.1.0

  Started:    2024-06-26 14:25:10 -0400 EDT
  Completed:  2024-06-26 14:25:21 -0400 EDT

  Expiration:  2024-07-26 14:25:10 -0400 EDT

  Total items to be backed up:  79
  Items backed up:              79

  Resource List:
    apiextensions.k8s.io/v1/CustomResourceDefinition:
      - clusterserviceversions.operators.coreos.com
    apps.openshift.io/v1/DeploymentConfig:
      - mysql-persistent/todolist
    apps/v1/Deployment:
      - mysql-persistent/mysql
    apps/v1/ReplicaSet:
      - mysql-persistent/mysql-5d58d79cfd
    authorization.openshift.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin
      - mysql-persistent/alert-routing-edit-dedicated-admins
      - mysql-persistent/dedicated-admins-project-dedicated-admins
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin
      - mysql-persistent/system:deployers
      - mysql-persistent/system:image-builders
      - mysql-persistent/system:image-pullers
    discovery.k8s.io/v1/EndpointSlice:
      - mysql-persistent/mysql-78qfx
      - mysql-persistent/todolist-2vpkv
    operators.coreos.com/v1alpha1/ClusterServiceVersion:
      - mysql-persistent/observability-operator.v0.0.28
      - mysql-persistent/route-monitor-operator.v0.1.648-22d7745
    rbac.authorization.k8s.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin
      - mysql-persistent/alert-routing-edit-dedicated-admins
      - mysql-persistent/dedicated-admins-project-dedicated-admins
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin
      - mysql-persistent/system:deployers
      - mysql-persistent/system:image-builders
      - mysql-persistent/system:image-pullers
    route.openshift.io/v1/Route:
      - mysql-persistent/todolist-route
    security.openshift.io/v1/SecurityContextConstraints:
      - mysql-persistent-scc
    v1/ConfigMap:
      - mysql-persistent/kube-root-ca.crt
      - mysql-persistent/openshift-service-ca.crt
    v1/Endpoints:
      - mysql-persistent/mysql
      - mysql-persistent/todolist
    v1/Event:
      - mysql-persistent/mysql-5d58d79cfd-dwh45.17dca114ece7ace4
      - mysql-persistent/mysql-5d58d79cfd-dwh45.17dca11576822937
      - mysql-persistent/mysql-5d58d79cfd-dwh45.17dca115962b2208
      - mysql-persistent/mysql-5d58d79cfd-dwh45.17dca115978946ff
      - mysql-persistent/mysql-5d58d79cfd-dwh45.17dca115bf987990
      - mysql-persistent/mysql-5d58d79cfd-dwh45.17dca115c574aed9
      - mysql-persistent/mysql-5d58d79cfd-dwh45.17dca115c6b3b35b
      - mysql-persistent/mysql-5d58d79cfd.17dca113c1ac1365
      - mysql-persistent/mysql.17dca113b663ad5a
      - mysql-persistent/mysql.17dca113c0d52cda
      - mysql-persistent/mysql.17dca113c2e5e05e
      - mysql-persistent/mysql.17dca113c2e6efde
      - mysql-persistent/mysql.17dca114c6b01691
      - mysql-persistent/todolist-1-deploy.17dca113c6fa0338
      - mysql-persistent/todolist-1-deploy.17dca113df8facdc
      - mysql-persistent/todolist-1-deploy.17dca113e0fbc979
      - mysql-persistent/todolist-1-deploy.17dca113e6837cea
      - mysql-persistent/todolist-1-deploy.17dca113e791a9ae
      - mysql-persistent/todolist-1-qh2qq.17dca113ef10134e
      - mysql-persistent/todolist-1-qh2qq.17dca11407666635
      - mysql-persistent/todolist-1-qh2qq.17dca11408c224c1
      - mysql-persistent/todolist-1-qh2qq.17dca1140e130333
      - mysql-persistent/todolist-1-qh2qq.17dca1140f1e050e
      - mysql-persistent/todolist-1-qh2qq.17dca11d96ef5623
      - mysql-persistent/todolist-1-qh2qq.17dca11d9c623a5e
      - mysql-persistent/todolist-1-qh2qq.17dca11d9d8894fa
      - mysql-persistent/todolist-1.17dca113eeb78b78
      - mysql-persistent/todolist.17dca113c53f3fb2
    v1/Namespace:
      - mysql-persistent
    v1/PersistentVolume:
      - pvc-5e2fc7aa-4a5e-4467-bcea-0087e81524e2
    v1/PersistentVolumeClaim:
      - mysql-persistent/mysql
    v1/Pod:
      - mysql-persistent/mysql-5d58d79cfd-dwh45
      - mysql-persistent/todolist-1-deploy
      - mysql-persistent/todolist-1-qh2qq
    v1/ReplicationController:
      - mysql-persistent/todolist-1
    v1/Secret:
      - mysql-persistent/builder-dockercfg-cc7d8
      - mysql-persistent/builder-token-rbkbv
      - mysql-persistent/default-dockercfg-bgr9d
      - mysql-persistent/default-token-x4ts5
      - mysql-persistent/deployer-dockercfg-f2gtq
      - mysql-persistent/deployer-token-tmkl2
      - mysql-persistent/mysql-persistent-sa-dockercfg-gvrwk
      - mysql-persistent/mysql-persistent-sa-token-hhqtf
    v1/Service:
      - mysql-persistent/mysql
      - mysql-persistent/todolist
    v1/ServiceAccount:
      - mysql-persistent/builder
      - mysql-persistent/default
      - mysql-persistent/deployer
      - mysql-persistent/mysql-persistent-sa

  Velero-Native Snapshots: <none included>

  kopia Backups:
    Completed:
      mysql-persistent/mysql-5d58d79cfd-dwh45: mysql-data

  Name:         mysql-kopia-e2e-501f640a-33e9-11ef-a1bc-1acc26a6847d
  Namespace:    openshift-adp
  Labels:       <none>
  Annotations:  <none>

  Phase:                       Failed (run 'velero restore logs mysql-kopia-e2e-501f640a-33e9-11ef-a1bc-1acc26a6847d' for more information)
  Total items to be restored:  50
  Items restored:              50

  Started:    2024-06-26 14:25:41 -0400 EDT
  Completed:  2024-06-26 14:26:15 -0400 EDT

  Warnings:
    Velero:     <none>
    Cluster:  could not restore, CustomResourceDefinition "clusterserviceversions.operators.coreos.com" already exists. Warning: the in-cluster version is different than the backed-up version
    Namespaces:
      mysql-persistent:  could not restore, RoleBinding "dedicated-admins-project-system:serviceaccounts:dedicated-admin" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, RoleBinding "system:image-pullers" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, ConfigMap "openshift-service-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, ClusterServiceVersion "route-monitor-operator.v0.1.648-22d7745" already exists. Warning: the in-cluster version is different than the backed-up version
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/admin-dedicated-admins: rolebindings.authorization.openshift.io "admin-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/admin-system:serviceaccounts:dedicated-admin: rolebindings.authorization.openshift.io "admin-system:serviceaccounts:dedicated-admin" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/alert-routing-edit-dedicated-admins: rolebindings.authorization.openshift.io "alert-routing-edit-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/dedicated-admins-project-dedicated-admins: rolebindings.authorization.openshift.io "dedicated-admins-project-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin: rolebindings.authorization.openshift.io "dedicated-admins-project-system:serviceaccounts:dedicated-admin" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:deployers: rolebindings.authorization.openshift.io "system:deployers" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:image-builders: rolebindings.authorization.openshift.io "system:image-builders" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:image-pullers: rolebindings.authorization.openshift.io "system:image-pullers" not found

  Backup:  mysql-kopia-e2e-501f63ec-33e9-11ef-a1bc-1acc26a6847d

  Namespaces:
    Included:  all namespaces found in the backup
    Excluded:  <none>

  Resources:
    Included:        *
    Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
    Cluster-scoped:  auto

  Namespace mappings:  <none>

  Label selector:  <none>

  Or label selector:  <none>

  Restore PVs:  auto

  kopia Restores:
    Completed:
      mysql-persistent/mysql-5d58d79cfd-dwh45: mysql-data

  Existing Resource Policy:   <none>
  ItemOperationTimeout:       4h0m0s

  Preserve Service NodePorts:  auto

  Resource List:
    apiextensions.k8s.io/v1/CustomResourceDefinition:
      - clusterserviceversions.operators.coreos.com(failed)
    apps.openshift.io/v1/DeploymentConfig:
      - mysql-persistent/todolist(created)
    apps/v1/Deployment:
      - mysql-persistent/mysql(created)
    apps/v1/ReplicaSet:
      - mysql-persistent/mysql-5d58d79cfd(created)
    authorization.openshift.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins(failed)
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/alert-routing-edit-dedicated-admins(failed)
      - mysql-persistent/dedicated-admins-project-dedicated-admins(failed)
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/system:deployers(failed)
      - mysql-persistent/system:image-builders(failed)
      - mysql-persistent/system:image-pullers(failed)
    discovery.k8s.io/v1/EndpointSlice:
      - mysql-persistent/mysql-78qfx(created)
      - mysql-persistent/todolist-2vpkv(created)
    operators.coreos.com/v1alpha1/ClusterServiceVersion:
      - mysql-persistent/observability-operator.v0.0.28(created)
      - mysql-persistent/route-monitor-operator.v0.1.648-22d7745(failed)
    rbac.authorization.k8s.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins(created)
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin(created)
      - mysql-persistent/alert-routing-edit-dedicated-admins(created)
      - mysql-persistent/dedicated-admins-project-dedicated-admins(created)
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/system:deployers(created)
      - mysql-persistent/system:image-builders(created)
      - mysql-persistent/system:image-pullers(failed)
    route.openshift.io/v1/Route:
      - mysql-persistent/todolist-route(created)
    security.openshift.io/v1/SecurityContextConstraints:
      - mysql-persistent-scc(created)
    v1/ConfigMap:
      - mysql-persistent/kube-root-ca.crt(failed)
      - mysql-persistent/openshift-service-ca.crt(failed)
    v1/Endpoints:
      - mysql-persistent/mysql(created)
      - mysql-persistent/todolist(created)
    v1/Namespace:
      - mysql-persistent(created)
    v1/PersistentVolume:
      - pvc-5e2fc7aa-4a5e-4467-bcea-0087e81524e2(skipped)
    v1/PersistentVolumeClaim:
      - mysql-persistent/mysql(created)
    v1/Pod:
      - mysql-persistent/mysql-5d58d79cfd-dwh45(created)
      - mysql-persistent/todolist-1-qh2qq(skipped)
    v1/ReplicationController:
      - mysql-persistent/todolist-1(skipped)
    v1/Secret:
      - mysql-persistent/builder-dockercfg-cc7d8(created)
      - mysql-persistent/builder-token-rbkbv(skipped)
      - mysql-persistent/default-dockercfg-bgr9d(created)
      - mysql-persistent/default-token-x4ts5(skipped)
      - mysql-persistent/deployer-dockercfg-f2gtq(created)
      - mysql-persistent/deployer-token-tmkl2(skipped)
      - mysql-persistent/mysql-persistent-sa-dockercfg-gvrwk(created)
      - mysql-persistent/mysql-persistent-sa-token-hhqtf(skipped)
    v1/Service:
      - mysql-persistent/mysql(created)
      - mysql-persistent/todolist(created)
    v1/ServiceAccount:
      - mysql-persistent/builder(skipped)
      - mysql-persistent/default(skipped)
      - mysql-persistent/deployer(skipped)
      - mysql-persistent/mysql-persistent-sa(created)

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:203 @ 06/26/24 14:26:22.481
  Printing app namespace events
  Event: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered., Type: Normal, Count: 2, Src: {PersistentVolumeClaim mysql-persistent mysql 766fbe87-096a-41af-b2f4-83a32be4dec7 v1 108627251 }, Reason: ExternalProvisioning
  Event: External provisioner is provisioning volume for claim "mysql-persistent/mysql", Type: Normal, Count: 1, Src: {PersistentVolumeClaim mysql-persistent mysql 766fbe87-096a-41af-b2f4-83a32be4dec7 v1 108627251 }, Reason: Provisioning
  Event: Created container deployment, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 8ab4b5ed-44b0-46fa-85b4-18a6e3ff0135 v1 108627313 spec.containers{deployment}}, Reason: Created
  Event: Created new replication controller "todolist-1" for version 1, Type: Normal, Count: 1, Src: {DeploymentConfig mysql-persistent todolist 0f3fd546-1671-44a5-8a57-a06e66b54838 apps.openshift.io/v1 108627301 }, Reason: DeploymentCreated
  Event: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:108cb092eb74afbdcb5733986d429cf591586242481217960e37c20eaba38582" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 8ab4b5ed-44b0-46fa-85b4-18a6e3ff0135 v1 108627313 spec.containers{deployment}}, Reason: Pulled
  Event: Started container deployment, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 8ab4b5ed-44b0-46fa-85b4-18a6e3ff0135 v1 108627313 spec.containers{deployment}}, Reason: Started
  Event: Add eth0 [10.131.0.222/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 8ab4b5ed-44b0-46fa-85b4-18a6e3ff0135 v1 108627320 }, Reason: AddedInterface
  Event: Failed to update endpoint mysql-persistent/todolist: Operation cannot be fulfilled on endpoints "todolist": the object has been modified; please apply your changes to the latest version and try again, Type: Warning, Count: 1, Src: {Endpoints mysql-persistent todolist 515aeb09-41e5-49ba-b14d-e7d891326974 v1 108627285 }, Reason: FailedToUpdateEndpoint
  Event: Created pod: todolist-1-6n7q8, Type: Normal, Count: 1, Src: {ReplicationController mysql-persistent todolist-1 ded2fdd6-6ee6-4d14-b633-da8ce4e734c0 v1 108627345 }, Reason: SuccessfulCreate
  Event: Started container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-6n7q8 75777850-32f7-40b2-89b9-a8189d599757 v1 108627349 spec.initContainers{init-myservice}}, Reason: Started
  Event: Created container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-6n7q8 75777850-32f7-40b2-89b9-a8189d599757 v1 108627349 spec.initContainers{init-myservice}}, Reason: Created
  Event: Add eth0 [10.128.2.33/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-6n7q8 75777850-32f7-40b2-89b9-a8189d599757 v1 108627356 }, Reason: AddedInterface
  Event: Container image "docker.io/curlimages/curl:8.5.0" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-6n7q8 75777850-32f7-40b2-89b9-a8189d599757 v1 108627349 spec.initContainers{init-myservice}}, Reason: Pulled
  Event: Successfully provisioned volume pvc-766fbe87-096a-41af-b2f4-83a32be4dec7, Type: Normal, Count: 1, Src: {PersistentVolumeClaim mysql-persistent mysql 766fbe87-096a-41af-b2f4-83a32be4dec7 v1 108627252 }, Reason: ProvisioningSucceeded
  Event: AttachVolume.Attach succeeded for volume "pvc-766fbe87-096a-41af-b2f4-83a32be4dec7" , Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 }, Reason: SuccessfulAttachVolume
  Event: Started container restore-wait, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 spec.initContainers{restore-wait}}, Reason: Started
  Event: Created container restore-wait, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 spec.initContainers{restore-wait}}, Reason: Created
  Event: Container image "quay.io/konveyor/velero-restore-helper:oadp-1.3" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 spec.initContainers{restore-wait}}, Reason: Pulled
  Event: Add eth0 [10.131.0.221/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627404 }, Reason: AddedInterface
  Event: Pulling image "registry.redhat.io/rhel8/mariadb-105:latest", Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 spec.containers{mysql}}, Reason: Pulling
  Event: Started container mysql, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 spec.containers{mysql}}, Reason: Started
  Event: Created container mysql, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 spec.containers{mysql}}, Reason: Created
  Event: Successfully pulled image "registry.redhat.io/rhel8/mariadb-105:latest" in 746ms (746ms including waiting), Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-5d58d79cfd-dwh45 1f509d25-57fd-42fb-ab8b-02c399dd16a4 v1 108627402 spec.containers{mysql}}, Reason: Pulled
  Printing oadp namespace events

  Attempt #1 Failed.  Retrying ↺ @ 06/26/24 14:27:03.288

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:27:03.29
  Printing app namespace events
  Printing oadp namespace events
  Event: Stopping container velero, Type: Normal, Count: 1, Src: {Pod openshift-adp velero-6565f59f7c-7v4fw ae15b576-138d-4ce2-b221-283630ca8d4b v1 108625711 spec.containers{velero}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-b6ghg e6ca39f7-908f-4e3e-9c9a-f348c7d47ab6 v1 108625727 spec.containers{node-agent}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-hpmww 9e0ef6a1-f0e2-4f31-b887-de4176b3ee38 v1 108625715 spec.containers{node-agent}}, Reason: Killing

  Attempt #2 Failed.  Retrying ↺ @ 06/26/24 14:27:03.656

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:27:03.657
  Printing app namespace events
  Printing oadp namespace events
  Event: Stopping container velero, Type: Normal, Count: 1, Src: {Pod openshift-adp velero-6565f59f7c-7v4fw ae15b576-138d-4ce2-b221-283630ca8d4b v1 108625711 spec.containers{velero}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-b6ghg e6ca39f7-908f-4e3e-9c9a-f348c7d47ab6 v1 108625727 spec.containers{node-agent}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-hpmww 9e0ef6a1-f0e2-4f31-b887-de4176b3ee38 v1 108625715 spec.containers{node-agent}}, Reason: Killing
  << Timeline

  [FAILED] No known FLAKE found in a previous run, marking test as failed.
  In [It] at: /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:27:03.657

  There were additional failures detected.  To view them in detail run ginkgo -vv
------------------------------
2024/06/26 14:27:04 Waiting for velero pod to be running
2024/06/26 14:27:04 velero pods not found
2024/06/26 14:27:09 pod: velero-68c4958ddc-bks4x is not yet running: phase is Pending
2024/06/26 14:27:14 velero pods are running
2024/06/26 14:27:14 Waiting for Node Agent pods to be running
2024/06/26 14:27:14 Checking for correct number of running Node Agent pods...
2024/06/26 14:27:14 Creating VolumeSnapshotClass for CSI backuprestore of mongo-datamover-e2e
2024/06/26 14:27:14 Installing application for case mongo-datamover-e2e
diff found for key: readOnlyRootFilesystem
diff found for key: allowHostNetwork
diff found for key: groups
diff found for key: metadata
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostPorts
diff found for key: volumes
diff found for key: allowHostIPC
diff found for key: allowHostPID
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mongo-persistent-scc
2024/06/26 14:27:14 Creating pvc for case mongo-datamover-e2e
2024/06/26 14:27:14 Pod mongo-7c674cb76c-zff8g not yet succeeded: phase is Pending
2024/06/26 14:27:19 Pod mongo-7c674cb76c-zff8g not yet succeeded: phase is Pending
2024/06/26 14:27:24 Pod mongo-7c674cb76c-zff8g not yet succeeded: phase is Pending
2024/06/26 14:27:29 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:27:35 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:27:40 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:27:45 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:27:50 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:27:55 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:28:00 Pod todolist-1-fq66f not yet succeeded: phase is Pending
2024/06/26 14:28:05 Running pre-backup function for case mongo-datamover-e2e
2024/06/26 14:28:05 Verifying backup/restore data of todolist
2024/06/26 14:28:05 PrebackState: true
2024/06/26 14:28:05 Data before the curl request:
 [{"Id":"667c5db36ea2262dc709fcf1","Description":"time","Completed":false}]
2024/06/26 14:28:05 Writing data to backupFile (backup-data.txt):
 [{"Id":"667c5db36ea2262dc709fcf1","Description":"time","Completed":false},{"Id":"667c5db56ea2262dc709fcf2","Description":"2024-06-26 14:28:05.635935 -0400 EDT m=+2054.621982626","Completed":false},{"Id":"667c5db56ea2262dc709fcf3","Description":"Wednesday","Completed":false}]
2024/06/26 14:28:05 Sleeping for 0s to allow application to be ready for case mongo-datamover-e2e
2024/06/26 14:28:05 Creating backup mongo-datamover-e2e-b5d996e4-33e9-11ef-a1bc-1acc26a6847d for case mongo-datamover-e2e
2024/06/26 14:28:15 backup phase: InProgress
2024/06/26 14:28:25 backup phase: WaitingForPluginOperations
2024/06/26 14:28:35 backup phase: WaitingForPluginOperations
2024/06/26 14:28:46 backup phase: WaitingForPluginOperations
2024/06/26 14:28:56 backup phase: WaitingForPluginOperations
2024/06/26 14:29:06 backup phase: Completed
2024/06/26 14:29:07 Backup for case mongo-datamover-e2e succeeded
2024/06/26 14:29:07 Uninstalling app for case mongo-datamover-e2e
2024/06/26 14:29:22 Creating restore mongo-datamover-e2e-b5d9972a-33e9-11ef-a1bc-1acc26a6847d for case mongo-datamover-e2e
2024/06/26 14:29:22 restore phase: InProgress
2024/06/26 14:29:32 restore phase: InProgress
2024/06/26 14:29:42 restore phase: InProgress
2024/06/26 14:29:52 restore phase: WaitingForPluginOperations
2024/06/26 14:30:02 restore phase: WaitingForPluginOperations
2024/06/26 14:30:12 restore phase: WaitingForPluginOperations
2024/06/26 14:30:22 restore phase: WaitingForPluginOperations
2024/06/26 14:30:32 restore phase: Completed
2024/06/26 14:30:33 Running dc-post-restore.sh script.
2024/06/26 14:30:33 Running post restore script for mongo-datamover-e2e-b5d9972a-33e9-11ef-a1bc-1acc26a6847d
2024/06/26 14:30:34 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mongo-datamover-e2e-b5d9972a-33e9-11ef-a1bc-1acc26a6847d
2024/06/26 14:30:34 stdout:
restore: mongo-datamover-e2e-b5d9972a-33e9-11ef-a1bc-1acc26a6847d
label:   mongo-datamover-e2e-b5d9972a-33e9-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 14:30:34 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:30:34 err: <nil>
2024/06/26 14:30:34 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:30:40 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:30:45 Pod mongo-7c674cb76c-zff8g not yet succeeded: condition is false: ContainersReady
2024/06/26 14:30:50 Running post-restore function for case mongo-datamover-e2e
2024/06/26 14:30:50 Verifying backup/restore data of todolist
2024/06/26 14:30:50 PrebackState: false
2024/06/26 14:30:50 Data came from backup-file
 [{"Id":"667c5db36ea2262dc709fcf1","Description":"time","Completed":false},{"Id":"667c5db56ea2262dc709fcf2","Description":"2024-06-26 14:28:05.635935 -0400 EDT m=+2054.621982626","Completed":false},{"Id":"667c5db56ea2262dc709fcf3","Description":"Wednesday","Completed":false}]

2024/06/26 14:30:50 Data from the response after restore
 [{"Id":"667c5db36ea2262dc709fcf1","Description":"time","Completed":false},{"Id":"667c5db56ea2262dc709fcf2","Description":"2024-06-26 14:28:05.635935 -0400 EDT m=+2054.621982626","Completed":false},{"Id":"667c5db56ea2262dc709fcf3","Description":"Wednesday","Completed":false}]
2024/06/26 14:30:50 Post backup and restore state:  passed
2024/06/26 14:30:50 FLAKE DETECTION: No known flakes found.
2024/06/26 14:30:50 Deleting VolumeSnapshot for CSI backuprestore of mongo-datamover-e2e
•
2024/06/26 14:31:05 Waiting for velero pod to be running
2024/06/26 14:31:05 velero pods not found
2024/06/26 14:31:10 pod: velero-68c4958ddc-gb9qr is not yet running: phase is Pending
2024/06/26 14:31:15 velero pods are running
2024/06/26 14:31:15 Waiting for Node Agent pods to be running
2024/06/26 14:31:15 Checking for correct number of running Node Agent pods...
2024/06/26 14:31:15 Creating VolumeSnapshotClass for CSI backuprestore of mysql-datamover-e2e
2024/06/26 14:31:15 Installing application for case mysql-datamover-e2e
diff found for key: groups
diff found for key: readOnlyRootFilesystem
diff found for key: volumes
diff found for key: metadata
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostIPC
diff found for key: allowHostPID
diff found for key: allowHostPorts
diff found for key: allowHostNetwork
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mysql-persistent-scc
2024/06/26 14:31:16 Creating pvc for case mysql-datamover-e2e
2024/06/26 14:31:16 Pod mysql-74f94448f9-5j2j5 not yet succeeded: phase is Pending
2024/06/26 14:31:21 Pod mysql-74f94448f9-5j2j5 not yet succeeded: phase is Pending
2024/06/26 14:31:26 Pod mysql-74f94448f9-5j2j5 not yet succeeded: phase is Pending
2024/06/26 14:31:31 Pod mysql-74f94448f9-5j2j5 not yet succeeded: phase is Pending
2024/06/26 14:31:36 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:31:41 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:31:46 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:31:51 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:31:56 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:32:01 Pod todolist-1-m5whr not yet succeeded: phase is Pending
2024/06/26 14:32:06 Running pre-backup function for case mysql-datamover-e2e
2024/06/26 14:32:06 checking for the NAMESPACE: mysql-persistent
2024/06/26 14:32:06 Verifying backup/restore data of todolist
2024/06/26 14:32:06 PrebackState: true
2024/06/26 14:32:07 Data before the curl request:
 [{"Id":1,"Description":"time to make the donuts","Completed":false}]
2024/06/26 14:32:07 Writing data to backupFile (backup-data.txt):
 [{"Id":1,"Description":"time to make the donuts","Completed":false},{"Id":3,"Description":"2024-06-26 14:32:07.110063 -0400 EDT m=+2296.093945293","Completed":false},{"Id":4,"Description":"Wednesday","Completed":false}]
2024/06/26 14:32:07 Sleeping for 0s to allow application to be ready for case mysql-datamover-e2e
2024/06/26 14:32:07 Creating backup mysql-datamover-e2e-45d51638-33ea-11ef-a1bc-1acc26a6847d for case mysql-datamover-e2e
2024/06/26 14:32:17 backup phase: InProgress
2024/06/26 14:32:27 backup phase: WaitingForPluginOperations
2024/06/26 14:32:37 backup phase: WaitingForPluginOperations
2024/06/26 14:32:47 backup phase: WaitingForPluginOperations
2024/06/26 14:32:57 backup phase: WaitingForPluginOperations
2024/06/26 14:33:07 backup phase: Completed
2024/06/26 14:33:08 Backup for case mysql-datamover-e2e succeeded
2024/06/26 14:33:08 Uninstalling app for case mysql-datamover-e2e
2024/06/26 14:33:24 Creating restore mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d for case mysql-datamover-e2e
2024/06/26 14:33:24 restore phase: InProgress
2024/06/26 14:33:34 restore phase: InProgress
2024/06/26 14:33:44 restore phase: InProgress
2024/06/26 14:33:54 restore phase: WaitingForPluginOperations
2024/06/26 14:34:04 restore phase: WaitingForPluginOperations
2024/06/26 14:34:14 restore phase: WaitingForPluginOperations
2024/06/26 14:34:24 restore phase: Completed
2024/06/26 14:34:25 Running dc-post-restore.sh script.
2024/06/26 14:34:25 Running post restore script for mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d
2024/06/26 14:34:26 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d
2024/06/26 14:34:26 stdout:
restore: mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d
label:   mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 14:34:26 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:34:26 err: <nil>
2024/06/26 14:34:26 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:34:31 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:34:36 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:34:41 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:34:46 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:34:51 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:34:56 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:01 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:06 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:11 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:16 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:21 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:26 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:31 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:35:36 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:35:42 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:35:47 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:35:52 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:35:57 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:02 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:07 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:12 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:17 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:22 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:27 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:32 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:36:37 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:36:42 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:36:47 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:36:52 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:36:57 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:02 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:07 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:12 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:18 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:23 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:28 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:33 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:37:38 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:37:43 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:37:48 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:37:53 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:37:58 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:38:03 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:38:08 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:38:13 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:38:18 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:38:23 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:38:28 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:38:33 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:38:38 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:38:43 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:38:49 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:38:54 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:38:59 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:39:04 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:39:09 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:39:14 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:39:19 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:39:24 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:39:29 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:39:34 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:39:39 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:39:44 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:39:49 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:39:54 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:39:59 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:40:04 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:40:09 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:40:14 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:40:20 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:40:25 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:40:30 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:40:35 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:40:40 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:40:45 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:40:50 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:40:55 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:00 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:05 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:10 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:15 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:20 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:25 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:30 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:35 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:40 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:45 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:51 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:41:56 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:42:01 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:42:06 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:42:11 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:42:16 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:42:21 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:42:26 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:42:31 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:42:41 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:42:46 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:42:51 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:42:56 Pod todolist-1-l52t6 not yet succeeded: phase is Pending
2024/06/26 14:43:01 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:43:06 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:43:11 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:43:17 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:43:22 Pod mysql-74f94448f9-5j2j5 not yet succeeded: condition is false: ContainersReady
2024/06/26 14:43:26 Post backup and restore state:  failed
2024/06/26 14:43:26 FLAKE DETECTION: No known flakes found.
2024/06/26 14:43:26 Printing events for namespace:  mysql-persistent
2024/06/26 14:43:26 Printing events for namespace:  openshift-adp
2024/06/26 14:43:26 Deleting VolumeSnapshot for CSI backuprestore of mysql-datamover-e2e
2024/06/26 14:44:07 Post backup and restore state:  failed
2024/06/26 14:44:07 FLAKE DETECTION: No known flakes found.
2024/06/26 14:44:07 Printing events for namespace:  mysql-persistent
2024/06/26 14:44:07 Printing events for namespace:  openshift-adp
2024/06/26 14:44:07 Deleting VolumeSnapshot for CSI backuprestore of mysql-datamover-e2e
2024/06/26 14:44:07 Post backup and restore state:  failed
2024/06/26 14:44:07 FLAKE DETECTION: No known flakes found.
2024/06/26 14:44:07 Printing events for namespace:  mysql-persistent
2024/06/26 14:44:07 Printing events for namespace:  openshift-adp
2024/06/26 14:44:07 Deleting VolumeSnapshot for CSI backuprestore of mysql-datamover-e2e

------------------------------
• [FAILED] [782.306 seconds]
Backup and restore tests Backup and restore applications [It] MySQL application DATAMOVER
/Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:356

  Timeline >>
    any(
  - 	[]any{},
    )

    any(
  - 	bool(false),
    )

    []any{
  - 	string("awsElasticBlockStore"),
  + 	string("*"),
  - 	string("azureDisk"),
  - 	string("azureFile"),
  - 	string("cephFS"),
  - 	string("cinder"),
  - 	string("configMap"),
  - 	string("csi"),
  - 	string("downwardAPI"),
  - 	string("emptyDir"),
  - 	string("ephemeral"),
  - 	string("fc"),
  - 	string("flexVolume"),
  - 	string("flocker"),
  - 	string("gcePersistentDisk"),
  - 	string("gitRepo"),
  - 	string("glusterfs"),
  - 	string("iscsi"),
  - 	string("nfs"),
  - 	string("persistentVolumeClaim"),
  - 	string("photonPersistentDisk"),
  - 	string("portworxVolume"),
  - 	string("projected"),
  - 	string("quobyte"),
  - 	string("rbd"),
  - 	string("scaleIO"),
  - 	string("secret"),
  - 	string("storageOS"),
  - 	string("vsphere"),
    }

    map[string]any{
    	"creationTimestamp": string("2024-06-26T18:25:59Z"),
    	"generation":        int64(1),
    	"labels": map[string]any{
    		"e2e-app":                string("true"),
  - 		"velero.io/backup-name":  string("mysql-kopia-e2e-501f63ec-33e9-11ef-a1bc-1acc26a6847d"),
  - 		"velero.io/restore-name": string("mysql-kopia-e2e-501f640a-33e9-11ef-a1bc-1acc26a6847d"),
    	},
    	"managedFields": []any{map[string]any{"apiVersion": string("security.openshift.io/v1"), "fieldsType": string("FieldsV1"), "fieldsV1": map[string]any{"f:allowPrivilegeEscalation": map[string]any{}, "f:allowPrivilegedContainer": map[string]any{}, "f:fsGroup": map[string]any{".": map[string]any{}, "f:type": map[string]any{}}, "f:groups": map[string]any{}, ...}, "manager": string("__debug_bin2136896783"), ...}},
    	"name":          string("mysql-persistent-scc"),
    	... // 2 identical entries
    }

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

    any(
  - 	bool(false),
    )

  Name:         mysql-datamover-e2e-45d51638-33ea-11ef-a1bc-1acc26a6847d
  Namespace:    openshift-adp
  Labels:       velero.io/storage-location=ts-example-velero-1
  Annotations:  velero.io/resource-timeout=10m0s
                velero.io/source-cluster-k8s-gitversion=v1.28.7+f1b5f6c
                velero.io/source-cluster-k8s-major-version=1
                velero.io/source-cluster-k8s-minor-version=28

  Phase:  Completed

  Namespaces:
    Included:  mysql-persistent
    Excluded:  <none>

  Resources:
    Included:        *
    Excluded:        <none>
    Cluster-scoped:  auto

  Label selector:  <none>

  Or label selector:  <none>

  Storage Location:  ts-example-velero-1

  Velero-Native Snapshot PVs:  auto
  Snapshot Move Data:          true
  Data Mover:                  velero

  TTL:  720h0m0s

  CSISnapshotTimeout:    10m0s
  ItemOperationTimeout:  4h0m0s

  Hooks:  <none>

  Backup Format Version:  1.1.0

  Started:    2024-06-26 14:32:07 -0400 EDT
  Completed:  2024-06-26 14:33:05 -0400 EDT

  Expiration:  2024-07-26 14:32:07 -0400 EDT

  Total items to be backed up:  80
  Items backed up:              80

  Backup Item Operations:
    Operation for persistentvolumeclaims mysql-persistent/mysql:
      Backup Item Action Plugin:  velero.io/csi-pvc-backupper
      Operation ID:               du-25658995-6d11-4460-b8a5-9046fa44e0ee.f70dabb2-03f8-4cf32c38e
      Items to Update:
                             datauploads.velero.io openshift-adp/mysql-datamover-e2e-45d51638-33ea-11ef-a1bc-1acc26a6847d-zrtzp
      Phase:                 Completed
      Progress:              36956832 of 36956832 complete (Bytes)
      Progress description:  Completed
      Created:               2024-06-26 14:32:18 -0400 EDT
      Started:               2024-06-26 14:32:18 -0400 EDT
      Updated:               2024-06-26 14:32:55 -0400 EDT
  Resource List:
    apiextensions.k8s.io/v1/CustomResourceDefinition:
      - clusterserviceversions.operators.coreos.com
    apps.openshift.io/v1/DeploymentConfig:
      - mysql-persistent/todolist
    apps/v1/Deployment:
      - mysql-persistent/mysql
    apps/v1/ReplicaSet:
      - mysql-persistent/mysql-74f94448f9
    authorization.openshift.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin
      - mysql-persistent/alert-routing-edit-dedicated-admins
      - mysql-persistent/dedicated-admins-project-dedicated-admins
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin
      - mysql-persistent/system:deployers
      - mysql-persistent/system:image-builders
      - mysql-persistent/system:image-pullers
    discovery.k8s.io/v1/EndpointSlice:
      - mysql-persistent/mysql-92xhk
      - mysql-persistent/todolist-h92vr
    operators.coreos.com/v1alpha1/ClusterServiceVersion:
      - mysql-persistent/observability-operator.v0.0.28
      - mysql-persistent/route-monitor-operator.v0.1.648-22d7745
    rbac.authorization.k8s.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin
      - mysql-persistent/alert-routing-edit-dedicated-admins
      - mysql-persistent/dedicated-admins-project-dedicated-admins
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin
      - mysql-persistent/system:deployers
      - mysql-persistent/system:image-builders
      - mysql-persistent/system:image-pullers
    route.openshift.io/v1/Route:
      - mysql-persistent/todolist-route
    security.openshift.io/v1/SecurityContextConstraints:
      - mysql-persistent-scc
    v1/ConfigMap:
      - mysql-persistent/kube-root-ca.crt
      - mysql-persistent/openshift-service-ca.crt
    v1/Endpoints:
      - mysql-persistent/mysql
      - mysql-persistent/todolist
    v1/Event:
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca173b9449dc7
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca17528c5b11e
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca175b3628f68
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca177124602b7
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca17713bb1a5a
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca177499f4d9e
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca1774dbdb1a8
      - mysql-persistent/mysql-74f94448f9-5j2j5.17dca1774ec08504
      - mysql-persistent/mysql-74f94448f9.17dca173b94c9afd
      - mysql-persistent/mysql.17dca173b84b079a
      - mysql-persistent/mysql.17dca173c039aa73
      - mysql-persistent/mysql.17dca173fea9812d
      - mysql-persistent/mysql.17dca173feab4dcf
      - mysql-persistent/mysql.17dca175031b3c52
      - mysql-persistent/todolist-1-deploy.17dca173c1e8ae9e
      - mysql-persistent/todolist-1-deploy.17dca173d9ebbbac
      - mysql-persistent/todolist-1-deploy.17dca173db780d95
      - mysql-persistent/todolist-1-deploy.17dca173e12241b3
      - mysql-persistent/todolist-1-deploy.17dca173e235a4eb
      - mysql-persistent/todolist-1-m5whr.17dca173e9c764ca
      - mysql-persistent/todolist-1-m5whr.17dca174024ce9df
      - mysql-persistent/todolist-1-m5whr.17dca17403bae7bf
      - mysql-persistent/todolist-1-m5whr.17dca17408d445f5
      - mysql-persistent/todolist-1-m5whr.17dca17409e9b98d
      - mysql-persistent/todolist-1-m5whr.17dca17ea81d7e2e
      - mysql-persistent/todolist-1-m5whr.17dca17eae03ae9d
      - mysql-persistent/todolist-1-m5whr.17dca17eaf563537
      - mysql-persistent/todolist-1.17dca173e96b3549
      - mysql-persistent/todolist.17dca173bda5412d
    v1/Namespace:
      - mysql-persistent
    v1/PersistentVolume:
      - pvc-f70dabb2-03f8-4cfc-a3b8-8baafb91ad90
    v1/PersistentVolumeClaim:
      - mysql-persistent/mysql
    v1/Pod:
      - mysql-persistent/mysql-74f94448f9-5j2j5
      - mysql-persistent/todolist-1-deploy
      - mysql-persistent/todolist-1-m5whr
    v1/ReplicationController:
      - mysql-persistent/todolist-1
    v1/Secret:
      - mysql-persistent/builder-dockercfg-rdxzd
      - mysql-persistent/builder-token-7j7kf
      - mysql-persistent/default-dockercfg-rxgdb
      - mysql-persistent/default-token-85h9j
      - mysql-persistent/deployer-dockercfg-dq5bx
      - mysql-persistent/deployer-token-pwlcf
      - mysql-persistent/mysql-persistent-sa-dockercfg-b6skk
      - mysql-persistent/mysql-persistent-sa-token-hzwrn
    v1/Service:
      - mysql-persistent/mysql
      - mysql-persistent/todolist
    v1/ServiceAccount:
      - mysql-persistent/builder
      - mysql-persistent/default
      - mysql-persistent/deployer
      - mysql-persistent/mysql-persistent-sa

  Velero-Native Snapshots: <none included>

  Name:         mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d
  Namespace:    openshift-adp
  Labels:       <none>
  Annotations:  <none>

  Phase:                       Completed
  Total items to be restored:  51
  Items restored:              51

  Started:    2024-06-26 14:33:24 -0400 EDT
  Completed:  2024-06-26 14:34:14 -0400 EDT

  Warnings:
    Velero:     <none>
    Cluster:  could not restore, CustomResourceDefinition "clusterserviceversions.operators.coreos.com" already exists. Warning: the in-cluster version is different than the backed-up version
    Namespaces:
      mysql-persistent:  could not restore, RoleBinding "dedicated-admins-project-system:serviceaccounts:dedicated-admin" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, RoleBinding "system:image-pullers" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                         could not restore, ConfigMap "openshift-service-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/admin-dedicated-admins: rolebindings.authorization.openshift.io "admin-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/admin-system:serviceaccounts:dedicated-admin: rolebindings.authorization.openshift.io "admin-system:serviceaccounts:dedicated-admin" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/alert-routing-edit-dedicated-admins: rolebindings.authorization.openshift.io "alert-routing-edit-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/dedicated-admins-project-dedicated-admins: rolebindings.authorization.openshift.io "dedicated-admins-project-dedicated-admins" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin: rolebindings.authorization.openshift.io "dedicated-admins-project-system:serviceaccounts:dedicated-admin" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:deployers: rolebindings.authorization.openshift.io "system:deployers" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:image-builders: rolebindings.authorization.openshift.io "system:image-builders" not found
                         error getting resource from lister for rolebindings.authorization.openshift.io, mysql-persistent/system:image-pullers: rolebindings.authorization.openshift.io "system:image-pullers" not found

  Backup:  mysql-datamover-e2e-45d51638-33ea-11ef-a1bc-1acc26a6847d

  Namespaces:
    Included:  all namespaces found in the backup
    Excluded:  <none>

  Resources:
    Included:        *
    Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
    Cluster-scoped:  auto

  Namespace mappings:  <none>

  Label selector:  <none>

  Or label selector:  <none>

  Restore PVs:  auto

  Existing Resource Policy:   <none>
  ItemOperationTimeout:       4h0m0s

  Preserve Service NodePorts:  auto

  Restore Item Operations:
    Operation for persistentvolumeclaims mysql-persistent/mysql:
      Restore Item Action Plugin:  velero.io/csi-pvc-restorer
      Operation ID:                dd-b6597e92-7590-4acb-a99f-ba09da632652.f70dabb2-03f8-4cfde44c7
      Phase:                       Completed
      Progress:                    36956832 of 36956832 complete (Bytes)
      Progress description:        Completed
      Created:                     2024-06-26 14:33:42 -0400 EDT
      Started:                     2024-06-26 14:33:42 -0400 EDT
      Updated:                     2024-06-26 14:34:12 -0400 EDT
  Resource List:
    apiextensions.k8s.io/v1/CustomResourceDefinition:
      - clusterserviceversions.operators.coreos.com(failed)
    apps.openshift.io/v1/DeploymentConfig:
      - mysql-persistent/todolist(created)
    apps/v1/Deployment:
      - mysql-persistent/mysql(created)
    apps/v1/ReplicaSet:
      - mysql-persistent/mysql-74f94448f9(created)
    authorization.openshift.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins(failed)
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/alert-routing-edit-dedicated-admins(failed)
      - mysql-persistent/dedicated-admins-project-dedicated-admins(failed)
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/system:deployers(failed)
      - mysql-persistent/system:image-builders(failed)
      - mysql-persistent/system:image-pullers(failed)
    discovery.k8s.io/v1/EndpointSlice:
      - mysql-persistent/mysql-92xhk(created)
      - mysql-persistent/todolist-h92vr(created)
    operators.coreos.com/v1alpha1/ClusterServiceVersion:
      - mysql-persistent/observability-operator.v0.0.28(created)
      - mysql-persistent/route-monitor-operator.v0.1.648-22d7745(created)
    rbac.authorization.k8s.io/v1/RoleBinding:
      - mysql-persistent/admin-dedicated-admins(created)
      - mysql-persistent/admin-system:serviceaccounts:dedicated-admin(created)
      - mysql-persistent/alert-routing-edit-dedicated-admins(created)
      - mysql-persistent/dedicated-admins-project-dedicated-admins(created)
      - mysql-persistent/dedicated-admins-project-system:serviceaccounts:dedicated-admin(failed)
      - mysql-persistent/system:deployers(created)
      - mysql-persistent/system:image-builders(created)
      - mysql-persistent/system:image-pullers(failed)
    route.openshift.io/v1/Route:
      - mysql-persistent/todolist-route(created)
    security.openshift.io/v1/SecurityContextConstraints:
      - mysql-persistent-scc(created)
    v1/ConfigMap:
      - mysql-persistent/kube-root-ca.crt(failed)
      - mysql-persistent/openshift-service-ca.crt(failed)
    v1/Endpoints:
      - mysql-persistent/mysql(created)
      - mysql-persistent/todolist(created)
    v1/Namespace:
      - mysql-persistent(created)
    v1/PersistentVolume:
      - pvc-f70dabb2-03f8-4cfc-a3b8-8baafb91ad90(skipped)
    v1/PersistentVolumeClaim:
      - mysql-persistent/mysql(created)
    v1/Pod:
      - mysql-persistent/mysql-74f94448f9-5j2j5(created)
      - mysql-persistent/todolist-1-m5whr(skipped)
    v1/ReplicationController:
      - mysql-persistent/todolist-1(skipped)
    v1/Secret:
      - mysql-persistent/builder-dockercfg-rdxzd(created)
      - mysql-persistent/builder-token-7j7kf(skipped)
      - mysql-persistent/default-dockercfg-rxgdb(created)
      - mysql-persistent/default-token-85h9j(skipped)
      - mysql-persistent/deployer-dockercfg-dq5bx(created)
      - mysql-persistent/deployer-token-pwlcf(skipped)
      - mysql-persistent/mysql-persistent-sa-dockercfg-b6skk(created)
      - mysql-persistent/mysql-persistent-sa-token-hzwrn(skipped)
    v1/Service:
      - mysql-persistent/mysql(created)
      - mysql-persistent/todolist(created)
    v1/ServiceAccount:
      - mysql-persistent/builder(skipped)
      - mysql-persistent/default(skipped)
      - mysql-persistent/deployer(skipped)
      - mysql-persistent/mysql-persistent-sa(created)
    velero.io/v2alpha1/DataUpload:
      - openshift-adp/mysql-datamover-e2e-45d51638-33ea-11ef-a1bc-1acc26a6847d-zrtzp(skipped)
  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:219 @ 06/26/24 14:43:26.105
  Printing app namespace events
  Event: waiting for first consumer to be created before binding, Type: Normal, Count: 2, Src: {PersistentVolumeClaim mysql-persistent mysql 107254a4-ed46-469b-82cc-e948a535ca55 v1 108634944 }, Reason: WaitForFirstConsumer
  Event: Created new replication controller "todolist-1" for version 1, Type: Normal, Count: 1, Src: {DeploymentConfig mysql-persistent todolist 0c7a5de7-cf72-41ee-99ae-902a2be0d56e apps.openshift.io/v1 108635002 }, Reason: DeploymentCreated
  Event: Add eth0 [10.131.0.236/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 2790460b-d3b9-4b84-a88b-3432e656b30a v1 108635018 }, Reason: AddedInterface
  Event: Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:108cb092eb74afbdcb5733986d429cf591586242481217960e37c20eaba38582" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 2790460b-d3b9-4b84-a88b-3432e656b30a v1 108635010 spec.containers{deployment}}, Reason: Pulled
  Event: Started container deployment, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 2790460b-d3b9-4b84-a88b-3432e656b30a v1 108635010 spec.containers{deployment}}, Reason: Started
  Event: Add eth0 [10.128.2.41/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-l52t6 f5b5a7f5-ca06-42e9-8e92-fb552ed45251 v1 108635053 }, Reason: AddedInterface
  Event: Created container deployment, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-deploy 2790460b-d3b9-4b84-a88b-3432e656b30a v1 108635010 spec.containers{deployment}}, Reason: Created
  Event: Container image "docker.io/curlimages/curl:8.5.0" already present on machine, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-l52t6 f5b5a7f5-ca06-42e9-8e92-fb552ed45251 v1 108635045 spec.initContainers{init-myservice}}, Reason: Pulled
  Event: Created container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-l52t6 f5b5a7f5-ca06-42e9-8e92-fb552ed45251 v1 108635045 spec.initContainers{init-myservice}}, Reason: Created
  Event: Started container init-myservice, Type: Normal, Count: 1, Src: {Pod mysql-persistent todolist-1-l52t6 f5b5a7f5-ca06-42e9-8e92-fb552ed45251 v1 108635045 spec.initContainers{init-myservice}}, Reason: Started
  Event: Created pod: todolist-1-l52t6, Type: Normal, Count: 1, Src: {ReplicationController mysql-persistent todolist-1 7346a2d0-6f44-45f2-8efa-46a7c8c10e8a v1 108635040 }, Reason: SuccessfulCreate
  Event: External provisioner is provisioning volume for claim "mysql-persistent/mysql", Type: Normal, Count: 5, Src: {PersistentVolumeClaim mysql-persistent mysql 107254a4-ed46-469b-82cc-e948a535ca55 v1 108634973 }, Reason: Provisioning
  Event: failed to provision volume with StorageClass "gp2-csi": claim Selector is not supported, Type: Warning, Count: 5, Src: {PersistentVolumeClaim mysql-persistent mysql 107254a4-ed46-469b-82cc-e948a535ca55 v1 108634973 }, Reason: ProvisioningFailed
  Event: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered., Type: Normal, Count: 3, Src: {PersistentVolumeClaim mysql-persistent mysql 107254a4-ed46-469b-82cc-e948a535ca55 v1 108634973 }, Reason: ExternalProvisioning
  Event: volume "pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9" already bound to a different claim., Type: Warning, Count: 1, Src: {PersistentVolumeClaim mysql-persistent mysql 107254a4-ed46-469b-82cc-e948a535ca55 v1 108635341 }, Reason: FailedBinding
  Event: AttachVolume.Attach succeeded for volume "pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9" , Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 }, Reason: SuccessfulAttachVolume
  Event: Add eth0 [10.128.2.40/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635447 }, Reason: AddedInterface
  Event: Successfully pulled image "registry.redhat.io/rhel8/mariadb-105:latest" in 569ms (569ms including waiting), Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Pulled
  Event: Successfully pulled image "registry.redhat.io/rhel8/mariadb-105:latest" in 431ms (431ms including waiting), Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Pulled
  Event: Successfully pulled image "registry.redhat.io/rhel8/mariadb-105:latest" in 776ms (776ms including waiting), Type: Normal, Count: 1, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Pulled
  Event: Started container mysql, Type: Normal, Count: 3, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Started
  Event: Created container mysql, Type: Normal, Count: 3, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Created
  Event: Container mysql failed liveness probe, will be restarted, Type: Normal, Count: 3, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Killing
  Event: Pulling image "registry.redhat.io/rhel8/mariadb-105:latest", Type: Normal, Count: 4, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Pulling
  Event: Liveness probe failed: dial tcp 10.128.2.40:3306: i/o timeout, Type: Warning, Count: 15, Src: {Pod mysql-persistent mysql-74f94448f9-5j2j5 307ddf86-1c49-44d0-af99-80110fd157ed v1 108635444 spec.containers{mysql}}, Reason: Unhealthy
  Printing oadp namespace events
  Event: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered., Type: Normal, Count: 2, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: ExternalProvisioning
  Event: External provisioner is provisioning volume for claim "openshift-adp/mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw", Type: Normal, Count: 1, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: Provisioning
  Event: Unable to attach or mount volumes: unmounted volumes=[08ba7439-afd7-464a-b163-412a205e39b6], unattached volumes=[], failed to process volumes=[08ba7439-afd7-464a-b163-412a205e39b6]: error processing PVC openshift-adp/mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw: PVC is not bound, Type: Warning, Count: 2, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 }, Reason: FailedMount
  Event: Successfully provisioned volume pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9, Type: Normal, Count: 1, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: ProvisioningSucceeded
  Event: AttachVolume.Attach succeeded for volume "pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9" , Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635078 }, Reason: SuccessfulAttachVolume
  Event: Add eth0 [10.131.0.237/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635078 }, Reason: AddedInterface
  Event: Container image "quay.io/konveyor/velero:oadp-1.3" already present on machine, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Pulled
  Event: Created container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Created
  Event: Started container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Started
  Event: Stopping container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Killing

  Attempt #1 Failed.  Retrying ↺ @ 06/26/24 14:44:07.133

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:44:07.134
  Printing app namespace events
  Printing oadp namespace events
  Event: External provisioner is provisioning volume for claim "openshift-adp/mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw", Type: Normal, Count: 1, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: Provisioning
  Event: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered., Type: Normal, Count: 2, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: ExternalProvisioning
  Event: Unable to attach or mount volumes: unmounted volumes=[08ba7439-afd7-464a-b163-412a205e39b6], unattached volumes=[], failed to process volumes=[08ba7439-afd7-464a-b163-412a205e39b6]: error processing PVC openshift-adp/mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw: PVC is not bound, Type: Warning, Count: 2, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 }, Reason: FailedMount
  Event: Successfully provisioned volume pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9, Type: Normal, Count: 1, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: ProvisioningSucceeded
  Event: AttachVolume.Attach succeeded for volume "pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9" , Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635078 }, Reason: SuccessfulAttachVolume
  Event: Add eth0 [10.131.0.237/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635078 }, Reason: AddedInterface
  Event: Container image "quay.io/konveyor/velero:oadp-1.3" already present on machine, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Pulled
  Event: Created container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Created
  Event: Started container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Started
  Event: Stopping container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-kcwmr 38e363f4-82e7-4591-b8a9-fabe5c224d6a v1 108632462 spec.containers{node-agent}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-hsd6g 3cf33edb-810e-457f-ad82-251d77162012 v1 108632456 spec.containers{node-agent}}, Reason: Killing
  Event: Stopping container velero, Type: Normal, Count: 1, Src: {Pod openshift-adp velero-68c4958ddc-gb9qr 994d7ed7-2964-471d-84c1-88f773004bae v1 108632457 spec.containers{velero}}, Reason: Killing

  Attempt #2 Failed.  Retrying ↺ @ 06/26/24 14:44:07.546

  [FAILED] in [It] - /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:44:07.546
  Printing app namespace events
  Printing oadp namespace events
  Event: External provisioner is provisioning volume for claim "openshift-adp/mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw", Type: Normal, Count: 1, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: Provisioning
  Event: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered., Type: Normal, Count: 2, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: ExternalProvisioning
  Event: Unable to attach or mount volumes: unmounted volumes=[08ba7439-afd7-464a-b163-412a205e39b6], unattached volumes=[], failed to process volumes=[08ba7439-afd7-464a-b163-412a205e39b6]: error processing PVC openshift-adp/mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw: PVC is not bound, Type: Warning, Count: 2, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 }, Reason: FailedMount
  Event: Successfully provisioned volume pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9, Type: Normal, Count: 1, Src: {PersistentVolumeClaim openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 647e1bbc-d534-4c78-876b-b5a39b1ec4e9 v1 108635075 }, Reason: ProvisioningSucceeded
  Event: AttachVolume.Attach succeeded for volume "pvc-647e1bbc-d534-4c78-876b-b5a39b1ec4e9" , Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635078 }, Reason: SuccessfulAttachVolume
  Event: Add eth0 [10.131.0.237/23] from ovn-kubernetes, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635078 }, Reason: AddedInterface
  Event: Container image "quay.io/konveyor/velero:oadp-1.3" already present on machine, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Pulled
  Event: Created container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Created
  Event: Started container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Started
  Event: Stopping container 08ba7439-afd7-464a-b163-412a205e39b6, Type: Normal, Count: 1, Src: {Pod openshift-adp mysql-datamover-e2e-45d51642-33ea-11ef-a1bc-1acc26a6847d-2j5kw 5c68072e-06d1-4ff8-ab9d-dc2ddfea987c v1 108635072 spec.containers{08ba7439-afd7-464a-b163-412a205e39b6}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-kcwmr 38e363f4-82e7-4591-b8a9-fabe5c224d6a v1 108632462 spec.containers{node-agent}}, Reason: Killing
  Event: Stopping container node-agent, Type: Normal, Count: 1, Src: {Pod openshift-adp node-agent-hsd6g 3cf33edb-810e-457f-ad82-251d77162012 v1 108632456 spec.containers{node-agent}}, Reason: Killing
  Event: Stopping container velero, Type: Normal, Count: 1, Src: {Pod openshift-adp velero-68c4958ddc-gb9qr 994d7ed7-2964-471d-84c1-88f773004bae v1 108632457 spec.containers{velero}}, Reason: Killing
  << Timeline

  [FAILED] No known FLAKE found in a previous run, marking test as failed.
  In [It] at: /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287 @ 06/26/24 14:44:07.546

  There were additional failures detected.  To view them in detail run ginkgo -vv
------------------------------
2024/06/26 14:44:07 Waiting for velero pod to be running
2024/06/26 14:44:07 velero pods not found
2024/06/26 14:44:13 pod: velero-68c4958ddc-pm87j is not yet running: phase is Pending
2024/06/26 14:44:18 velero pods are running
2024/06/26 14:44:18 Waiting for Node Agent pods to be running
2024/06/26 14:44:18 Checking for correct number of running Node Agent pods...
2024/06/26 14:44:18 Creating VolumeSnapshotClass for CSI backuprestore of mongo-blockdevice-e2e
2024/06/26 14:44:18 Installing application for case mongo-blockdevice-e2e
diff found for key: allowHostPorts
diff found for key: volumes
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostPID
diff found for key: groups
diff found for key: readOnlyRootFilesystem
diff found for key: metadata
diff found for key: allowHostIPC
diff found for key: allowHostNetwork
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mongo-persistent-scc
2024/06/26 14:44:18 Creating pvc for case mongo-blockdevice-e2e
2024/06/26 14:44:18 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: phase is Pending
2024/06/26 14:44:23 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: phase is Pending
2024/06/26 14:44:28 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: phase is Pending
2024/06/26 14:44:33 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: phase is Pending
2024/06/26 14:44:38 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:44:44 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:44:49 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:44:54 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:44:59 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:45:04 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:45:09 Pod todolist-1-4t7rl not yet succeeded: phase is Pending
2024/06/26 14:45:14 Running pre-backup function for case mongo-blockdevice-e2e
2024/06/26 14:45:14 Verifying backup/restore data of todolist
2024/06/26 14:45:14 PrebackState: true
2024/06/26 14:45:14 Data before the curl request:
 [{"Id":"667c61b7213f88703e0776e6","Description":"time","Completed":false}]
2024/06/26 14:45:14 Writing data to backupFile (backup-data.txt):
 [{"Id":"667c61b7213f88703e0776e6","Description":"time","Completed":false},{"Id":"667c61ba213f88703e0776e7","Description":"2024-06-26 14:45:14.571478 -0400 EDT m=+3083.554339709","Completed":false},{"Id":"667c61ba213f88703e0776e8","Description":"Wednesday","Completed":false}]
2024/06/26 14:45:14 Sleeping for 0s to allow application to be ready for case mongo-blockdevice-e2e
2024/06/26 14:45:14 Creating backup mongo-blockdevice-e2e-18223b1a-33ec-11ef-a1bc-1acc26a6847d for case mongo-blockdevice-e2e
2024/06/26 14:45:24 backup phase: InProgress
2024/06/26 14:45:34 backup phase: WaitingForPluginOperations
2024/06/26 14:45:44 backup phase: WaitingForPluginOperations
2024/06/26 14:45:54 backup phase: WaitingForPluginOperations
2024/06/26 14:46:05 backup phase: WaitingForPluginOperations
2024/06/26 14:46:15 backup phase: WaitingForPluginOperations
2024/06/26 14:46:25 backup phase: Completed
2024/06/26 14:46:26 Backup for case mongo-blockdevice-e2e succeeded
2024/06/26 14:46:26 Uninstalling app for case mongo-blockdevice-e2e
2024/06/26 14:47:11 Creating restore mongo-blockdevice-e2e-18223b42-33ec-11ef-a1bc-1acc26a6847d for case mongo-blockdevice-e2e
2024/06/26 14:47:11 restore phase: InProgress
2024/06/26 14:47:21 restore phase: InProgress
2024/06/26 14:47:31 restore phase: WaitingForPluginOperations
2024/06/26 14:47:41 restore phase: WaitingForPluginOperations
2024/06/26 14:47:51 restore phase: WaitingForPluginOperations
2024/06/26 14:48:01 restore phase: WaitingForPluginOperations
2024/06/26 14:48:11 restore phase: WaitingForPluginOperations
2024/06/26 14:48:21 restore phase: WaitingForPluginOperations
2024/06/26 14:48:31 restore phase: WaitingForPluginOperations
2024/06/26 14:48:41 restore phase: WaitingForPluginOperations
2024/06/26 14:48:51 restore phase: WaitingForPluginOperations
2024/06/26 14:49:02 restore phase: WaitingForPluginOperations
2024/06/26 14:49:12 restore phase: Completed
2024/06/26 14:49:13 Running dc-post-restore.sh script.
2024/06/26 14:49:13 Running post restore script for mongo-blockdevice-e2e-18223b42-33ec-11ef-a1bc-1acc26a6847d
2024/06/26 14:49:14 command: /bin/bash /Users/tiger/oadp-operator/docs/scripts/dc-post-restore.sh mongo-blockdevice-e2e-18223b42-33ec-11ef-a1bc-1acc26a6847d
2024/06/26 14:49:14 stdout:
restore: mongo-blockdevice-e2e-18223b42-33ec-11ef-a1bc-1acc26a6847d
label:   mongo-blockdevice-e2e-18223b42-33ec-11ef-a1bc-1acc26a6847d
Deleting disconnected restore pods
No resources found
2024/06/26 14:49:14 stderr:
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
2024/06/26 14:49:14 err: <nil>
2024/06/26 14:49:14 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:49:19 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:49:24 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:49:29 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:49:34 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:49:39 Pod mongo-6cf7cb4f67-p5xrr not yet succeeded: condition is false: ContainersReady
2024/06/26 14:49:44 Running post-restore function for case mongo-blockdevice-e2e
2024/06/26 14:49:44 Verifying backup/restore data of todolist
2024/06/26 14:49:44 PrebackState: false
2024/06/26 14:49:44 Data came from backup-file
 [{"Id":"667c61b7213f88703e0776e6","Description":"time","Completed":false},{"Id":"667c61ba213f88703e0776e7","Description":"2024-06-26 14:45:14.571478 -0400 EDT m=+3083.554339709","Completed":false},{"Id":"667c61ba213f88703e0776e8","Description":"Wednesday","Completed":false}]

2024/06/26 14:49:44 Data from the response after restore
 [{"Id":"667c61b7213f88703e0776e6","Description":"time","Completed":false},{"Id":"667c61ba213f88703e0776e7","Description":"2024-06-26 14:45:14.571478 -0400 EDT m=+3083.554339709","Completed":false},{"Id":"667c61ba213f88703e0776e8","Description":"Wednesday","Completed":false}]
2024/06/26 14:49:44 Post backup and restore state:  passed
2024/06/26 14:49:44 FLAKE DETECTION: No known flakes found.
2024/06/26 14:49:44 Deleting VolumeSnapshot for CSI backuprestore of mongo-blockdevice-e2e
•SSSSSSSSSSSSSSSSSSSSSS
2024/06/26 14:50:30 Deleting Velero CR

Summarizing 3 Failures:
  [FAIL] Backup and restore tests Backup and restore applications [It] MySQL application two Vol CSI
  /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287
  [FAIL] Backup and restore tests Backup and restore applications [It] MySQL application KOPIA
  /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287
  [FAIL] Backup and restore tests Backup and restore applications [It] MySQL application DATAMOVER
  /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287

Ran 10 of 32 Specs in 3399.284 seconds
FAIL! -- 7 Passed | 3 Failed | 0 Pending | 22 Skipped
--- FAIL: TestOADPE2E (3399.29s)
FAIL
Process 2852 has exited with status 1
Detaching

ahem.. been testing quay image.. oops.. not this image.. put unsupportedOverrides in the wrong spot in e2e code.. so it got overriden to empty.

This has to be new... aws bsl cert error..
time="2024-06-26T23:13:58Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"ts-example-velero-1\" is unavailable: rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get \"https://tkaovila-aug30-velero-bsl.s3.amazonaws.com/?delimiter=%2F&list-type=2&prefix=velero-e2e-857c2fa9-2fd1-421c-af5b-fe3aad8b6d49%2F\": tls: failed to verify certificate: x509: certificate signed by unknown authority)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
time="2024-06-26T23:13:58Z" level=error msg="Error listing backups in backup store" backupLocation=openshift-adp/ts-example-velero-1 controller=backup-sync error="rpc error: code = Unknown desc = RequestError: send request failed\ncaused by: Get \"https://tkaovila-aug30-velero-bsl.s3.amazonaws.com/?delimiter=%2F&list-type=2&prefix=velero-e2e-857c2fa9-2fd1-421c-af5b-fe3aad8b6d49%2Fbackups%2F\": tls: failed to verify certificate: x509: certificate signed by unknown authority" error.file="/opt/app-root/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:456" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:107"

After setting insecureskip
time="2024-06-26T23:26:02Z" level=error msg="Error backing up item" backup=openshift-adp/mongo-kopia-e2e-54650860-3413-11ef-b3f0-1acc26a6847d error="failed to wait BackupRepository: backup repository is not ready: error to init backup repo: error to connect repo with storage: error to connect to repository: unable to write config file: unable to create config directory: mkdir /udmrepo: permission denied" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/repository/backup_repo_op.go:83" error.function=github.com/vmware-tanzu/velero/pkg/repository.GetBackupRepository logSource="pkg/backup/backup.go:448" name=mongo-7c674cb76c-qk2fc

kopias are failing with insecure..
mkdir /udmrepo: permission denied

might need to add these kopia volumeMounts to velero deploy? how did this e2e work before...
https://github.com/vmware-tanzu/velero/issues/6909#issuecomment-1748253240

                        volumesnapshotcontents.snapshot.storage.k8s.io /snapcontent-cee10ba0-74fa-4dc5-baad-686f404daa28
      Phase:            Failed
      Operation Error:  Failed to create snapshot: error updating status for volume snapshot content snapcontent-cee10ba0-74fa-4dc5-baad-686f404daa28: snapshot controller failed to update snapcontent-cee10ba0-74fa-4dc5-baad-686f404daa28 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-cee10ba0-74fa-4dc5-baad-686f404daa28": the object has been modified; please apply your changes to the latest version and try again

great start

Summarizing 1 Failure:
  [FAIL] Backup and restore tests Backup and restore applications [It] MySQL application CSI
  /Users/tiger/oadp-operator/tests/e2e/backup_restore_suite_test.go:287

Ran 10 of 32 Specs in 2343.908 seconds
FAIL! -- 9 Passed | 1 Failed | 0 Pending | 22 Skipped

doing one more backup restore suite and I think this is good.

token expired mid run.. running another again

    Velero:    name: /mongo-7c674cb76c-c6bvb error: /pod volume backup failed: error to initialize data path: error to boost backup repository connection ts-example-velero-1-mongo-persistent-kopia: error to connect backup repo: error to connect repo with storage: error to connect to repository: repository not initialized in the provided storage

Good run! SUCCESS! -- 10 Passed | 0 Failed | 0 Pending | 22 Skipped
backup restore suite test.
Details

Starting: /Users/tiger/go/bin/dlv dap --listen=127.0.0.1:57321 --log-dest=3 from /Users/tiger/oadp-operator/tests/e2e
DAP server listening at: 127.0.0.1:57321
Type 'dlv help' for list of commands.
Running Suite: OADP E2E using velero prefix: velero-e2e-b1c1791d-d068-42c4-8609-c25e9d73da8f - /Users/tiger/oadp-operator/tests/e2e
===================================================================================================================================
Random Seed: 1719511211

Will run 10 of 32 specs
S
2024/06/27 14:00:11 Waiting for velero pod to be running
2024/06/27 14:00:11 velero pods are running
2024/06/27 14:00:11 Creating VolumeSnapshotClass for CSI backuprestore of mysql-csi-e2e
2024/06/27 14:00:11 Installing application for case mysql-csi-e2e
diff found for key: allowHostIPC
diff found for key: allowHostNetwork
diff found for key: volumes
diff found for key: groups
diff found for key: allowHostDirVolumePlugin
diff found for key: allowHostPID
diff found for key: allowHostPorts
diff found for key: readOnlyRootFilesystem
updating resource: security.openshift.io/v1, Kind=SecurityContextConstraints; name: mysql-persistent-scc
2024/06/27 14:00:12 Creating pvc for case mysql-csi-e2e
2024/06/27 14:00:12 Pod mysql-74f94448f9-rc7zr not yet succeeded: phase is Pending
2024/06/27 14:00:17 Pod mysql-74f94448f9-rc7zr not yet succeeded: phase is Pending
2024/06/27 14:00:22 Pod mysql-74f94448f9-rc7zr not yet succeeded: phase is Pending
2024/06/27 14:00:27 Pod mysql-74f94448f9-rc7zr not yet succeeded: condition is false: ContainersReady
2024/06/27 14:00:32 Pod mysql-74f94448f9-rc7zr not yet succeeded: condition is false: ContainersReady
2024/06/27 14:00:37 Pod mysql-74f94448f9-rc7zr not yet succeeded: condition is false: ContainersReady
2024/06/27 14:00:42 Pod mysql-74f94448f9-rc7zr not yet succeeded: condition is false: ContainersReady
2024/06/27 14:00:47 Pod mysql-74f94448f9-rc7zr not yet succeeded: condition is false: ContainersReady

/lgtm

/approve

If this works it would be a miracle.
/cherry-pick oadp-1.4

authors: kaovilai sseago weshayutin

repo: openshift/velero