text:
Multiple secondary storage in one zone; problem with volume snapshots

ISSUE TYPE

Bug Report

CLOUDSTACK VERSION

4.19.1.0, xcp-ng hypervisor

SUMMARY

We are planning to add multiple NFS secondary storages for backup purpose. And I don't understand the principle of distribution of storage snapshots among secstors.
I created 10 Intsances in one Account and all their volume snapshots exept one were copied to the first secstore. Only one volume snapshot was copied to the second NFS secstore, but I can't create addiditinal snapshots of this volume. I attach logs in comments.
STEPS TO REPRODUCE

Add multiple NFS secondary storages, try to create volume snapshots of different instances

EXPECTED RESULTS

Snapshots are copied based on free space allocation mode to all secstores
ACTUAL RESULTS

All snapshotes are copied to the first secstore

Example of successful snapshot
2024-09-26 17:21:41,283 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464) (logid:c8e45246) Executing AsyncJobVO: {id:72464, userId: 298, accountId: 225, instanceType: Snapshot, instanceId: 870, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"asyncbackup":"false","response":"json","ctxUserId":"298","volumeid":"5e964ddb-3d14-4037-a241-12972ebc5354","httpmethod":"GET","ctxStartEventId":"1811491","id":"870","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"5e964ddb-3d14-4037-a241-12972ebc5354\",\"interface com.cloud.storage.Snapshot\":\"7bdc207e-974e-41ed-a51f-f104bac1102c\"}","ctxAccountId":"225","uuid":"7bdc207e-974e-41ed-a51f-f104bac1102c","cmdEventType":"SNAPSHOT.CREATE"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2024-09-26 17:21:41,293 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Access to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] granted to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] by DomainChecker
2024-09-26 17:21:41,304 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-285c7273) (logid:dbfd31f0) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:41,329 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-285c7273 ctx-655df114) (logid:dbfd31f0) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:41,333 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Access to org.apache.cloudstack.storage.volume.VolumeObject@46645c52 granted to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] by DomainChecker
2024-09-26 17:21:41,341 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Access to VM instance {"id":5474,"instanceName":"i-225-5474-VM","type":"User","uuid":"64c1d590-88a5-4dea-aede-76e0afa0bf32"} granted to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] by DomainChecker
2024-09-26 17:21:41,350 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Sync job-72465 execution on object VmWorkJobQueue.5474
2024-09-26 17:21:42,578 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465) (logid:c8e45246) Executing AsyncJobVO: {id:72465, userId: 298, accountId: 225, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAdaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAkwAB3pvbmVJZHN0ABBMamF2YS91dGlsL0xpc3Q7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAOEAAAAAAAABKgAAAAAAABVidAAUVm9sdW1lQXBpU2VydmljZUltcGwAAHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAgAAAAAAAADZnNxAH4ACAAAAAAAABgTcA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Sep 26 17:21:41 MSK 2024, removed: null}
2024-09-26 17:21:42,579 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465) (logid:c8e45246) Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5474, job origin: 72464
2024-09-26 17:21:42,581 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Execute VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":6163,"policyId":0,"snapshotId":870,"quiesceVm":false,"asyncBackup":false,"userId":298,"accountId":225,"vmId":5474,"handlerName":"VolumeApiServiceImpl"}
2024-09-26 17:21:42,594 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) StorpoolSnapshotStrategy.canHandle: snapshot=backup-test-karasev-9_ROOT-5474_20240926142141, uuid=7bdc207e-974e-41ed-a51f-f104bac1102c, op=TAKE
2024-09-26 17:21:42,632 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Taking snapshot of org.apache.cloudstack.storage.snapshot.SnapshotObject@4bd9bc84
2024-09-26 17:21:42,666 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Taking snapshot of org.apache.cloudstack.storage.snapshot.SnapshotObject@4bd9bc84 and encryption required is false
2024-09-26 17:21:42,666 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand
2024-09-26 17:21:42,667 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) We are returning the default host to execute commands because the command is not of Copy type.
2024-09-26 17:21:42,670 DEBUG [c.c.a.t.Request] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Seq 77360-7409547286931430185: Sending  { Cmd , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5474","size":"(10.00 GB) 10737418240","path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":"6163","vmName":"i-225-5474-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6163","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":"870","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 17:21:42,671 DEBUG [c.c.a.t.Request] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Seq 77360-7409547286931430185: Executing:  { Cmd , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5474","size":"(10.00 GB) 10737418240","path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":"6163","vmName":"i-225-5474-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6163","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":"870","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 17:21:42,672 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase] (DirectAgent-287:ctx-df71784d) (logid:c8e45246) Executing command CreateObjectCommand: [{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"name":"ROOT-5474","size":10737418240,"path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":6163,"vmName":"i-225-5474-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":54,"id":6163,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":870,"quiescevm":false,"physicalSize":0,"accountId":225,"followRedirects":false}},"wait":0,"bypassHostMaintenance":false}].
2024-09-26 17:21:44,421 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834338:ctx-38b579a2) (logid:2740bef9) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:44,446 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834338:ctx-38b579a2 ctx-9d7b2d8d) (logid:2740bef9) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:47,537 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834338:ctx-eb108eb7) (logid:25800034) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:47,561 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834338:ctx-eb108eb7 ctx-d3852c77) (logid:25800034) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:50,671 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-95e72378) (logid:4e24b10a) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:50,693 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-95e72378 ctx-9321eab1) (logid:4e24b10a) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:53,779 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-581ef07b) (logid:dca0df5b) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:53,811 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-581ef07b ctx-69923e87) (logid:dca0df5b) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:56,898 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-cc466224) (logid:d706fc8d) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:56,926 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-cc466224 ctx-f28715e9) (logid:d706fc8d) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:21:57,319 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-287:ctx-df71784d) (logid:c8e45246) Seq 77360-7409547286931430185: Response Received:
2024-09-26 17:21:57,319 DEBUG [c.c.a.t.Request] (DirectAgent-287:ctx-df71784d) (logid:c8e45246) Seq 77360-7409547286931430185: Processing:  { Ans: , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"d8f3d271-4b1d-4276-b4ea-390c3f82df57","id":"0","quiescevm":"false","physicalSize":"0","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 17:21:57,319 DEBUG [c.c.a.t.Request] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Seq 77360-7409547286931430185: Received:  { Ans: , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 10, { CreateObjectAnswer } }
2024-09-26 17:21:57,377 DEBUG [o.a.c.s.h.HeuristicRuleHelper] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) No heuristic rules found for zone with ID [7] and heuristic type [SNAPSHOT]. Returning null.
2024-09-26 17:21:57,381 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Verifying image storage [7]. Capacity: total=[785 GB], used=[243 GB], threshold=[89.99999761581421%].
2024-09-26 17:21:57,381 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Verifying image storage [9]. Capacity: total=[785 GB], used=[72 GB], threshold=[89.99999761581421%].
2024-09-26 17:21:57,382 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Verifying image storage [7]. Capacity: total=[785 GB], used=[243 GB], threshold=[89.99999761581421%].
2024-09-26 17:21:57,419 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2024-09-26 17:21:57,529 ERROR [o.a.c.s.e.DefaultEndPointSelector] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) IR24 select BACKUPSNAPSHOT from primary to secondary 870 dest=870
2024-09-26 17:21:57,535 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand
2024-09-26 17:21:57,537 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) We are returning the default host to execute commands because the source and destination objects are not NFS type.
2024-09-26 17:21:57,540 DEBUG [c.c.a.t.Request] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Seq 77360-7409547286931430189: Sending  { Cmd , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"d8f3d271-4b1d-4276-b4ea-390c3f82df57","volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5474","size":"(10.00 GB) 10737418240","path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":"6163","vmName":"i-225-5474-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6163","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":"870","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6163","volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5474","size":"(10.00 GB) 10737418240","path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":"6163","vmName":"i-225-5474-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6163","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":"870","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}] }
2024-09-26 17:21:57,562 DEBUG [c.c.a.t.Request] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Seq 77360-7409547286931430189: Executing:  { Cmd , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"d8f3d271-4b1d-4276-b4ea-390c3f82df57","volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5474","size":"(10.00 GB) 10737418240","path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":"6163","vmName":"i-225-5474-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6163","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":"870","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6163","volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5474","size":"(10.00 GB) 10737418240","path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":"6163","vmName":"i-225-5474-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6163","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":"870","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}] }
2024-09-26 17:21:57,562 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Executing command CopyCommand: [{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"d8f3d271-4b1d-4276-b4ea-390c3f82df57","volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"name":"ROOT-5474","size":10737418240,"path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":6163,"vmName":"i-225-5474-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":54,"id":6163,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":870,"quiescevm":false,"physicalSize":0,"accountId":225,"followRedirects":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6163","volume":{"uuid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"name":"ROOT-5474","size":10737418240,"path":"9c99b264-d618-4030-916b-1502ec5fadba","volumeId":6163,"vmName":"i-225-5474-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":54,"id":6163,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5474-VM","name":"backup-test-karasev-9_ROOT-5474_20240926142141","hypervisorType":"XenServer","id":870,"quiescevm":false,"physicalSize":0,"accountId":225,"followRedirects":false}},"executeInSequence":true,"options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":21600,"bypassHostMaintenance":false}].
2024-09-26 17:21:59,970 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) No file SR found for path: /var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6163
2024-09-26 17:21:59,970 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Creating file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6163] on host [7fe64564-c05c-4425-8436-ec2da15f740a]
2024-09-26 17:22:00,006 DEBUG [c.c.a.ApiServlet] (qtp501107890-4832815:ctx-29b0404b) (logid:360fad58) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:00,032 DEBUG [c.c.a.ApiServlet] (qtp501107890-4832815:ctx-29b0404b ctx-d7854f91) (logid:360fad58) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:03,119 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834338:ctx-c04fc898) (logid:2219e417) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:03,142 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834338:ctx-c04fc898 ctx-fec4282e) (logid:2219e417) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:06,257 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-63120037) (logid:8a9b8304) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:06,280 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-63120037 ctx-6adff946) (logid:8a9b8304) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:09,402 DEBUG [c.c.a.ApiServlet] (qtp501107890-4832815:ctx-b3166a90) (logid:b7878727) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:09,424 DEBUG [c.c.a.ApiServlet] (qtp501107890-4832815:ctx-b3166a90 ctx-af252453) (logid:b7878727) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:12,536 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-5e16f60d) (logid:ed82faf7) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:12,559 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-5e16f60d ctx-021e66d4) (logid:ed82faf7) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:15,663 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-516e5765) (logid:a4ec32b9) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:15,686 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-516e5765 ctx-f115c344) (logid:a4ec32b9) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:18,799 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824844:ctx-b60f4585) (logid:253c7fb4) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:18,822 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824844:ctx-b60f4585 ctx-cd2c67ed) (logid:253c7fb4) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:21,936 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-863d8a59) (logid:08b2e49f) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:21,959 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-863d8a59 ctx-95f1425b) (logid:08b2e49f) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:25,071 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-6e2f29f1) (logid:724bb944) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:25,094 DEBUG [c.c.a.ApiServlet] (qtp501107890-4821895:ctx-6e2f29f1 ctx-fbc38980) (logid:724bb944) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:27,025 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Host 10.69.105.21 OpaqueRef:ff865e44-ce8f-4670-b990-855700769647: Removing SR
2024-09-26 17:22:27,038 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Host 10.69.105.21 OpaqueRef:d5a56d3b-f5af-461e-bee4-b387938e18fe: Unplugging pbd
2024-09-26 17:22:27,449 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Host 10.69.105.21 OpaqueRef:ff865e44-ce8f-4670-b990-855700769647: Forgetting
2024-09-26 17:22:27,454 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Successfully destroyed snapshot on volume: 9c99b264-d618-4030-916b-1502ec5fadba execept this current snapshot d8f3d271-4b1d-4276-b4ea-390c3f82df57
2024-09-26 17:22:27,454 INFO  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) New snapshot details: SnapshotTO[datastore=null|volume=null|pathsnapshots/225/6163/07116d77-9286-4eaa-9e40-f4b754def19f.vhd]
2024-09-26 17:22:27,455 INFO  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) New snapshot physical utilization: (5.46 GB) 5866705408
2024-09-26 17:22:27,455 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Seq 77360-7409547286931430189: Response Received:
2024-09-26 17:22:27,455 DEBUG [c.c.a.t.Request] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Seq 77360-7409547286931430189: Processing:  { Ans: , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6163/07116d77-9286-4eaa-9e40-f4b754def19f.vhd","id":"0","quiescevm":"false","physicalSize":"5866705408","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 17:22:27,455 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-366:ctx-f42b18d7) (logid:c8e45246) Seq 77360-7409547286931430189: No more commands found
2024-09-26 17:22:27,455 DEBUG [c.c.a.t.Request] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Seq 77360-7409547286931430189: Received:  { Ans: , MgmtId: 108597816265214, via: 77360(xcp-sr246-u0708.aisrp.local), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2024-09-26 17:22:27,491 DEBUG [c.c.r.ResourceLimitManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Updating resource Type = secondary_storage count for Account = 225 Operation = decreasing Amount = (4.54 GB) 4870712832
2024-09-26 17:22:27,499 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Done executing VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":6163,"policyId":0,"snapshotId":870,"quiesceVm":false,"asyncBackup":false,"userId":298,"accountId":225,"vmId":5474,"handlerName":"VolumeApiServiceImpl"}
2024-09-26 17:22:27,500 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Complete async job-72465, jobStatus: SUCCEEDED, resultCode: 0, result: rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAADZg
2024-09-26 17:22:27,501 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Publish async job-72465 complete on message bus
2024-09-26 17:22:27,501 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Wake up jobs related to job-72465
2024-09-26 17:22:27,501 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Update db status for job-72465
2024-09-26 17:22:27,502 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465 ctx-75e57019) (logid:c8e45246) Wake up jobs joined with job-72465 and disjoin all subjobs created from job- 72465
2024-09-26 17:22:27,509 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465) (logid:c8e45246) Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5474, job origin: 72464
2024-09-26 17:22:27,509 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465) (logid:c8e45246) Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-72465
2024-09-26 17:22:27,511 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-152:ctx-2271fcd8 job-72464/job-72465) (logid:c8e45246) Remove job-72465 from job monitoring
2024-09-26 17:22:27,545 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) StorpoolSnapshotStrategy.canHandle: snapshot=backup-test-karasev-9_ROOT-5474_20240926142141, uuid=7bdc207e-974e-41ed-a51f-f104bac1102c, op=REVERT
2024-09-26 17:22:27,549 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Complete async job-72464, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"7bdc207e-974e-41ed-a51f-f104bac1102c","account":"karasev-cluster5","domainid":"ce7e4a4a-75b6-4c96-938c-fe3158da9a47","domain":"cluster5","snapshottype":"MANUAL","volumeid":"5e964ddb-3d14-4037-a241-12972ebc5354","volumename":"ROOT-5474","volumetype":"ROOT","created":"2024-09-26T17:21:41+0300","name":"backup-test-karasev-9_ROOT-5474_20240926142141","intervaltype":"MANUAL","state":"BackedUp","physicalsize":"5866705408","zoneid":"9505445a-7164-4675-876d-1a2fee1f7e46","zonename":"xcp-zone-02","revertable":"false","ostypeid":"7b53b54f-d3d2-409f-b9b0-acd39122368a","osdisplayname":"Ubuntu 22.04 LTS","virtualsize":"10737418240","tags":[],"hasannotations":"false"}
2024-09-26 17:22:27,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Publish async job-72464 complete on message bus
2024-09-26 17:22:27,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Wake up jobs related to job-72464
2024-09-26 17:22:27,550 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Update db status for job-72464
2024-09-26 17:22:27,551 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464 ctx-7ca9e6a7) (logid:c8e45246) Wake up jobs joined with job-72464 and disjoin all subjobs created from job- 72464
2024-09-26 17:22:27,555 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-63:ctx-968ed109 job-72464) (logid:c8e45246) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-72464
2024-09-26 17:22:27,555 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-63:ctx-968ed109 job-72464) (logid:c8e45246) Remove job-72464 from job monitoring
2024-09-26 17:22:28,208 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834306:ctx-41f1ef09) (logid:5789c854) ===START===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json
2024-09-26 17:22:28,238 DEBUG [c.c.a.ApiServlet] (qtp501107890-4834306:ctx-41f1ef09 ctx-be04a227) (logid:5789c854) ===END===  172.16.11.61 -- GET  jobId=c8e45246-57d3-4238-8cf1-6b7fa887c144&command=queryAsyncJobResult&response=json

Example when first volume snapshot stores on the second secstore and I create the next volume snapshot
2024-09-26 16:58:32,135 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Executing AsyncJobVO: {id:72454, userId: 298, accountId: 225, instanceType: Snapshot, instanceId: 867, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"asyncBackup":"false","quiescevm":"false","response":"json","ctxUserId":"298","volumeId":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","httpmethod":"GET","ctxStartEventId":"1811467","id":"867","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"da420806-0ff3-43b8-9ee0-4aa2adfd9b99\",\"interface com.cloud.storage.Snapshot\":\"c62b0aa0-5a1c-475c-b7f6-6c9fab17e325\"}","ctxAccountId":"225","uuid":"c62b0aa0-5a1c-475c-b7f6-6c9fab17e325","cmdEventType":"SNAPSHOT.CREATE"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2024-09-26 16:58:32,147 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96) Access to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] granted to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] by DomainChecker
2024-09-26 16:58:32,167 DEBUG [c.c.a.ApiServlet] (qtp501107890-4830812:ctx-4bacce10) (logid:11a55db0) ===START===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:32,183 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96) Access to org.apache.cloudstack.storage.volume.VolumeObject@f1e18b7 granted to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] by DomainChecker
2024-09-26 16:58:32,190 DEBUG [c.c.a.ApiServlet] (qtp501107890-4830812:ctx-4bacce10 ctx-aa1d9072) (logid:11a55db0) ===END===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:32,192 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96) Access to VM instance {"id":5468,"instanceName":"i-225-5468-VM","type":"User","uuid":"d1e26871-d01f-44cc-ac61-f19f46ed741a"} granted to Account [{"accountName":"karasev-cluster5","id":225,"uuid":"fd32d818-5d3b-4d3c-9084-34704beefaf5"}] by DomainChecker
2024-09-26 16:58:32,206 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96) Sync job-72455 execution on object VmWorkJobQueue.5468
2024-09-26 16:58:32,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Executing AsyncJobVO: {id:72455, userId: 298, accountId: 225, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAdaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAkwAB3pvbmVJZHN0ABBMamF2YS91dGlsL0xpc3Q7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAOEAAAAAAAABKgAAAAAAABVcdAAUVm9sdW1lQXBpU2VydmljZUltcGwAAHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAgAAAAAAAADY3NxAH4ACAAAAAAAABgMcA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Sep 26 16:58:32 MSK 2024, removed: null}
2024-09-26 16:58:32,575 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5468, job origin: 72454
2024-09-26 16:58:32,576 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Execute VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":6156,"policyId":0,"snapshotId":867,"quiesceVm":false,"asyncBackup":false,"userId":298,"accountId":225,"vmId":5468,"handlerName":"VolumeApiServiceImpl"}
2024-09-26 16:58:32,588 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) StorpoolSnapshotStrategy.canHandle: snapshot=backup-test-karasev-3_ROOT-5468_20240926135832, uuid=c62b0aa0-5a1c-475c-b7f6-6c9fab17e325, op=TAKE
2024-09-26 16:58:32,622 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Taking snapshot of org.apache.cloudstack.storage.snapshot.SnapshotObject@18f701c2
2024-09-26 16:58:32,653 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Taking snapshot of org.apache.cloudstack.storage.snapshot.SnapshotObject@18f701c2 and encryption required is false
2024-09-26 16:58:32,653 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand
2024-09-26 16:58:32,655 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) We are returning the default host to execute commands because the command is not of Copy type.
2024-09-26 16:58:32,657 DEBUG [c.c.a.t.Request] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Seq 77353-4517673376206130891: Sending  { Cmd , MgmtId: 108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00 GB) 21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 16:58:32,659 DEBUG [c.c.a.t.Request] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Seq 77353-4517673376206130891: Executing:  { Cmd , MgmtId: 108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00 GB) 21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 16:58:32,659 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase] (DirectAgent-128:ctx-052ce805) (logid:f365dd96) Executing command CreateObjectCommand: [{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"name":"ROOT-5468","size":21474836480,"path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":6156,"vmName":"i-225-5468-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":53,"id":6156,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":867,"quiescevm":false,"parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":0,"accountId":225,"followRedirects":false}},"wait":0,"bypassHostMaintenance":false}].
2024-09-26 16:58:35,268 DEBUG [c.c.a.ApiServlet] (qtp501107890-4820302:ctx-f711cc5b) (logid:4e44467e) ===START===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:35,290 DEBUG [c.c.a.ApiServlet] (qtp501107890-4820302:ctx-f711cc5b ctx-e8b6659b) (logid:4e44467e) ===END===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:38,370 DEBUG [c.c.a.ApiServlet] (qtp501107890-4830812:ctx-fd303fe2) (logid:ef8914d8) ===START===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:38,394 DEBUG [c.c.a.ApiServlet] (qtp501107890-4830812:ctx-fd303fe2 ctx-aec37e2b) (logid:ef8914d8) ===END===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:39,977 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-128:ctx-052ce805) (logid:f365dd96) Seq 77353-4517673376206130891: Response Received:
2024-09-26 16:58:39,977 DEBUG [c.c.a.t.Request] (DirectAgent-128:ctx-052ce805) (logid:f365dd96) Seq 77353-4517673376206130891: Processing:  { Ans: , MgmtId: 108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","id":"0","quiescevm":"false","physicalSize":"0","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 16:58:39,977 DEBUG [c.c.a.t.Request] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Seq 77353-4517673376206130891: Received:  { Ans: , MgmtId: 108597816265214, via: 77353(xcp-sr246-u1314.aisrp.local), Ver: v1, Flags: 10, { CreateObjectAnswer } }
2024-09-26 16:58:40,045 DEBUG [o.a.c.s.h.HeuristicRuleHelper] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) No heuristic rules found for zone with ID [7] and heuristic type [SNAPSHOT]. Returning null.
2024-09-26 16:58:40,049 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Verifying image storage [7]. Capacity: total=[785 GB], used=[243 GB], threshold=[89.99999761581421%].
2024-09-26 16:58:40,050 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Verifying image storage [9]. Capacity: total=[785 GB], used=[70 GB], threshold=[89.99999761581421%].
2024-09-26 16:58:40,051 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Verifying image storage [7]. Capacity: total=[785 GB], used=[243 GB], threshold=[89.99999761581421%].
2024-09-26 16:58:40,101 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT
2024-09-26 16:58:40,224 ERROR [o.a.c.s.e.DefaultEndPointSelector] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) IR24 select BACKUPSNAPSHOT from primary to secondary 867 dest=867
2024-09-26 16:58:40,236 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand
2024-09-26 16:58:40,238 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) We are returning the default host to execute commands because the source and destination objects are not NFS type.
2024-09-26 16:58:40,243 DEBUG [c.c.a.t.Request] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Seq 77358-1940207014466653252: Sending  { Cmd , MgmtId: 108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00 GB) 21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6156","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00 GB) 21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"false"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}] }
2024-09-26 16:58:40,245 DEBUG [c.c.a.t.Request] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Seq 77358-1940207014466653252: Executing:  { Cmd , MgmtId: 108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00 GB) 21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":"0","accountId":"225","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6156","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":"53","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":"0","url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":"false"}},"name":"ROOT-5468","size":"(20.00 GB) 21474836480","path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":"6156","vmName":"i-225-5468-VM","accountId":"225","format":"VHD","provisioningType":"THIN","poolId":"53","id":"6156","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":"867","quiescevm":"false","physicalSize":"0","accountId":"225","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"false"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}] }
2024-09-26 16:58:40,246 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Executing command CopyCommand: [{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e1971e58-5b6e-4f59-a256-154280cf4d16","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"name":"ROOT-5468","size":21474836480,"path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":6156,"vmName":"i-225-5468-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":53,"id":6156,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"parentSnapshotPath":"70424963-5933-4e73-b185-cfdc402175cd","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":867,"quiescevm":false,"parents":["70424963-5933-4e73-b185-cfdc402175cd"],"physicalSize":0,"accountId":225,"followRedirects":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/225/6156","volume":{"uuid":"da420806-0ff3-43b8-9ee0-4aa2adfd9b99","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN3","name":"cluster5-dm7100_lun3","id":53,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN3","port":0,"url":"PreSetup://localhost/DM7100_LUN3/?ROLE=Primary&STOREUUID=DM7100_LUN3","isManaged":false}},"name":"ROOT-5468","size":21474836480,"path":"9e2a4d6e-3ab8-46f4-85cf-c7454fc9ea17","volumeId":6156,"vmName":"i-225-5468-VM","accountId":225,"format":"VHD","provisioningType":"THIN","poolId":53,"id":6156,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.254/var/NFS","_role":"Image"}},"vmName":"i-225-5468-VM","name":"backup-test-karasev-3_ROOT-5468_20240926135832","hypervisorType":"XenServer","id":867,"quiescevm":false,"physicalSize":0,"accountId":225,"followRedirects":false}},"executeInSequence":true,"options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"false"},"options2":{},"wait":21600,"bypassHostMaintenance":false}].
2024-09-26 16:58:41,264 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) No file SR found for path: /var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156
2024-09-26 16:58:41,264 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Creating file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:41,495 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824844:ctx-107e93ec) (logid:1bc5742a) ===START===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:41,518 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824844:ctx-107e93ec ctx-f58fcdf2) (logid:1bc5742a) ===END===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:41,520 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Unpluging PBD [c9ae6161-7db8-f308-db93-4fa060c94f9c] of SR [f6204e02-6efe-3005-95a3-095f19b458ac] as it is not working properly.
2024-09-26 16:58:41,536 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Forgetting SR [f6204e02-6efe-3005-95a3-095f19b458ac] as it is not working properly.
2024-09-26 16:58:41,617 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Could not create file SR [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] on host [4208d709-b553-47c0-aee2-7942d370b70c].
2024-09-26 16:58:41,617 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Exception in backupsnapshot stage due to com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already used file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:41,617 WARN  [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) BackupSnapshot Failed due to Exception in backupsnapshot stage due to com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already used file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:44,625 DEBUG [c.c.a.ApiServlet] (qtp501107890-4825573:ctx-e72bd43c) (logid:d5a19d4e) ===START===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:44,651 DEBUG [c.c.a.ApiServlet] (qtp501107890-4825573:ctx-e72bd43c ctx-707eaf3e) (logid:d5a19d4e) ===END===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:47,762 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-517f2c19) (logid:c6e8be8b) ===START===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:47,784 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824845:ctx-517f2c19 ctx-9a6894c4) (logid:c6e8be8b) ===END===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:47,863 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Seq 77358-1940207014466653252: Response Received:
2024-09-26 16:58:47,864 DEBUG [c.c.a.t.Request] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Seq 77358-1940207014466653252: Processing:  { Ans: , MgmtId: 108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"BackupSnapshot Failed due to Exception in backupsnapshot stage due to com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already used file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]","wait":"0","bypassHostMaintenance":"false"}}] }
2024-09-26 16:58:47,864 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-322:ctx-abb19c96) (logid:f365dd96) Seq 77358-1940207014466653252: No more commands found
2024-09-26 16:58:47,864 DEBUG [c.c.a.t.Request] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Seq 77358-1940207014466653252: Received:  { Ans: , MgmtId: 108597816265214, via: 77358(xcp-sr246-u1112.aisrp.local), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2024-09-26 16:58:47,885 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Failed to create snapshotBackupSnapshot Failed due to Exception in backupsnapshot stage due to com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already used file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:47,885 DEBUG [c.c.r.ResourceLimitManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Updating resource Type = snapshot count for Account = 225 Operation = decreasing Amount = 1
2024-09-26 16:58:47,894 DEBUG [c.c.r.ResourceLimitManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Updating resource Type = secondary_storage count for Account = 225 Operation = decreasing Amount = (20.00 GB) 21474836480
2024-09-26 16:58:47,901 ERROR [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Take snapshot: 6156 failed
2024-09-26 16:58:47,902 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Exception in backupsnapshot stage due to com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already used file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:47,902 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455 ctx-d1890501) (logid:f365dd96) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Exception in backupsnapshot stage due to com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already used file SR for path [/var/cloud_mount/440acca8-622d-3dfb-825d-b3e62d7bba87/snapshots/225/6156] or create a new file SR on host [4208d709-b553-47c0-aee2-7942d370b70c]
2024-09-26 16:58:47,902 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5468, job origin: 72454
2024-09-26 16:58:47,902 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Unable to complete AsyncJobVO: {id:72455, userId: 298, accountId: 225, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAdaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAkwAB3pvbmVJZHN0ABBMamF2YS91dGlsL0xpc3Q7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAOEAAAAAAAABKgAAAAAAABVcdAAUVm9sdW1lQXBpU2VydmljZUltcGwAAHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAgAAAAAAAADY3NxAH4ACAAAAAAAABgMcA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Sep 26 16:58:32 MSK 2024, removed: null}, job origin:72454
2024-09-26 16:58:47,904 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Complete async job-72455, jobStatus: FAILED, resultCode: 0, result: [encoded data, 5742 characters]
2024-09-26 16:58:47,905 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Publish async job-72455 complete on message bus
2024-09-26 16:58:47,905 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Wake up jobs related to job-72455
2024-09-26 16:58:47,905 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Update db status for job-72455
2024-09-26 16:58:47,906 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Wake up jobs joined with job-72455 and disjoin all subjobs created from job- 72455
2024-09-26 16:58:47,914 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-72455
2024-09-26 16:58:47,916 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-148:ctx-ef42787d job-72454/job-72455) (logid:f365dd96) Remove job-72455 from job monitoring
2024-09-26 16:58:47,930 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-58:ctx-c9d433a7 job-72454 ctx-829ed64e) (logid:f365dd96) Failed to create snapshot due to an internal error creating snapshot for volume da420806-0ff3-43b8-9ee0-4aa2adfd9b99
2024-09-26 16:58:47,935 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Complete async job-72454, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume da420806-0ff3-43b8-9ee0-4aa2adfd9b99"}
2024-09-26 16:58:47,936 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Publish async job-72454 complete on message bus
2024-09-26 16:58:47,936 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Wake up jobs related to job-72454
2024-09-26 16:58:47,936 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Update db status for job-72454
2024-09-26 16:58:47,937 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Wake up jobs joined with job-72454 and disjoin all subjobs created from job- 72454
2024-09-26 16:58:47,941 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-72454
2024-09-26 16:58:47,941 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-58:ctx-c9d433a7 job-72454) (logid:f365dd96) Remove job-72454 from job monitoring
2024-09-26 16:58:50,891 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824742:ctx-04767efc) (logid:00d3b60a) ===START===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json
2024-09-26 16:58:50,913 DEBUG [c.c.a.ApiServlet] (qtp501107890-4824742:ctx-04767efc ctx-ff2eca28) (logid:00d3b60a) ===END===  172.16.11.61 -- GET  jobId=f365dd96-c018-454a-874c-37dce9396d96&command=queryAsyncJobResult&response=json

Regarding the first issue of all/most snapshots going to one store, it would be helpful knowing the following:

Are the 2 secondary stores of same or similar capacity?
What is the value set for the global setting: image.store.allocation.algorithm ?

@Pearl1594 1. the same capacity. Second is a copy of first
2. firstfitleastconsumed

@Pearl1594 if I set first secstore Read-Only, snapshots are copied to the second secstore. But after I unset Read-Only all snapshots are copied to the first secstore again

when creating the volume snaphots, were they are initiated at the same time i.e, either via API or through recurring snapshot schedule. If that was the case, this could possible happen because the stats collector would not have run to has updated the stats of the secondary store(s) to make the decision that store1 has capacity < than store2. Marking store1 to read only excludes it from the list of stores to which snapshot would be backed up to - expected behaviour.

repo: apache/cloudstack


text:
Projects listing breaks after deleting a user added to a project.

ISSUE TYPE

Bug Report

COMPONENT NAME

UI,API

CLOUDSTACK VERSION

4.19.1.2

CONFIGURATION

Projects
SUMMARY

Once you deleted a user in 'ROOT admin' role account, the projects can't be listed using UI/API. The deleted user was added to one of the projects.
STEPS TO REPRODUCE

1. Add a user to the 'ROOT admin' Account.
2. Create a project as another account/user in the same domain.
3. Add the user from step 1 to this project as Project Admin.
4. As an 'admin' user delete the above user.

EXPECTED RESULTS

Deleting a user shouldn't break project access.

ACTUAL RESULTS

Deleting a user breaks projects access.

Undo the user deletion. ( May not be safe for production)
Workaround example: delete from project_account where user_id=<user_id> OR ;update user set state='disabled',removed=NULL where id=<id>;

We could make the deleteUser API remove the user from all projects before deleting it (as we already do with accounts). To normalize environments that are affected, we could insert a query into the upgrade script. Something like:
delete from project_account where user_id in (select id from user where removed);

fixed in  #10008

source: gharchive/issue

authors: DaanHoogland rajujith winterhazel

license_type: permissive

license_source: github-api


text:
CLOUDSTACK-9727 Password reset discrepancy in RVR when one of the Rou…

…ter is not in Running state.

@bvbharatk can you format the code ?

@ustcweizhou
Reformatted the code.
thanks.

ACS CI BVT Run
Sumarry:
Build Number 394
Hypervisor xenserver
NetworkType Advanced
Passed=104
Failed=1
Skipped=7
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
Failed tests:

test_routers_network_ops.py

test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failed

Skipped tests:
test_01_test_vm_volume_snapshot
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

ACS CI BVT Run
Sumarry:
Build Number 400
Hypervisor xenserver
NetworkType Advanced
Passed=105
Failed=0
Skipped=7
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
Failed tests:
Skipped tests:
test_01_test_vm_volume_snapshot
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_routers_network_ops.py
test_disk_offerings.py

Code changes LGTM

@bvbharatk  I think the new password should be saved to one of the running routers, not all running routers.
I disagree with change line 797.

ACS CI BVT Run
Sumarry:
Build Number 438
Hypervisor xenserver
NetworkType Advanced
Passed=104
Failed=1
Skipped=7
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
Failed tests:

test_routers_network_ops.py

test_03_RVR_Network_check_router_state Failed

Skipped tests:
test_01_test_vm_volume_snapshot
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

@ustcweizhou
Hi we are not saving the password to the router. we are saving the password in the VM details.  When the VM starts we send the password to the router. Password reset cannot be called when the VM is running.
We save the password in the VM details if one of the routers is not running in case of rvr network. Even if the master went down and Backup came up at the time of userVM start, we will serve the correct password as we are saving it.

@bvbharatk Yes, you might know only the vm has sshkey attached will have the password in vm details. If the vm does not have ssh keypair, then the vm password will not saved into user_vm_details.
Actually the vm password should be synced between master and backup. Saving to only one of them or saving to both of them are not working fine.
for example, if we save password in master, but not save it in backup. Once the master is down, then vm cannot get password from backup vr.
another example is, if we save password on both of master and backup, if vm get the password from master and reset it, once the master is down (or master->backup switch) and we reboot vm later, the vm will get the old password from backup again.

@ustcweizhou
We are saving the password in the user_vm_details explicitly. We are not checking if ssh key pair is set for this vm or not.
I agree that ideally we should sync the password between the master and backup, For any kind of sync to work we need to know if the password was read from one of the VRs and In cases when one of the Vr is Stopped we will have to clear the password from db when it is read from the other one. These type of changes add complexity to the simple task of setting a password. The next best thing is to make sure we save the same password in both the routers. This will fix will at least solve the problem of sending the correct password even if the master and backup change state before the VM starts.
Yes like you pointed out this will lead to the problem that the user might receive the old password when he stop starts the VM, In this case the user will get a notification in the UI that his password will be changed.  So he at least knows what the password is and so he can log into the VM.

Hi,
Testing this PR end to end may not be possible now as there is some issue with the password server in case of RVRs.(CLOUDSTACK-9385). I have also encounter a similar problem when testing this manually, I saw that the password serve r is running on a different ip than expected and so the password script in the User VM failed to retrieve the password from the password server.
However I was able to verify the intended behavior because of this PR, i.e. saving the password to the RVRs was successful.

@bvbharatk
for this issue, I have some ideas.
(1) If nothing changes, the password will be applied to the first router. If the BACKUP VR is the first, then vm will not get new password.
(2) If this PR is merged, then password will be applied to both router. In case of master<-> backup switch, the password will be stored to MASTER VR (=old BACKUP), then vm password will be reset after reboot/restart.
(3) If password is only be applied to MASTER router, then there will be no password stored on BACKUP router. If vm password is reset (and stored to MASTER router), but vm is started after a MASTER<->BACKUP switch, then vm will not get new password.
Considering these three options, I think option 3 is best. What do you think ?

@ustcweizhou
Even if we store the password only in master option 2 can happen. Suppose the routers change the state again before the user starts the VM, the current master will become backup and will have the password stored on it. At some point of time when the state change happens and the VM restarts the password on the user VM will be reset again, without the knowledge of the user.
I think there is a design problem here and addressing this will need some fundamental changes, so rather than trying to force fit it, i think it is better if we leave it at this state and then document this behavior. The workaround for now would be to remove the password script from he userVM once he successfully logs in.
Thanks,
Bharat.

@bvbharatk to be honest, we used option 2 on our production before and received some complain all  vms' password are reset after reboot.
Then we used option 3. if customer cannot get new password, they can stop vm and reset vm password again (this happens not often, because normally customer start vm sooner after vm password reset so there is no master<->backup switch in this period).
Comparing these above 3 options, I do think option 3 has less impact and is more wise. VMpassword should be stored to only one place (yes I mean MASTER vr) if there is no sync.
We are using password sync between master and backup in our fork now. However, I have no time to create a PR.

@ustcweizhou
Made changes as per your suggestion, Will send the password only to master router now. However we may still fail if router changes state while the password reset operation is in progress.

ACS CI BVT Run
Sumarry:
Build Number 718
Hypervisor xenserver
NetworkType Advanced
Passed=105
Failed=8
Skipped=12
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0
Failed tests:

test_loadbalance.py

test_01_create_lb_rule_src_nat Failed

test_01_create_lb_rule_src_nat Failing since 2 runs

test_02_create_lb_rule_non_nat Failed

test_assign_and_removal_lb Failed

test_deploy_vm_iso.py

test_deploy_vm_from_iso Failed

test_volumes.py

test_06_download_detached_volume Failed

test_routers_network_ops.py

test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failed

test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

ACS CI BVT Run
Sumarry:
Build Number 736
Hypervisor xenserver
NetworkType Advanced
Passed=110
Failed=2
Skipped=12
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0
Failed tests:

test_volumes.py

test_06_download_detached_volume Failed

test_routers_network_ops.py

test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failing since 2 runs

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

@ustcweizhou
Hi,
I have made the suggested changes, Can you please review?.

@ustcweizhou
Can you please review this, I have made the suggested changes.

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1295

@weizhouapache is this still needed in your opinion?

@weizhouapache is this fixed in recent version or is this still an issue?

@rhtyd
I think it has been fixed by #3903

Thanks @weizhouapache closing on your remark
@bvbharatk pl raise a new PR if you still find the issue

source: gharchive/pull-request

authors: DaanHoogland blueorangutan bvbharatk cloudmonger jayapalu rhtyd ustcweizhou weizhouapache


text:
CLOUDSTACK-10085 : Upload volume from local fails when global config max.account.seconday.storage is set to -1

ISSUE
Upload volume from local fails when global config max.account.seconday.storage is set to -1
STEPS TO REPRODUCE

Download any volume on your local machine.
Change value of global configuration max.account.seconday.storage to -1.
Restart Management server.
Try to upload a volume from UI using "Upload from Local" option. Upload will fail.

FIX
Implementation of -1 option for unlimited secondary storage usage, which is absent in current code.

Code LGTM

@GabrielBrascher : I have rebase this PR against latest master and all checks are passing as well.
Can you please merge this PR ?

Thanks @niteshsarda but I am not the person you need now ;)

LGTM
we have the same fix in production for more than one year.

authors: GabrielBrascher mrunalinikankariya niteshsarda ustcweizhou


text:
Cloudstack-8301

PR for configuring local storage for system VMs at zone level.
FS @ https://cwiki.apache.org/confluence/display/CLOUDSTACK/Enable+configuring+local+storage+use+for+system+VMs+at+zone+level
Jira @ https://issues.apache.org/jira/browse/CLOUDSTACK-8301
@rags22489664 contibuted UI changes (https://github.com/apache/cloudstack/pull/259)
@nitt10prashant contributed tests (https://github.com/apache/cloudstack/pull/253)

Thanks @koushik-das for your reply and the fix, can you share any test results? LGTM.

List service offerings for systemvms and verify there should be two ... === TestName: test_01_list_system_offerngs_1_consoleproxy | Status : SUCCESS ===
ok
List service offerings for systemvms and verify there should be two ... === TestName: test_01_list_system_offerngs_2_secondarystoragevm | Status : SUCCESS ===
ok
List service offerings for systemvms and verify there should be two ... === TestName: test_01_list_system_offerngs_3_domainrouter | Status : SUCCESS ===
ok
List service offerings for systemvms and verify there should be two ... === TestName: test_01_list_system_offerngs_4_internalloadbalancervm | Status : SUCCESS ===
ok
Check if system vms are honouring zone level setting ... === TestName: test_02_system_vm_storage_1_consoleproxy | Status : SUCCESS ===
ok
Check if system vms are honouring zone level setting ... === TestName: test_02_system_vm_storage_2_secondarystoragevm | Status : SUCCESS ===
ok
update global setting with system offering and check if it is being ... === TestName: test_03_custom_so_1_consoleproxy | Status : SUCCESS ===
ok
update global setting with system offering and check if it is being ... === TestName: test_03_custom_so_2_secondarystoragevm | Status : SUCCESS ===
ok
Check if router  vm is honouring zone level setting ... === TestName: test_04_router_vms | Status : SUCCESS ===
ok

Ran 9 tests in 3201.200s
OK

authors: bhaisaab koushik-das nitt10prashant


text:
Log4j2 refactor cloud engine orchestration module

Description
With the new version of Log4j there is some space for improving log readability using the new features. This PR refactors the logs from the cloud-orchestration module.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[X] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[X] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

How Has This Been Tested?
I have done some operations on cloudstack to induce the generation of the logs
No problems or changes to the log output were found

Codecov Report
Attention: Patch coverage is 27.91328% with 266 lines in your changes are missing coverage. Please review.

Project coverage is 30.80%. Comparing base (a5508ac) to head (f323822).
Report is 562 commits behind head on main.

:exclamation: Current head f323822 differs from pull request most recent head b677fd4. Consider uploading reports for the commit b677fd4 to get more accurate results

Files
Patch %
Lines

...cloud/agent/manager/ClusteredAgentManagerImpl.java
7.89%
105 Missing :warning:

...java/com/cloud/agent/manager/AgentManagerImpl.java
21.34%
70 Missing :warning:

...stack/engine/orchestration/VolumeOrchestrator.java
30.35%
39 Missing :warning:

...ain/java/com/cloud/agent/manager/AgentAttache.java
31.81%
14 Missing and 1 partial :warning:

...va/com/cloud/agent/manager/DirectAgentAttache.java
47.61%
11 Missing :warning:

...com/cloud/agent/manager/ClusteredAgentAttache.java
0.00%
9 Missing :warning:

...tack/engine/orchestration/NetworkOrchestrator.java
83.33%
5 Missing :warning:

...ack/engine/orchestration/DataMigrationUtility.java
0.00%
3 Missing :warning:

...tack/engine/orchestration/StorageOrchestrator.java
0.00%
3 Missing :warning:

...a/com/cloud/agent/manager/SynchronousListener.java
33.33%
1 Missing and 1 partial :warning:

... and 3 more

Additional details and impacted files
@@              Coverage Diff              @@
##               main    #8742       +/-   ##
=============================================
+ Coverage     13.17%   30.80%   +17.63%
- Complexity     9214    33528    +24314
=============================================
  Files          2725     5397     +2672
  Lines        258235   379186   +120951
  Branches      40249    55175    +14926
=============================================
+ Hits          34013   116825    +82812
- Misses       219913   246935    +27022
- Partials       4309    15426    +11117

Flag
Coverage Δ

simulator-marvin-tests
24.27% <27.37%> (?)

uitests
4.34% <ø> (?)

unit-tests
16.89% <1.08%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

CLGTM, hope I didn't miss anything 😅

that you'll have to test ;)
the build fails @KlausDornsbach in engine according to https://github.com/apache/cloudstack/actions/runs/8757275560/job/24037204263?pr=8742#step:7:17606 , afraid you'll have to take another look :(

Packaging result [SF]: ✖ el7 ✖ el8 ✖ el9 ✖ debian ✖ suse15. SL-JID 9356

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 9370

@blueorangutan test

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

authors: DaanHoogland KlausDornsbach blueorangutan codecov-commenter


text:
Miscellaneous (cherry-picked) improvements

Various improvements highlighted by the IDE. The improvements are cherry-picked where it makes sense. Each improvement category is grouped as a single commit.

LGTM

@arturobernalg I kept them separate so its remove/rebase a specific change.

Will wait for #123 to avoid merge conflicts/duplicate work.

@singhbaljit, can you rebase and fix conflicts here? We can then merge this in.

@darkma773r its been rebased.

@darkma773r its been rebased.

Oops. Guess I missed that.

Oops. Guess I missed that.

authors: arturobernalg darkma773r singhbaljit

repo: apache/commons-geometry

license_source: bigquery


text:
true && false String constant

create true && false String constant in order to re use variable, and make the code more readable

Coverage remained the same at 94.957% when pulling be5c63a4cdd0538ec72fe7cb15448c048603ea4b on arturobernalg:feature/boolean_string_const into 5689c91cf2a3a27f0a3a8a362857f2cf0919d4f6 on apache:master.

authors: arturobernalg coveralls

repo: apache/commons-lang


text:
refactor!: Cleanup & dependency modernization

Motivation and Context

coho is... unloved 😬
Description

Update several dependencies and replace some with built-in NodeJS APIs
Bump the minimum NodeJS version to 20.14.0+
(this is a dev tool not intended for public use, so that should be okay)
Remove the code that tries to link Jira ticket IDs since we don't use Jira anymore
Replace the code that fetches RAT with fetch()
Replace q with standard Promises
Get rid of almost all the shelljs uses, still a few to clean up around sed for versioning
Fixes GH-239

There are some other quick wins that should be possible, but they are extensive enough that I didn't want to do them all in a single unreviewable PR. Things like getting rid of jira-client (see https://github.com/apache/cordova-coho/pull/236) and replacing the use of request in the github stats with fetch(). Need to investigate why newer versions of inquirer don't seem to work properly.
Testing

All existing unit tests pass (all very few of them)
Successfully tested coho verify-archive
Successfully tested coho audit-license-headers
Successfully tested coho check-license
Successfully tested coho verify-tags
Successfully tested coho update-release-notes
Successfully tested coho last-week
Successfully tested coho list-pulls
Successfully tested coho version
Successfully tested coho repo-update

Checklist

[x] I've run the tests to see all new and existing tests pass
[x] If this Pull Request resolves an issue, I linked to the issue in the text above (and used the correct keyword to close issues using keywords)

Codecov Report
Attention: Patch coverage is 71.42857% with 24 lines in your changes missing coverage. Please review.

Project coverage is 58.21%. Comparing base (53ac7ad) to head (51e8856).

Files with missing lines
Patch %
Lines

src/superspawn.js
60.71%
22 Missing :warning:

src/apputil.js
66.66%
1 Missing :warning:

src/audit-license-headers.js
92.30%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #327      +/-   ##
==========================================
- Coverage   58.80%   58.21%   -0.59%
==========================================
  Files           8        8
  Lines         500      493       -7
==========================================
- Hits          294      287       -7
  Misses        206      206

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter dpogue

repo: apache/cordova-coho