text:
Allow deletion of system VM templates

Description
Currently, ACS does not allow the deletion of system VM templates. However, it might be of interest to the operator to delete templates from older versions that are no longer in use.
A change was applied to remove the restriction on the deletion of system VM templates, allowing operators to delete such templates.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[x] Minor

How Has This Been Tested?
I applied the changes and successfully deleted old system VM templates

@GaOrtiga
If you update the template to USER, then you will be able to remove it.
this will prevent some human mistakes

@GaOrtiga If you update the template to USER, then you will be able to remove it. this will prevent some human mistakes

@GaOrtiga @DaanHoogland
do you agree and close this PR ?

@GaOrtiga If you update the template to USER, then you will be able to remove it. this will prevent some human mistakes

@weizhouapache I understand that there is a way to circumvent this restriction, however, since this would only be possible to Root Admins, I believe deleting it directly should still be an option. To address your concerns about this change, I could make the force flag necessary when the template's type is SYSTEM.
Would this measure be enough to offset the concerns regarding this implementation?

@GaOrtiga @DaanHoogland
sorry this PR is not needed in my opinion

@GaOrtiga as a user wants this convenience. Even when I agree that it is not vital, that would be enough for it to be needed.

Is there any more pressing objection @weizhouapache?
Alternatively, is there a modus operandus that would make for a more acceptable change?

@GaOrtiga as a user wants this convenience. Even when I agree that it is not vital, that would be enough for it to be needed.

Is there any more pressing objection @weizhouapache?
Alternatively, is there a modus operandus that would make for a more acceptable change?

@DaanHoogland
SYSTEM template is very important for cloudstack environments. We need to be very careful to remove it
a force flag is not enough to me.
since we already have a way to delete a SYSTEM template (see https://github.com/apache/cloudstack/pull/8556#issuecomment-1906168608), this change is unnecessary.
my question to @GaOrtiga @DaanHoogland
is there special use case that we must be able to delete a SYSTEM template in one step ?

@DaanHoogland @GaOrtiga
let me explain a bit more
users might delete the unused SYSTEM template one or twice every year by average. It takes just few minutes.
However, with this PR, they will be able to delete SYSTEM template by a single API/command.
If the SYSTEM template is removed by mistake, it will be a critical issue .
can you compare the pros and cons ?

@GaOrtiga @DaanHoogland sorry this PR is not needed in my opinion

Having a workaround should not be a reason to restrict an operation. If the operator does not know that you can circumvent this restriction they may feel obliged to perform the operation directly in the DB, especially since the exception thrown is very vague and editing the template's type to make it deletable is very non-intuitive.

@GaOrtiga as a user wants this convenience. Even when I agree that it is not vital, that would be enough for it to be needed.

Is there any more pressing objection @weizhouapache?
Alternatively, is there a modus operandus that would make for a more acceptable change?

@DaanHoogland SYSTEM template is very important for cloudstack environments. We need to be very careful to remove it a force flag is not enough to me.
since we already have a way to delete a SYSTEM template (see #8556 (comment)), this change is unnecessary.
my question to @GaOrtiga @DaanHoogland is there special use case that we must be able to delete a SYSTEM template in one step ?

The use case would be any operator that wants to delete SYSTEM templates and does not know about the workaround, since it's not really something self-explanatory.

The use case would be any operator that wants to delete SYSTEM templates and does not know about the workaround, since it's not really something self-explanatory.

@GaOrtiga
this can be fixed by a PR with changes in the error message.

@GaOrtiga @DaanHoogland sorry this PR is not needed in my opinion

Having a workaround should not be a reason to restrict an operation. If the operator does not know that you can circumvent this restriction they may feel obliged to perform the operation directly in the DB, especially since the exception thrown is very vague and editing the template's type to make it deletable is very non-intuitive.

If the solution is not clear, make it clear then. for example
diff --git a/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java b/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
index b886f0868f6..6456ca452ed 100644
--- a/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
+++ b/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
@@ -749,7 +749,7 @@ public class HypervisorTemplateAdapter extends TemplateAdapterBase {
         TemplateProfile profile = super.prepareDelete(cmd);
         VMTemplateVO template = profile.getTemplate();
         if (template.getTemplateType() == TemplateType.SYSTEM) {
-            throw new InvalidParameterValueException("The DomR template cannot be deleted.");
+            throw new InvalidParameterValueException("The SYSTEM template cannot be deleted. Please update template type to USER and retry.");
         }
         checkZoneImageStores(profile.getTemplate(), profile.getZoneIdList());
         return profile;

What do you think of @weizhouapache 's last sugestion @GaOrtiga ? would this satisfy both your requirements?

@GaOrtiga @DaanHoogland sorry this PR is not needed in my opinion

Having a workaround should not be a reason to restrict an operation. If the operator does not know that you can circumvent this restriction they may feel obliged to perform the operation directly in the DB, especially since the exception thrown is very vague and editing the template's type to make it deletable is very non-intuitive.

If the solution is not clear, make it clear then. for example
diff --git a/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java b/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
index b886f0868f6..6456ca452ed 100644
--- a/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
+++ b/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
@@ -749,7 +749,7 @@ public class HypervisorTemplateAdapter extends TemplateAdapterBase {
         TemplateProfile profile = super.prepareDelete(cmd);
         VMTemplateVO template = profile.getTemplate();
         if (template.getTemplateType() == TemplateType.SYSTEM) {
-            throw new InvalidParameterValueException("The DomR template cannot be deleted.");
+            throw new InvalidParameterValueException("The SYSTEM template cannot be deleted. Please update template type to USER and retry.");
         }
         checkZoneImageStores(profile.getTemplate(), profile.getZoneIdList());
         return profile;

@weizhouapache
While the proposed implementation provides clarity, it's not intuitive. Since changes are already necessary, instead of merely guiding users through an arbitrary process, the aim should be to enhance overall code understanding and user experience.
The force flag should be more than enough to address any remaining concerns. This flag serves as a clear indicator that the operation requires careful execution and is not intended for routine use.
Besides, deleting the current SYSTEM template does not pose a significant risk to the environment, as it can be reuploaded.

@GaOrtiga @DaanHoogland sorry this PR is not needed in my opinion

Having a workaround should not be a reason to restrict an operation. If the operator does not know that you can circumvent this restriction they may feel obliged to perform the operation directly in the DB, especially since the exception thrown is very vague and editing the template's type to make it deletable is very non-intuitive.

If the solution is not clear, make it clear then. for example
diff --git a/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java b/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
index b886f0868f6..6456ca452ed 100644
--- a/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
+++ b/server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java
@@ -749,7 +749,7 @@ public class HypervisorTemplateAdapter extends TemplateAdapterBase {
         TemplateProfile profile = super.prepareDelete(cmd);
         VMTemplateVO template = profile.getTemplate();
         if (template.getTemplateType() == TemplateType.SYSTEM) {
-            throw new InvalidParameterValueException("The DomR template cannot be deleted.");
+            throw new InvalidParameterValueException("The SYSTEM template cannot be deleted. Please update template type to USER and retry.");
         }
         checkZoneImageStores(profile.getTemplate(), profile.getZoneIdList());
         return profile;

@weizhouapache
While the proposed implementation provides clarity, it's not intuitive. Since changes are already necessary, instead of merely guiding users through an arbitrary process, the aim should be to enhance overall code understanding and user experience.

Can you tell me how frequent you deleted the SYSTEM template?

The force flag should be more than enough to address any remaining concerns. This flag serves as a clear indicator that the operation requires careful execution and is not intended for routine use.

force flag is being used by other purpose.

Besides, deleting the current SYSTEM template does not pose a significant risk to the environment, as it can be reuploaded.

That is not true. Without SYSTEM template, users cannot create new networks and recreate VRs/system vms. Do you think if it is a critical issue if it happens on your production?

@weizhouapache

Can you tell me how frequent you deleted the SYSTEM template?

Even if it's not something that is done frequently, it being purposely less intuitive impacts user experience.

That is not true. Without SYSTEM template, users cannot create new networks and recreate VRs/system vms. Do you think if it is a critical issue if it happens on your production?

While it is true that the environment will not function properly without the SYSTEM template, it can be easily reuploaded as soon as the operator realizes their mistake.

force flag is being used by other purpose.

I could add a new parameter, maybe system to make it very clear on what's the intent of the flag. This would make it so that these templates can be deleted without the need for workarounds, while adding extra security. This would also make it so that unused templates can be deleted only using the system flag, with the exception of the current template being used by the systemvms, which would require the force flag aswell as the system.

@weizhouapache

Can you tell me how frequent you deleted the SYSTEM template?

Even if it's not something that is done frequently, it being purposely less intuitive impacts user experience.

We need to compare the pros and cons ...
Comparing with possible unexpected removal of SYSTEM template, the 2 steps removal is a very minor issue.

That is not true. Without SYSTEM template, users cannot create new networks and recreate VRs/system vms. Do you think if it is a critical issue if it happens on your production?

While it is true that the environment will not function properly without the SYSTEM template, it can be easily reuploaded as soon as the operator realizes their mistake.

Are you operating a production ? If yes, you would not say this ...

force flag is being used by other purpose.

I could add a new parameter, maybe system to make it very clear on what's the intent of the flag. This would make it so that these templates can be deleted without the need for workarounds, while adding extra security. This would also make it so that unused templates can be deleted only using the system flag, with the exception of the current template being used by the systemvms, which would require the force flag aswell as the system.

IMHO do not make things too complicated...

I could add a new parameter, maybe system to make it very clear on what's the intent of the flag. This would make it so that these templates can be deleted without the need for workarounds, while adding extra security. This would also make it so that unused templates can be deleted only using the system flag, with the exception of the current template being used by the systemvms, which would require the force flag aswell as the system.

Although I think it is unnecessary to add an API parameter which reduces the steps from 2 to 1, for an operation which run once or twice a year, it looks ok to me as it does not lead to any risk.
If you want to implement it, go ahead. we have spent too much time on this topic.

@GaOrtiga
will you support it on UI ?
API is tested ok
(localcloud) 🐱 > delete template id=d0b0a4db-1125-4a99-8cca-cf7934bbc19b
{
  "account": "admin",
  "accountid": "75837c93-aff0-11ee-8ad9-1e00940002db",
  "cmd": "org.apache.cloudstack.api.command.user.template.DeleteTemplateCmd",
  "completed": "2024-01-26T20:25:33+0000",
  "created": "2024-01-26T20:25:33+0000",
  "domainid": "476c9122-aff0-11ee-8ad9-1e00940002db",
  "domainpath": "ROOT",
  "jobid": "b9ba7dd4-58a4-4647-936a-44ca0278532f",
  "jobinstanceid": "d0b0a4db-1125-4a99-8cca-cf7934bbc19b",
  "jobinstancetype": "Template",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 431,
    "errortext": "Could not delete template as it is a SYSTEM template and isSystem is set to false."
  },
  "jobresultcode": 431,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "7584f1d6-aff0-11ee-8ad9-1e00940002db"
}
🙈 Error: async API failed for job b9ba7dd4-58a4-4647-936a-44ca0278532f
(localcloud) 🐱 > delete template id=d0b0a4db-1125-4a99-8cca-cf7934bbc19b issystem=true
{
  "success": true
}

@GaOrtiga
will you support it on UI ?

@weizhouapache No I will not. In my opinion, adding it to the UI would risk some of the security concerns that you pointed out earlier in the discussion.
Also, I can not think of a use case that would require it as of now, and if one eventually comes up, we can implement it to the UI when needed.

ok @GaOrtiga
so admins have to use api or cmk, right ? this does not help admins a lot.

ok @GaOrtiga so admins have to use api or cmk, right ?

Yes.

this does not help admins a lot.

Since some security concerns have been brought up, I felt that limiting it to the API would be a good compromise in order to implement this functionality while avoiding further security discussions, since, as you mentioned, we have spent too much time on this. However, I would be happy to implement this in the UI if the community sees it as beneficial.
@weizhouapache @DaanHoogland What are your opinions on this?

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 8465

I think both sides of the discussion have valid points and this will have to prove itself in production. I am ok with the API only implementation.

I think both sides of the discussion have valid points and this will have to prove itself in production. I am ok with the API only implementation.

thanks @DaanHoogland
I am ok with both. that's say, I am ok with this PR.

@rohityadavcloud

Would this also deletion of systemvmtemplate current in use?
If that's the case, can we add a check so it won't allow deleting systemvmtemplate in use

It will not allow the deletion of the current template unless both the isSystem and the forced flag are passed.

Does it need some UI changes, so admins can delete systemvmtemplates (not in use) from the UI?

Given some of the previous discussions regarding this implementation, mainly about security, I believe this PR should focus solely on the backend. If, in the future, we decide that UI changes are beneficial, we can make a new PR to implement it.

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 8686

@blueorangutan test

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[SF] Trillian test result (tid-9263)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 53810 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8556-t9263-kvm-centos7.zip
Smoke tests completed. 126 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_02_trigger_shutdown
Failure
341.61
test_safe_shutdown.py

test_03_secured_to_nonsecured_vm_migration
Error
276.26
test_vm_life_cycle.py

test_05_rvpc_multi_tiers
Failure
500.57
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
500.59
test_vpc_redundant.py

@blueorangutan package

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 9279

@DaanHoogland @sureshanaparti @rohityadavcloud @shwstppr could we run the CI here?

@blueorangutan test

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[SF] Trillian test result (tid-9858)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 53146 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8556-t9858-kvm-centos7.zip
Smoke tests completed. 126 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_01_events_resource
Error
306.50
test_events_resource.py

test_01_events_resource
Error
306.51
test_events_resource.py

test_04_deploy_vm_for_other_user_and_test_vm_operations
Failure
88.70
test_network_permissions.py

ContextSuite context=TestNetworkPermissions>:teardown
Error
1.43
test_network_permissions.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Failure
389.98
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Failure
979.53
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Failure
468.55
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
468.58
test_vpc_redundant.py

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 10461

@blueorangutan package

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✖ el8 ✖ el9 ✖ debian ✖ suse15. SL-JID 10638

Packaging result [SF]: ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 10652

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-11074)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 50926 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8556-t11074-kvm-ol8.zip
Smoke tests completed. 139 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

Merging based on approvals, CI result and manual tests (https://github.com/apache/cloudstack/pull/8556#issuecomment-1912653892)

authors: DaanHoogland GaOrtiga JoaoJandre blueorangutan weizhouapache


text:
CLOUDSTACk-9002: VM deployment is successful even when dhcp entry com…

…mand fails - Fixed
Reason: The return value of the call to accept() function in the applyRules() function of BasicNetworkTopology.java was not checked for success or failure. As a result even if it fails, exception was not thrown and VM deployment went ahead without any errors.
Fix: Added the necessary checks.

Seems good to me, but while we are working on this. Any way we can Unit Test this? Looking at the code it's possible.
If you could just write a test to validate the change you made we have a beginning.

LGTM, based on a set of tests that I run on this branch (which I rebased myself first):
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
component/test_vpc_redundant.py \
component/test_routers_iptables_default_policy.py \
component/test_routers_network_ops.py \
component/test_vpc_router_nics.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py \
smoke/test_network.py

Result:
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Stop existing router, add a PF rule and check we can access the VM ... === TestName: test_isolate_network_FW_PF_default_routes | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_RVR_Network_FW_PF_SSH_default_routes | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : FAILED ===
FAIL
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : FAILED ===
FAIL
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : FAILED ===
FAIL
----------------------------------------------------------------------
Ran 27 tests in 11421.526s

FAILED (failures=3)

The 3 errors at the bottom are due to CLOUDSTACK-8991 and unrelated to this PR.
And:
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_privategw_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_vpc_vpn.py \
smoke/test_service_offerings.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

Result:
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 44 tests in 8799.943s

OK

Testing this PR now.
Cheers,
Wilder

Ping @remibergsma @karuturi @kansal
Partial test results:

Hardware required: false
Management Server + MySQL on CentOS 7.1
One KVM host on CentOS 7.1
ACS Agent + Common RPMs built from source

Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... SKIP: At least two hosts should be present in the zone for migration
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 40 tests in 7705.217s

OK (SKIP=1)
(END)

Cheers,
Wilder

@wilderrodrigues did you run more tests?

LGTM I agree with @wido that unit tests are desirable but I will not block this for them. It will automatically block on regression.

LGTM :+1:  based on the 40 tests I execute + the 71 tests @remibergsma executed!
Cheers,
Wilder

source: gharchive/pull-request

authors: DaanHoogland kansal remibergsma wido wilderrodrigues

repo: apache/cloudstack

license_type: permissive

license_source: github-api


text:
IO-658 Upgrade to JUnit Pioneer v0.5.4

Handles https://issues.apache.org/jira/browse/IO-658
Also based upon https://issues.apache.org/jira/browse/IO-656 branch so pr https://github.com/apache/commons-io/pull/104

Coverage decreased (-0.07%) to 89.408% when pulling 7cbe1a5d817d5658ca457568a7e17fc7db84ebd2 on nhojpatrick:IO-658 into 11f0abe7a3fb6954b2985ca4ab0697b2fb489e84 on apache:master.

No longer applicable as the version in master is now 0.5.6.

authors: coveralls garydgregory nhojpatrick

repo: apache/commons-io

license_source: bigquery


text:
refactor: Migrate to AndroidX

Platforms affected
Android
What does this PR do?
Migrate to AndroidX (resolves parts of https://github.com/apache/cordova-plugin-camera/issues/405). This plugin used a single class from the (now) deprecated Android Support Library:

The dependency to the com.android.support:support-v4 artifact was replaced with the newer androidx.legacy:legacy-support-v4 as suggested by the AndroidX migration guides: https://developer.android.com/jetpack/androidx/migrate
All use of android.support.v4.content.FileProvider was replaced with then newer androidx.core.content.FileProvider as suggested by the AndroidX migration guide.
The plugin property ANDROID_SUPPORT_V4_VERSION was replaced with a more accurately named one called ANDROIDX_LEGACY_SUPPORT_V4_VERSION which now defaults to 1.0.0.

What testing has been done on this change?
Manual testing has been performed as part of reporting to the initial issue that lead to this PR: https://github.com/apache/cordova-android/issues/565
Checklist

[ ] Reported an issue in the JIRA database
[ ] Commit message follows the format: "CB-3232: (android) Fix bug with resolving file paths", where CB-xxxx is the JIRA ID & "android" is the platform affected.
[ ] Added automated test coverage as appropriate for this change.

@davidschreiber are you still interested in getting this fixed?
I personally do not have so much time to work on it.
I would personally recommend that you raise each of the changes needed in separate PRs for us to review.
I would personally favor that we start a new major version that drops Node.js 6 and maybe even Node.js 8 to avoid the build problems.

Well, a few months later the CI problems are fixed and this successfully passes. Could you have another look @davidschreiber if the changes make sense? How would one properly test this manually?

This is great @janpio! Thanks a lot for the update. Work on this PR has long been overdue on my end 🙈
The simplest way to test is, probably is to create a sample project and use cordova-plugin-androidx to enable AndroidX and Jetifier inside the project.
Adding the currently stable version of cordova-plugin-camera will yield an error after AndroidX was enabled inside the project (in this specific case several classes can't be found because the Support Library dependencies are no longer present since Gradle replaces them with AndroidX):
The easiest way to manually test the plugin is using the projects tests/ sub projects. You can quickly create a test application using this:
# Clone the project, since cordova CLI does not support plugins from sub directories.
git clone https://github.com/davidschreiber/cordova-plugin-camera.git && cd cordova-plugin-camera && git checkout davidschreiber/androidx-migration && cd ..
# Create a test project.
cordova create androidx-camera-test com.example.cordovacameratest androidx-camera-test
cd androidx-camera-test
# Add Android platform support
cordova platform add android
# Add the test dependencies
cordova plugin add cordova-plugin-test-framework
cordova plugin add ../cordova-plugin-camera/tests
# Add the plugin itself
cordova plugin add ../cordova-plugin-camera
# Enable AndroidX inside the project
cordova plugin add cordova-plugin-androidx
# Set the entrypoint of the app to start the tests.
sed -i '' 's/index.html/cdvtests\/index.html/g' config.xml
# Build and run
cordova run android

You should now see the Jasmine test framework and can run the automated and manual tests of the plugin and verify that stuff behaves as it should.
Alternatively, if you want to test the current stable version of the plugin, you can run it like so:
git clone https://github.com/apache/cordova-plugin-camera.git && cd cordova-plugin-camera && git checkout 4.1.0 && cd ..
cordova create androidx-camera-test com.example.cordovacameratest androidx-camera-test
cd androidx-camera-test
cordova platform add android
cordova plugin add cordova-plugin-test-framework
cordova plugin add ../cordova-plugin-camera/tests
cordova plugin add ../cordova-plugin-camera
cordova plugin add cordova-plugin-androidx
sed -i '' 's/index.html/cdvtests\/index.html/g' config.xml
cordova run android

This will yield a build error due to the sources of the stable version still referencing old Support Library classes
...
> Task :app:compileDebugJavaWithJavac FAILED
/Users/davidschreiber-ranner/temp/androidx-update-test/platforms/android/app/src/main/java/org/apache/cordova/camera/CameraLauncher.java:64: error: package android.support.v4.content does not exist
import android.support.v4.content.FileProvider;
...

I hope this helps to verify that the changes in this PR are good to go. Please let me know if I can do anything else here.

No, perfect explanation - will come back in a bit with my results.
(Ping @dpa99c for a check here, as this uses your plugin!)
(Semi offtopic: Should we add a test run that adds cordova plugin add cordova-plugin-androidx to all our plugins?)

Don't worry, we're performing tests for our own Cordova plugin like this too, so I already knew the procedure.
Adding an extra test for AndroidX support would definitely be a good step in the right direction — AndroidX is going to stay in the Android ecosystem.

🎁 Paramedic feature request being tracked in https://github.com/apache/cordova-paramedic/issues/189

Don't worry, we're performing tests for our own Cordova plugin like this too, so I already knew the procedure.

CI or manually? Public? Couldn't find any obvious CI config on the 2 Cordova plugin repos I know from you.

Manually and private. We haven't found any satisfying solution to automate tests for our plugin. We'd need some testing framework that invokes our plugin's APIs in JavaScript and performs assertions in Java and JavaScript (something we haven't figured out yet).

Contacted you via LinkedIn with my email address, so we can talk about that further.

Is it correct that using this plugin in an AndroidX enabled app would currently crash with these during build:
/tmp/tmp-3971dYpBCp3HDhgA/platforms/android/app/src/main/java/org/apache/cordova/camera/CameraLauncher.java:40: error: package android.support.v4.content does not exist
import android.support.v4.content.FileProvider;
                                 ^
/tmp/tmp-3971dYpBCp3HDhgA/platforms/android/app/src/main/java/org/apache/cordova/camera/CordovaUri.java:25: error: package android.support.v4.content does not exist
import android.support.v4.content.FileProvider;
                                 ^
/tmp/tmp-3971dYpBCp3HDhgA/platforms/android/app/src/main/java/org/apache/cordova/camera/FileProvider.java:21: error: package android.support.v4.content does not exist
public class FileProvider extends android.support.v4.content.FileProvider {}
                                                            ^
/tmp/tmp-3971dYpBCp3HDhgA/platforms/android/app/src/main/java/org/apache/cordova/camera/CameraLauncher.java:303: error: cannot find symbol
        this.imageUri = new CordovaUri(FileProvider.getUriForFile(cordova.getActivity(),
                                                   ^
  symbol:   method getUriForFile(Activity,String,File)
  location: class FileProvider
/tmp/tmp-3971dYpBCp3HDhgA/platforms/android/app/src/main/java/org/apache/cordova/camera/CameraLauncher.java:801: error: cannot find symbol
                        Uri tmpFile = FileProvider.getUriForFile(cordova.getActivity(),
                                                  ^
  symbol:   method getUriForFile(Activity,String,File)
  location: class FileProvider

And that after merging these changes, that would not be the case any more?

@brodybits This should be considered a BREAKING change because we are taking android.support.v4.content away and replacing it with Androidx equivalents?

Yes I would consider this to be a breaking change.

Yes, update to AndroidX should be considered a breaking change. You are also correct about the symbol errors you are seeing when using the stable version of this plugin in an AndroidX-enabled app.
I'm a bit torn about the idea of adding cordova-plugin-androidx (or cordova-plugin-androidx-adapter) as a dependency to this plugin. Doing so would impose a "single right way" to implement AndroidX in your Cordova application — which in my opinion does not exist. For example, developers could also manually enable AndroidX in their app or use different plugins that do the job via hooks.
On the other hand, from my experience, I can tell that most Cordova users wouldn't be able to identify AndroidX/Support Library conflicts in their apps, let alone know how to resolve them. While adding documentation would be a good first step, making sure they never run into these kinds of issues (i.e. by automatically transitioning their app to AndroidX using the aforementioned plugins) would definitely be helpful for most users.
If the plugin should depend on a specific plugins that enable AndroidX, I would argue that they should be official Apache Cordova plugins. (In other words, the AndroidX support should be provided by the framework itself.)

So we should probably move this discussion to cordova-android and find a more general solution?

As I just stumbled over this: https://github.com/EddyVerbruggen/nativescript-plugin-firebase/pull/1261/files On NativeScript plugins they seem to have a way to differentiate if they need AndroidX support or not, which makes it possible for the plugins to be used in both scenarios. MAybe this is possible here as well somehow?

authors: brodybits davidschreiber janpio

repo: apache/cordova-plugin-camera


text:
Mango: implicit coverage for _id and _rev during index selection

Overview
Mango requires that a JSON index can only be used to fulfil
a query if the "selector" field covers all of the fields the
index. For example, if an index contains ["a", "b"] but the
selector only requires field ["a"] to exist in the matching documents,
the index would not be valid for the query (because it only includes
documents containing both "a" and "b").
There is a special case here around built-in fields; _id and _rev
specifically, because they are guaranteed to exist in any matching
documents.
If a user declares an index ["a", "_id"], we can safely exclude "_id"
from the index coverage check, so a selector of {"a": "foo"} should be
able to use this index.
Prior to this commit, a user would have to alter the selector so that
it covered the "_id" field, e.g. {"a": "foo",
"_id": {"$exists": true}}. The commit removes the need to explicitly
cover _id or _rev fields in the query selector.
Testing recommendations
Tests have been added to the Mango test suite. To manually test, create an index which includes "_id" and/or "_rev" and another field. Check that calling _find (or _explain) with a selector that does not include "_id" or "_rev" but does reference the other indexed fields will use the index.
Related Issues or Pull Requests

Checklist

[ ] Code is written and works correctly;
[ ] Changes are covered by tests;
[ ] Documentation reflects the changes;

thanks @eiri - updated to remove the superfluous sort

repo: apache/couchdb


text:
Handle spurious 1009 (future_version) errors in couch_jobs pending

We already handle them in couch_jobs_type_monitor so let's do it in
couch_jobs:wait_pending as well. Recent fixes in FDB 6.2 didn't completely
fix the issue and there are still spurious 1009 errors dumped in the logs. They
seem to be benign as far as couch_jobs operation goes as type monitor code
already showed, so let's not pollute the logs with them.

I was mainly copy and pasting from https://apple.github.io/foundationdb/api-error-codes.html


text:
Allow to pass properties to Weld

Currently, there is no way to parametrise Weld when using org.apache.deltaspike.cdise.api.CdiContainer interface as CdiContainer.boot(Map<?,?>) does not pass parameters to Weld instance it creates.
CdiContainer cdiContainer = CdiContainerLoader.getCdiContainer();
cdiContainer.boot(params);

This PR adds passing properties to Weld by calling weld.setProperties() method.

Txs for the PR! applied and enhanced! setProperties will now get called via reflection on newer Weld versions.

@struberg where I can see it applied?

authors: struberg vanta

repo: apache/deltaspike


text:
[Improvement][API] Refactor monitor page API

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

Refactor monitor page API

Use case
No response
Related issues
related: #10257
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

WIP

source: gharchive/issue

repo: apache/dolphinscheduler


text:
[Bug] [script] dolphinscheduler-daemon.sh Status error

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

Query health status is always running.The task process has been queried and is not actually running
What you expected to happen
CLASS is not defined in the script, but it is used. I added
How to reproduce
I added the name corresponding to CLASS
Anything else
No response
Version
dev
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Could you describe your problems more specifically? I'm not quite understand it.

你能更具体地描述你的问题吗？我不太明白。
./bin/status-all.sh   have been in the state of running all the time. However, I have not started, and I have tried to execute ./stop-all.sh ，or so

authors: SbloodyS TangTangGrit


text:
[Bug] [Master] Scheduling time output failed when complementing numbers, reporting null pointer  version: 3.1.4

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
When I make up the schedule time is not generated, not all workflows are like this.

What you expected to happen
The scheduling time display is normal when complementing the number, and the null pointer error is no longer reported
How to reproduce
Manually click the workflow run button, click the complement number, and select the complement date according to the time control to execute.
Anything else
No response
Version
3.1.x
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[x]  I had searched in the issues and found no similar issues.

What happened
When I make up the schedule time is not generated, not all workflows are like this.
What you expected to happen
The scheduling time display is normal when complementing the number, and the null pointer error is no longer reported
How to reproduce
Manually click the workflow run button, click the complement number, and select the complement date according to the time control to execute.
Anything else
No response
Version
3.1.x
Are you willing to submit PR?

[ ]  Yes I am willing to submit a PR!

Code of Conduct

[x]  I agree to follow this project's Code of Conduct

https://test1.unicompayment.com/ito-bdp-web/auth/appl

Search before asking

[x]  I had searched in the issues and found no similar issues.

What happened
When I make up the schedule time is not generated, not all workflows are like this.
What you expected to happen
The scheduling time display is normal when complementing the number, and the null pointer error is no longer reported
How to reproduce
Manually click the workflow run button, click the complement number, and select the complement date according to the time control to execute.
Anything else
No response
Version
3.1.x
Are you willing to submit PR?

[ ]  Yes I am willing to submit a PR!

Code of Conduct

[x]  I agree to follow this project's Code of Conduct

Fixed.


text:
[Bug-RD][UI Next][V1.0.0-Alpha]Workflow delete error report

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
Workflow delete error report
http://ds2:12345/dolphinscheduler/ui/projects/5050147778880/workflow-definition

What you expected to happen
Workflow deletion is normal
How to reproduce
Workflow delete error report
Anything else
No response
Version
dev
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@XuXuClassMate Thank you very much for testing.
Meanwhile, that would be very helpful if there are some error logs in API server.

cannot reproduce, it works well on me

authors: Tianqi-Dotes XuXuClassMate lenboo


text:
[Improvement][Monitor] Show master && worker Busy Or Normal Status and Show Commands table list

close #15977

Please put some pictures in the document. @qingwli

OK, Will update later

Codecov Report
Attention: Patch coverage is 6.66667% with 42 lines in your changes are missing coverage. Please review.

Project coverage is 39.86%. Comparing base (ace20f9) to head (a6edf17).

:exclamation: Current head a6edf17 differs from pull request most recent head 4b7553e. Consider uploading reports for the commit 4b7553e to get more accurate results

Files
Patch %
Lines

...uler/api/service/impl/DataAnalysisServiceImpl.java
0.00%
33 Missing :warning:

...heduler/api/controller/DataAnalysisController.java
0.00%
8 Missing :warning:

...cheduler/api/service/impl/ExecutorServiceImpl.java
50.00%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #15978      +/-   ##
============================================
- Coverage     39.86%   39.86%   -0.01%
- Complexity     5060     5073      +13
============================================
  Files          1369     1369
  Lines         45635    45582      -53
  Branches       4869     4863       -6
============================================
- Hits          18193    18171      -22
+ Misses        25544    25511      -33
- Partials       1898     1900       +2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@SbloodyS  Done

Since this is compatible with 3.2.1, so I change the milestone to 3.2.2.

I will add some UT later

authors: codecov-commenter qingwli ruanwenjun


text:
[Feature-7346] [Python]Add workflow as code task type spark

close #7346

PTAL @zhongjiajie

I restart the failing CI test

PTAL @zhongjiajie

I restart the failing e2e test

Looking good.

Thank's for your review.

Thank's for your review.

No, I thank you, Kai, for the contribution. BTW, We still need to waiting CI passed 👀

Codecov Report

Merging #7968 (c5aeb18) into dev (1067444) will increase coverage by 0.19%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev    #7968      +/-   ##
============================================
+ Coverage     41.10%   41.30%   +0.19%
- Complexity     3705     3726      +21
============================================
  Files           637      637
  Lines         27058    27129      +71
  Branches       3073     3083      +10
============================================
+ Hits          11122    11205      +83
+ Misses        14862    14849      -13
- Partials       1074     1075       +1

Impacted Files
Coverage Δ

...he/dolphinscheduler/common/enums/SqoopJobType.java
0.00% <0.00%> (-88.89%)
:arrow_down:

...er/master/processor/queue/TaskResponseService.java
50.00% <0.00%> (-18.19%)
:arrow_down:

...rver/master/processor/queue/TaskResponseEvent.java
93.54% <0.00%> (-6.46%)
:arrow_down:

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...inscheduler/common/task/sqoop/SqoopParameters.java
74.00% <0.00%> (-2.00%)
:arrow_down:

...olphinscheduler/plugin/alert/email/MailSender.java
72.72% <0.00%> (-1.40%)
:arrow_down:

...cheduler/plugin/alert/dingtalk/DingTalkSender.java
34.42% <0.00%> (-0.53%)
:arrow_down:

...er/server/master/runner/WorkflowExecuteThread.java
9.59% <0.00%> (-0.32%)
:arrow_down:

...er/api/service/impl/TaskDefinitionServiceImpl.java
21.99% <0.00%> (-0.06%)
:arrow_down:

...atasource/api/provider/JDBCDataSourceProvider.java
2.22% <0.00%> (-0.06%)
:arrow_down:

... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1067444...c5aeb18. Read the comment docs.

authors: codecov-commenter devosend zhongjiajie


text:
fix fix be core dump caused by pushing down the double column predicate

Proposed changes
Issue Number: close #15678
related pr: #15558
Problem summary
The double column predicate is pushed down to the storage layer to cause a core dump, because there is no PredicateCreator corresponding to the double type. It looks like none of the double column predicates were pushed to the storage layer before, and I'm not quite sure why.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.07 seconds
load time: 475 seconds
storage size: 17121111998 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230107032959_clickbench_pr_75215.html

PTAL @xiaokang

PTAL @xiaokang

I will check it.

The root cause is that there is no implementation for float/doube in get_creator function in be/src/olap/predicate_creator.h. get_creator will return default value nullptr and cause create_predicate core in the following code.

template <PredicateType PT, typename ConditionType>
inline ColumnPredicate* create_predicate(const TabletColumn& column, int index,
const ConditionType& conditions, bool opposite,
MemPool* pool) {
return get_creator<PT, ConditionType>(column.type())
->create(column, index, conditions, opposite, pool);
}

create_predicate

Yes, but what confuses me is why the create_predicate of type double was not implemented before? In other words, the double type was not pushed down to the storage layer before because of any pitfalls?

The root cause is that there is no implementation for float/doube in get_creator function in be/src/olap/predicate_creator.h. get_creator will return default value nullptr and cause create_predicate core in the following code.

template <PredicateType PT, typename ConditionType>
inline ColumnPredicate* create_predicate(const TabletColumn& column, int index,
const ConditionType& conditions, bool opposite,
MemPool* pool) {
return get_creator<PT, ConditionType>(column.type())
->create(column, index, conditions, opposite, pool);
}

Yes, but what confuses me is why the create_predicate of type double was not implemented before? In other words, the double type was not pushed down to the storage layer before because of any pitfalls?

Yes, but what confuses me is why the create_predicate of type double was not implemented before? In other words, the double type was not pushed down to the storage layer before because of any pitfalls?

I have confirmed float/double type was not pushed down to the storage layer by checking the code in _normalize_conjuncts() and discussing with @moonming . The pitfalls may be the precision problem. I will do some research on other databases.

authors: Gabriel39 hello-stephen luozenglin xiaokang

repo: apache/doris


text:
feat: Simplify Agg GroupBy

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

[x] Does it affect the original behavior
[x] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
[Only-Test] Some pull requests cause memory errors when pipeline is not enabled.

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
decimalv2 add config to allow invalid decimalv2 literal

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
(fix)[compaction] Restarting the BE causes write amplification --- time series compaction

Proposed changes
Restarting the BE leads to increased write amplification during compaction.
In a time-series compaction scenario, persistent storage of cumulative point, restoring compaction point upon BE restart.
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.62 seconds
stream load tsv:          458 seconds loaded 74807831229 Bytes, about 155 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet:          29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select:          69.7 seconds inserted 10000000 Rows, about 143K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230706021823_clickbench_pr_173182.html

authors: csun5285 hello-stephen xiaokang


text:
improvement support load into mow table

Proposed changes
Issue Number: close #xxx
support load into mow table.
For the table of the unique model, after enabling the merge on write feature, the aggregate type is none, which causes spark load to fail to preprocess.
Therefore, when generating the configuration, process the table with merge on write enabled, and set the aggregate type of the non-key column to replace.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.3 seconds
stream load tsv:          530 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162244769 Bytes

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.15 seconds
stream load tsv:          535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select:          29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162013549 Bytes

authors: dataroaring gnehil hello-stephen


text:
fix the hyperlink for the create tpch table statement in the document

fix the hyperlink for the create tpch table statement in the document

run buildall

authors: XuJianxu XueYuhai


text:
script add paimon-oss/s3 jar in release package

Proposed changes
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix modify @@auto_commit column type to BIGINT in Nereids.

Proposed changes
#33282 only change the type in old planner.
This PR modify it in Nereids
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
[test](inverted index) nonConcurrent is added to the test case

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
feature support max_filter_ratio  (#35431)

bp #35431

run buildall


text:
[Regression](inverted index) fix build index case for wrong debug point #38112

cherry pick from #38112

run buildall

run buildall

run buildall


text:
fix Incorrect scan keys lead to wrong query results.

Proposed changes
mysql [doris_14555]>select * from table_9436528_3;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4                   | col6               | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 |    1 |  -82 |    1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 |    1 |   92 |    1 | 2024-02-16 04:37:37.00 |   23423423.0324234 | NULL |
| -100 |    0 |  -82 |    0 | 2023-11-11 10:49:43.00 |   840968969.872149 | NULL |

wrong result:
mysql [doris_14555]>select * from table_9436528_3 where col1 <= -100 and col2 in (true, false) and col3 = -82;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4                   | col6               | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 |    1 |  -82 |    1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 |    1 |   92 |    1 | 2024-02-16 04:37:37.00 |   23423423.0324234 | NULL |
+------+------+------+------+------------------------+--------------------+------+

run buildall

run buildall

run buildall

does 2.1 and 2.0 also have this problem?

run buildall

authors: cambyzju mrhhsg


text:
[Shared sink buffer test]

What problem does this PR solve?
only test this pr https://github.com/apache/doris/pull/44616
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall

run buildall

run buildall


text:
fix fix UnknownValue's reference in  simplify range rule #44637

cherry pick from #44637

run buildall


text:
How to delete older version segment after compact?

After the compact task completes, the new version of the segment is generated, but the old version of the segment is not automatically cleaned up.Do we have any good cleaning plan?

did you find a fix for this ?.

authors: damnMeddlingKid tomscut

repo: apache/druid


text:
The default `WHERE' filter for automatically generated SQL queries is returned

Returned the default WHERE filter to auto-generated SQL queries, which was lost after web-console updates in druid version 24.0.0 (#12919, #13169)

This PR has:

[x] been self-reviewed.
[ ] added documentation for new or modified features or behaviors.
[ ] a release note entry in the PR description.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[x] been tested in a test Druid cluster.

@nozjkoitop Could you please fix the linting issues.

Fixed


text:
Translation of dubboAsync_server.md

What is the purpose of the change
Translation of dubboAsync_server.md
Brief changelog
blog/en-us/dubboAsync_server.md
blog/zh-cn/dubboAsync_server.md（fix typos and simplified description）
site_config/blog.js
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[X] Format the pull request title like Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[X] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[X] Test your code locally by running docsite start, and make sure it works as expected.
[X] Make sure no files under build directory is added.
[X] If this contribution is large, please follow the Software Donation Guide.

thanks

authors: htynkn sofkyle

repo: apache/dubbo-website


text:
谁替代了RpcContext.getContext()

dubbo 我从2.X 版本升级到了3.0.2.1 版本看到 RpcContext.getContext() 已经弃用了。请问下现在隐式参数应该怎么传，我看文档上也没有更新

看下这个是不是你需要的 RpcContextAttachment

看下这个是不是你需要的 RpcContextAttachment

是不是消费者用：RpcContextAttachment.getClientAttachment().setAttachment() 传，生产者用:RpcContextAttachment.getServerAttachment().getAttachment() 取数据

可以在https://github.com/apache/dubbo-samples/tree/master/dubbo-samples-triple/ 中找一下相关样例

看下这个是不是你需要的 RpcContextAttachment

是不是消费者用：RpcContextAttachment.getClientAttachment().setAttachment() 传，生产者用:RpcContextAttachment.getServerAttachment().getAttachment() 取数据

3.0 文档我们正在完善文档中

https://dubbo.apache.org/zh/docs3-building/java-sdk/advanced-features-and-usage/rpc/context/
https://www.yuque.com/apache-dubbo/dubbo3/sw5s6w

authors: EarthChen chickenlj fantasylion wangchengming666

repo: apache/dubbo


text:
[Dependency] Exclude transitive spring dependencies from dubbo-all or dubbo-spring-boot-starter

… dubbo-spring-boot-starter
What is the purpose of the change
[Dependency] Exclude transitive spring dependencies from dubbo-all or dubbo-spring-boot-starter #14032
Brief changelog
Exclude transitive spring dependencies from  dubbo-spring-boot-starter
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Check #14137 please


text:
[Dubbo-6510] fix issue: dubboBootstrapApplicationListener is registered repeatedly

What is the purpose of the change
fix issue #6510: dubboBootstrapApplicationListener is registered repeatedly in ServiceClassPostProcessor
Brief changelog
ServiceClassPostProcessor.java
Verifying this change
dubboBootstrapApplicationListener is is registered repeatedly in ServiceClassPostProcessor, dubboBootstrapApplicationListener have bean registered through DubboBeanUtils.registerCommonBeans by DubboNamespaceHandler 、@EnableDubboConfig and @DubboComponentScan.

我有个疑问：在2.7.8版本中，为何自动配置注入方法registerCommonBeans 需要执行两次，我也看到之前的issue中说重复注入的问题，目前虽说启动是没有问题的。但是不理解为何需要执行两次的registerCommonBeans。

我有个疑问：在2.7.8版本中，为何自动配置注入方法registerCommonBeans 需要执行两次，我也看到之前的issue中说重复注入的问题，目前虽说启动是没有问题的。但是不理解为何需要执行两次的registerCommonBeans。

@yiocio hi, thanks for your contribution.
The issue has been fixed in #7000
Please feel free to reopen if you have any question.

authors: AlbumenJ MentosL yiocio


text:
[3.0] Fix concurrent modification in Invocation attribute

What is the purpose of the change
Brief changelog
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #9805 (65225a0) into 3.0 (7e09abd) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                3.0    #9805      +/-   ##
============================================
- Coverage     65.60%   65.56%   -0.04%
  Complexity      319      319
============================================
  Files          1208     1208
  Lines         52545    52545
  Branches       7972     7972
============================================
- Hits          34470    34451      -19
- Misses        14329    14343      +14
- Partials       3746     3751       +5

Impacted Files
Coverage Δ

.../main/java/org/apache/dubbo/rpc/RpcInvocation.java
68.78% <100.00%> (ø)

.../remoting/api/SingleProtocolConnectionManager.java
61.53% <0.00%> (-23.08%)
:arrow_down:

...nt/metadata/ServiceInstanceHostPortCustomizer.java
65.78% <0.00%> (-21.06%)
:arrow_down:

...ubbo/registry/client/AbstractServiceDiscovery.java
81.35% <0.00%> (-3.39%)
:arrow_down:

.../registry/zookeeper/ZookeeperServiceDiscovery.java
79.36% <0.00%> (-3.18%)
:arrow_down:

...exchange/support/header/HeaderExchangeHandler.java
61.06% <0.00%> (-1.77%)
:arrow_down:

...dubbo/remoting/exchange/support/DefaultFuture.java
87.17% <0.00%> (-0.86%)
:arrow_down:

...java/org/apache/dubbo/remoting/api/Connection.java
68.64% <0.00%> (-0.85%)
:arrow_down:

...pache/dubbo/registry/support/AbstractRegistry.java
75.08% <0.00%> (-0.69%)
:arrow_down:

...apache/dubbo/rpc/protocol/dubbo/DubboProtocol.java
61.51% <0.00%> (-0.66%)
:arrow_down:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7e09abd...65225a0. Read the comment docs.

authors: AlbumenJ codecov-commenter


text:
[Bug]invalid configuration of radiusAxis.axisLabel.inside and radiusAxis.axisTick.inside

Version
5.3.1
Link to Minimal Reproduction
No response
Steps to Reproduce
配置如下
option = {
angleAxis: {
clockwise:false
},
radiusAxis: {
type: 'category',
data: ['Mon', 'Tue', 'Wed', 'Thu'],
z: 10,
axisTick:{
inside:true,
},
axisLabel:{
inside:true
}
},
polar: {},
series: [
{
type: 'bar',
data: [1, 2, 3, 4],
coordinateSystem: 'polar',
name: 'A',
stack: 'a',
emphasis: {
focus: 'series'
}
}
],
};
Current Behavior
标签（axisLabel）和刻度线（axisTick）没有朝内
Expected Behavior
标签（axisLabel）和刻度线（axisTick）朝内
Environment
- OS:
- Browser:
- Framework:

Any additional comments?
No response

Any solutions to this? I am experiencing the same issue

repo: apache/echarts


text:
[Bug] Empty class member cause NullpointerException

Search before asking

[X] I had searched in the issues and found no similar issues.

Environment
Mac
EventMesh version
master
What happened
Runtime（Storage：Local Rocketmq ） + Http Source Connector

请求的路径应该是：curl发送请求->CloudEvent->ConnectRecord-> runtime(storage)
然后我发现是在CloudEvent->ConnectRecord转化的过程中position属性为null：

但是关于消息写入的offset和partition这些属性 应该是由storage的相关模块写入，但是我在CloudEventUtil里看到了关于这个的todo？
How to reproduce
run Runtime（Storage：Local Rocketmq ） + Http Source Connector
Debug logs
2024-02-16 00:11:21,099 DEBUG [vert.x-eventloop-thread-0] ZlibCodecFactory(ZlibCodecFactory.java:39) - -Dio.netty.noJdkZlibDecoder: false
2024-02-16 00:11:21,099 DEBUG [vert.x-eventloop-thread-0] ZlibCodecFactory(ZlibCodecFactory.java:42) - -Dio.netty.noJdkZlibEncoder: false
2024-02-16 00:11:21,170 INFO  [vert.x-eventloop-thread-0] HttpSourceConnector(HttpSourceConnector.java:100) - [HttpSourceConnector] Succeed to convert payload into CloudEvent. StatusCode=200
2024-02-16 00:11:21,174 INFO  [pool-3-thread-1] CloudEventUtil(CloudEventUtil.java:69) - handle receive events testdata

Exception in thread "pool-2-thread-1" java.lang.NullPointerException
	at org.apache.eventmesh.openconnect.offsetmgmt.api.storage.OffsetStorageWriterImpl.writeOffset(OffsetStorageWriterImpl.java:57)
	at java.util.HashMap.forEach(HashMap.java:1290)
	at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1507)
	at org.apache.eventmesh.openconnect.SourceWorker.commitOffsets(SourceWorker.java:346)
	at org.apache.eventmesh.openconnect.SourceWorker.startPollAndSend(SourceWorker.java:219)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct *

可以修复一下这个bug，对于http source connector不需要对应的offset，可以对空指针做一个兼容

repo: apache/eventmesh


text:
[AVRO-1603]: Avro generated fixed. Added pluginmanager to settings.

buildscript build.gradle
Description
Added avro plugin to build.gradle file.
Added pluginmnagement to settings.gradle file.
Added anotation "@SuppressWarnings" to
fineract-provider/src/main/java/org/apache/fineract/cob/COBBusinessStepServiceImpl.java Run method.
Changed syntax from build.gradle and settings.gradle.
Ignore if these details are present on the associated Apache Fineract JIRA ticket.
Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!

[x] Write the commit message as per https://github.com/apache/fineract/#pull-requests

[x] Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.

[x] Create/update unit or integration tests for verifying the changes made.

[x] Follow coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions.

[x] Add required Swagger annotation and update API documentation at fineract-provider/src/main/resources/static/legacy-docs/apiLive.htm with details of any API changes

[x] Submission is not a "code dump".  (Large changes can be made "in repository" via a branch.  Ask on the developer mailing list for guidance, if required.)

FYI our guidelines for code reviews are at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide.

And: the title... is the prefix "AVRO-" intended? That would indicate we are trying to solve another project's issue... please pick the right Fineract Jira ticket. Thanks.

authors: lllagger vidakovic

repo: apache/fineract


text:
When will opengauss or gaussdb be supported?

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
Opengauss and Huawei Cloud's gaussdb have been widely used in the financial industry in mainland China, and I hope that flink-cdc-connectors will have plans to support opengauss and gaussdb.
Solution
I see a two year old case here, not sure if it can work. Reference: https://github.com/dafei1288/flink-connector-opengauss
Alternatives
No response
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Closing this issue as it has been migrated to Apache Jira.

authors: PatrickRen wangbinhang

repo: apache/flink-cdc


text:
[FLINK-16916][serialization] Fix the logic of NullableSerializer#copy

What is the purpose of the change
Fix the wrong logic of NullableSerializer#copy,
currently, the logic is such as below:
public void copy(DataInputView source, DataOutputView target) throws IOException {
   boolean isNull = source.readBoolean();
   target.writeBoolean(isNull);
   if (isNull) {
      target.write(padding);
   }
   else {
      originalSerializer.copy(source, target);
   }
}

we forgot to skip paddings.length bytes when if the padding's length is not 0.
We can correct the logic such as below
public void copy(DataInputView source, DataOutputView target) throws IOException {
   boolean isNull = deserializeNull(source); // this will skip the padding values.
   target.writeBoolean(isNull);
   if (isNull) {
      target.write(padding);
   }
   else {
      originalSerializer.copy(source, target);
   }
}

Verifying this change
Harden the test of NullalbeSerialierTest#testSerializedCopyAsSequence by update the test data.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (yes)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

cc @aljoscha

Good fix! I will wait for Travis or Azure and then merge.

@flinkbot run azure

@flinkbot run travis

@flinkbot run azure

@flinkbot run travis

Thanks for the fix! 🍵

@aljoscha thanks for the review and merging.

repo: apache/flink


text:
[FLINK-24543][runtime] Avoid possible inconsistencies of ZookeeperSta…

This PR introduces idempotent transactions in ZooKeeperStateHandleStore#addAndLock method. In case of unstable connection we may actually retry transactions that have already succeeded and therefore encounter NodeAlreadyExists exception.
https://issues.apache.org/jira/browse/FLINK-24543

@tillrohrmann Thanks for the review! I've over-complicated the initial change-set. I think it's way better now 👍
Ready for another pass.


text:
[BP-1.14][FLINK-24506][config] Adds checkpoint directory to CheckpointConfig

Backport for PR #17972 excluding the assertj changes.

@flinkbot run azure

authors: XComp zentol


text:
[hotfix][misprint] Remove duplicate word version

What is the purpose of the change
There is a duplicate version in a phrase All Flink Scala APIs are deprecated and will be removed in a future Flink version version. The PR removes one extra word.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): ( no)
The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
The serializers: ( no)
The runtime per-record code paths (performance sensitive): ( no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
The S3 file system connector: ( no)

Documentation

Does this pull request introduce a new feature? ( no)
If yes, how is the feature documented? (not applicable )

@flinkbot run azure

@MartijnVisser sorry for the poke
could you please have a look once you have time?


text:
[FLINK-4452] TaskManager network buffer gauges

Adds gauges for the number of total and available TaskManager network memory segments.

Good idea.
+1 to merge this

authors: StephanEwen greghogan


text:
[FLINK-10050] Support allowedLateness in CoGroupedStreams

What is the purpose of the change
https://issues.apache.org/jira/browse/FLINK-10050
Add 'allowedLateness' method to coGroup and join streams API.
Brief change log

add 'allowedLateness' for CoGroupedStreams/JoinedStreams java and scala API
delegate calls to underlying WindowedStream (as for Trigger/Evictor scenario)

Verifying this change
This change is a trivial rework.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no

@aljoscha @tillrohrmann
guys, please can you take a glance on it?

Hi @EugeneYushin . This PR has no tests. Please add them.

Thanks for updating the PR @EugeneYushin . I will have a look today.

@kl0u I didn't want to ping you until Travis is finished.
Thanks for quick turnaround

I see one build from Travis has been finished with error:
10:15:13.087 [ERROR] Failed to execute goal on project flink-storm-examples_2.11: Could not resolve dependencies for project org.apache.flink:flink-storm-examples_2.11:jar:1.7-SNAPSHOT: Could not transfer artifact org.apache.storm:storm-starter:jar:1.0.0 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/storm/storm-starter/1.0.0/storm-starter-1.0.0.jar from central failed: Connection reset -> [Help 1]

Looks like it's not related to proposed changes.
What is the best way to deal with such kind of failures during CI?
Is there any way to rerun PR?

I would recommend to relaunch that specific build and see if it happens again.

Nice, CI checks have been passed. I think I have no access to Travis to relaunch builds (at least, I don't see any button there to perform such kind of operation).
Thanks for cooperation

@kl0u Great news!


text:
[FLINK-10790][network] Refactor StreamPartitioner implementations into runtime module

What is the purpose of the change
Before refactoring ChannelSelector interface mentioned in FLINK-10662, we found there are many duplicated implementations for ChannelSelector in stream and batch modes separately. The root cause is that they are locating in different modules, so can not reference each other.
The core runtime codes should be unified for both stream and batch modes as much as possible, then we can reduce the work overhead to maintain the duplicated codes, especially for the following refactor.
This PR is based on #7031 , then review the second commit for changes.
Next we would remove duplicated implementations for OutputEmitter and RoundRobinChannelSelector.
Brief change log

Refactor the existing partitioner package from flink-streaming-jave to flink-runtime.
Remove StreamRecord type explicitly from SerializationDelegate.
Modify related tests based on refactor changes.

Verifying this change
This change is already covered by existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (yes)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@pnowojski , this is the preparation for unifying implementations in both stream and batch modes. As I described in above purpose, I would submit the following PRs for de-duplication if you think this module refactor make sense.

@pnowojski , there are some problems for this issue now. I will update the codes later. Sorry for interrupting. I will cc you after ready.

I'm closing this PR @zhijiangW for now. If you will want to re-open it, please feel free to do so.

authors: pnowojski zhijiangW


text:
[FLINK-13206][sql client]replace use database xxx with use xxx in sql client parser

What is the purpose of the change
*Replace use database xxx with use xxx in sql client parser *
Brief change log

modify use database in sql client

Verifying this change
This change is already covered by existing tests, such as SqlCommandParserTest.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): ( no)
The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no )
The S3 file system connector: (no )

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable )

cc @bowenli86 @xuefuz @lirui-apache to review.

please also update the corresponding documentation in sql.md

I'm thinking maybe we can keep 'USE DATABASE' to match 'USE CATALOG', and make both 'USE DATABASE XXX' and 'USE XXX' be interpreted as using a database?
Besides, can we add some negative tests?

I like the idea that keeps 'USE DATABASE' to match 'USE CATALOG', maybe we should think it furthermore in the DDL/DML design and support in sql parser.

address the coments.cc @bowenli86

just ran a search and found catalog.md also need modification:
$ ag 'use database'
docs/dev/table/sql.md
325:USE DATABASE mydatabase;

docs/dev/table/catalog.md
66:Users can set current catalog and database via `TableEnvironment.useCatalog(...)` and `TableEnvironment.useDatabase(...)` in Table API, or `USE CATALOG ...` and `USE DATABASE ...` in Flink SQL.
340:Flink SQL> use database myDb;

I'm not convinced that we should introduce "use database database_name" grammar. I don't see a need, and keeping it simple will probably make our life easier.

just ran a search and found catalog.md also need modification:
$ ag 'use database'
docs/dev/table/sql.md
325:USE DATABASE mydatabase;

docs/dev/table/catalog.md
66:Users can set current catalog and database via `TableEnvironment.useCatalog(...)` and `TableEnvironment.useDatabase(...)` in Table API, or `USE CATALOG ...` and `USE DATABASE ...` in Flink SQL.
340:Flink SQL> use database myDb;

Sorry to miss these, I have correct them in the lastest commit.

cc @bowenli86 to review.

LGTM, will merge once CI passes

can you rerun the tests. seems to have some failures

can you rerun the tests. seems to have some failures

It's caused by the python module, I just rebase the master.

@bowenli86 I have tested in my traivs account, there is a failure in blink-planner that is not related to this commit.
CI BUILD

cc @bowenli86 again after addressing twalthr's comments.

authors: bowenli86 xuefuz zjuwangg


text:
[FLINK-12940][docs-zh] Translate Apache Cassandra Connector page into…

What is the purpose of the change
(This pull request to Translate "Cassandra Connector" page into Chinese.)
Brief change log

doc locates in flink/docs/dev/connectors/cassandra.zh.md

Verifying this change
(Please pick either of the following options)
no change to test.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

hi @klion26   Thank you very much, I addressed commits.

@XuQianJin-Stars thanks very much for the contribution, could you please rebase the master to resolve the conflict.


text:
[FLINK-13405][docs-zh] Translate "Basic API Concepts" page into Chinese

All nouns are singular
All Tuple are untranslated and capitalized
All Case Classes are untranslated and capitalized
@klion26 @xccui @wuchong PTAL

Thanks for the translation, @YngwieWang. This is quite a large effort.
The document is well translated in general. I've left some comments, most of which are minor language issues that may make it easier to understand for the readers. But there are a few issues regarding correctness, which I think should be fixed before merging.

Thanks very much for your review, it really helped me. I fix all of them except "bounded or unbounded". Do you think I'm right?

Thanks for addressing the comments, @YngwieWang. The translation looks good to me now. I think it should be good to merge this after resolving the conflicts.

@xintongsong , all right, thanks very much.

@wuchong ，谢谢。
我刚才检查，第319行的java没有改成scala，第538行的六没有改成七，请问如何修正?

@YngwieWang , yes, you can create a hotfix pull request.

authors: YngwieWang wuchong xintongsong


text:
GEODE-6818: Fix PutAllClientServerDistributedTest

GEODE-6818: Fix PutAllClientServerDistributedTest
Overhaul test and fix sources of flakiness (wait/notify and sleeps).

Replace object wait/notify with CountDownLatch(es)
Replace sleeps with Awaitility
Inline super-class and lots of methods
Remove unused or unnecessary code
Update deprecated API usage
Update to AssertJ and Awaitility

dunitrunner shows some failures. Closing PR.

repo: apache/geode


text:
[Improvement] Trino connector binary package shouldn't contain trino-jdbc jar

What would you like to be improved?
Trino connector binary package shouldn't contain trino-jdbc jar. We would better use the framework jar of Trino.
How should we improve?
Make the dependency provided.

Hi @jerqi ,
I want to work on this issue.
Could this task be assigned to me?
Best Regards,
Edward

@LindaSummer
Nice, I will assign it to you.

Hi @jerqi ,
I apologize for the interruption.
After updating the trino-jdbc dependency in the trino-connector to be used only during the compile and test phases, I encountered the following exception when launching Trino 435.
java.lang.NoClassDefFoundError: io/trino/jdbc/TrinoDriver.
Upon investigating, I noticed that the dependencies for TrinoServer are defined within the trino-main module, but the trino-jdbc dependency is not listed there.
I may be misunderstanding this issue. Could you please provide some guidance or suggestion on this issue? 😊
Best Regards,
Edward

Hi @jerqi ,
I apologize for the interruption.
After updating the trino-jdbc dependency in the trino-connector to be used only during the compile and test phases, I encountered the following exception when launching Trino 435. java.lang.NoClassDefFoundError: io/trino/jdbc/TrinoDriver.
Upon investigating, I noticed that the dependencies for TrinoServer are defined within the trino-main module, but the trino-jdbc dependency is not listed there.
I may be misunderstanding this issue. Could you please provide some guidance or suggestion on this issue? 😊
Best Regards, Edward

Sorry. It won't be fixed. It's my fault. I ever thought the Trino has included this dependency. We can close this issue directly. Thanks for your efforet.

authors: LindaSummer jerqi yuqi1129

repo: apache/gravitino