text:
CLOUDSTACK-8485: listAPIs are taking too long to return results

Removed regex. based search/replace of sensitive data on API response introduced as part of commit b0c6d4734724358df97b6fa4d8c5beb0f447745e
Added new response serializer to skip sensitive data from getting logged based on annotation present in resposne object fields
Added annotation (@LogLevel(Log4jLevel.Off)) to sensitive response object fields

Ran the following tests on simulator:
test_vm_life_cycle.py
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok

Ran 10 tests in 306.429s
OK
test_volumes.py
Download a Volume attached to a VM ... === TestName: test_03_download_attached_volume | Status : SUCCESS ===
ok
Delete a Volume attached to a VM ... === TestName: test_04_delete_attached_volume | Status : SUCCESS ===
ok
Detach a Volume attached to a VM ... === TestName: test_05_detach_volume | Status : SUCCESS ===
ok
Delete a Volume unattached to an VM ... === TestName: test_09_delete_detached_volume | Status : SUCCESS ===
ok

Ran 4 tests in 184.132s
OK
test_network.py
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok

Ran 4 tests in 783.726s
OK
test_routers.py
Test router internal advanced zone ... SKIP: Marvin configuration has no host credentials                            to check router services
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok

Ran 7 tests in 42.958s
OK (SKIP=1)
test_global_settings.py
test update configuration setting at zone level scope ... === TestName: test_UpdateConfigParamWithScope | Status : SUCCESS ===
ok

Ran 1 test in 0.127s
OK
test_resource_detail.py
Test volume detail ... === TestName: test_01_updatevolumedetail | Status : SUCCESS ===
ok

Ran 1 test in 11.492s
OK

I have tried to look for the existing API response objects based on some keyword search for sensitive fields. Please let me know if some response objects are left out.

registerUserKeys log based on this change
2015-11-02 17:15:19,061 INFO  [a.c.c.a.ApiServer] (2001334745@qtp-678426242-0:ctx-608cad64 ctx-6e00b6fc) (logid:bfb12e9d) (userId=2 accountId=2 sessionId=9g6dq5sjxlk7137t68hfkux8) 0:0:0:0:0:0:0:1 -- GET command=registerUserKeys&response=json&id=e2fdf8e4-73be-11e5-8882-249684d59f9c&_=1446464718702 200 {"registeruserkeysresponse":{"userkeys":{}}}

Jenkins build failed with the following error. I saw some other PRs with the same failure. So looks like some random issue.
[ERROR] SLF4J: Class path contains multiple SLF4J bindings.

@koushik-das I've seen it before and the actual error message is
#
# A fatal error has been detected by the Java Runtime Environment:
#
SUREFIRE-859: #  SIGSEGV (0xb) at pc=0xada0752c, pid=22300, tid=4136188736
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) Server VM (23.25-b01 mixed mode linux-x86 )
# Problematic frame:
# C  [libnet.so+0x352c]  _init+0x620
#
SUREFIRE-859: # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/jenkins/jenkins-slave/workspace/cloudstack-pull-analysis/server/hs_err_pid22300.log
#
SUREFIRE-859: # If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I think java needs updating on the slave (or we need to reconfigure the job to use a better version)

Usage of @LogLevel(Log4jLevel.Off) seems inappropriate to avoid logging sensitive information, why not still check the sensitive param/annotation to log or not log data?

@bhaisaab What are your reasons for saying that @LogLevel(Log4jLevel.Off) is inappropriate?
By sensitive param/annotation are you referring to the existing @APICommand annotation params requestHasSensitiveInfo/responseHasSensitiveInfo? The problem with this approach is that it doesn't tell about the specific field which is sensitive and so the entire response needs to be searched for sensitive data by means of regex which is very bad in terms of performance.

@bhaisaab @koushik-das Isn't there an annotation on field level for APIs as well? A change was accepted for that. I am looking for it and will update this comment.

@DaanHoogland @bhaisaab I also couldn't find any annotation on API fields for this. So ended up using the LogLevel which is also used in the agent commands.

The field level annotation was discussed as a prefered solution but in the end a regexp on field names was used. I am fine with re-using the LogLevel annotation. It does what is needed and the name suggests this use.

@koushik-das I think it's inappropriate as we're not marking those fields to note that they have hold sensitive information; but anyway that works

Also, it is possible to avoid serializing the objects by hand (like adding/appending {, >, <, } etc by hand?). Probably use any available library, and if that's not possible write unit tests to do some crud like comparisons?

@bhaisaab can you explain ? I like your comment about the annotation not being descriptive on the sensitivity of the field. So for the best solution we would have to write another annotation for that. It would be tchnical similar to the @LogLevel and the paramter of the @APICommand. thoughts?

@DaanHoogland we can simply use @LogLevel I simply shared that there is a slight chance on confusion in future, and do we then remove the requestHasSensitiveInfo/responseHasSensitiveInfo from Param/Apis throughout the codebase?

@bhaisaab no we shouldn't remove those both are used. I didn't understand the {, <, >, }  part.
The @LogLevel is used in the gson serialization and we can abuse it in the API but your comment makes sense. In the meanwhile that doesn't invalidate Koushik's change.
let's make the right decision

@DaanHoogland @bhaisaab requestHasSensitiveInfo/responseHasSensitiveInfo can be removed once this PR is accepted. I don't see any other use of these parameters.

@koushik-das You don't agree that the name LogLevel obfuscates that it is used to hide sensitive data?

@DaanHoogland I think LogLevel is a better/generic name to suppress a field from getting logged. Currently only sensitive field is getting annotated with it. Going forward some non-sensitive field may also be annotated if someone wants to prevent it from getting logged. Also since LogLevel is already used in agent commands, having an uniform approach throughout the code is always better and creates less confusion.

@DaanHoogland if you look at the changes or the code that takes in Object (Response object) and serializes it; we create the json or xml response string by manually adding the delimiters ({, }, <, >, : characters etc). I was wondering if there is a way to avoid that.

@bhaisaab Agree that the existing JSON and XML serialization can be improved but it is better to do it as a separate PR. For JSON, the standard Gson library is used. For XML we may have to check for a ASF compatible library that solves the purpose.

@koushik-das makes sense so we have a policy that for any sensitive field LogLevel(LogLevel.OFF) be used. We should add that as a comment to the enum to document it.

2015-11-04 14:23:51,076 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-34:ctx-0d2ee513 job-104/job-105 ctx-0d5c6252) Done executing VM work job: com.cloud.vm.VmWorkStart{"dcId
":1,"podId":1,"clusterId":1,"hostId":2,"rawParams":{"VmPassword":"<snip/>"},"userId":2,"accountId":2,"vmId":13,"handlerName":"VirtualMachineManagerImpl"}

And some other examples in there. I am looking further. @koushik-das this is going to take a lot of work to get right, right? Like this it is not done I think.

@koushik-das On second thought LogLevel can not be used I think: It is used for json serialization and one might have to serialize a password to be send to an agent while it may not be logged at the same time. We do need a separate annotation. this is not going to fly and hide sensitive data at the same time.

I think we should add isSensitive() to @Parameter and use that.
Also we should base this on gson 2+ and not on the version we are presently using.

@DaanHoogland About the issue you pointed out in com.cloud.vm.VmWorkStart serialization. Nice find but these are already existing issues. They surely needs fixing but I feel a separate effort is better. What do you say?
LogLevel can be safely used. If you look at the code there are 2 different GSON serializers - one for serialising the command that needs to be transmitted over the wire, and another is for logging purposes only. LogLevel is used in the latter one only.

@koushik-das I don't think you are right, command serialisation uses LogLevel as well. Look at com.cloud.agent.transport.RequestTest and tell me if you agree.

@DaanHoogland That's right. I had already mentioned about the agent commands in one of my previous comments

@koushik-das you said 'LogLevel is used in the latter one only.'. That is not true, I think. see com.cloud.agent.transport.RequestTest

@DaanHoogland If you look at ApiResponseGsonHelper.java in this PR there are GSon builders
private static final GsonBuilder s_gBuilder;
private static final GsonBuilder s_gLogBuilder;
Similarly for agent commands, if you look at GsonHelper.java there are 2 of them
protected static final Gson s_gson;
protected static final Gson s_gogger;

I am not fine with this change but must be honest and report regressions tests from the bubble: they all passed.
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 42 tests in 8451.564s

OK

and
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Stop existing router, add a PF rule and check we can access the VM ... === TestName: test_isolate_network_FW_PF_default_routes | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_RVR_Network_FW_PF_SSH_default_routes | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Check that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 29 tests in 13087.859s

OK

@DaanHoogland Updated based on our discussion last week

great @koushik-das I'll rerun the tests.

@DaanHoogland did you get a chance to test it?

@koushik-das sorry, no. I have not much test infra and run a lot of tests. Could you force push your changes, please. I'll review your last changes.

@DaanHoogland I have force pushed again. Hopefully a good jenkins slave will pick this up :)

code reviewed. LGTM awaiting tests. it is running on one of the INFRA-10703 hosts but I know @abayer did some work on that issue...

also running the regressions suites on it now

1021.network.results.txt
1021.vpc.results.txt
all tests passed

@bhaisaab @jlk want to review give a second lgtm?

LGTM based on these tests (may not have tested the actual change):
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
component/test_vpc_redundant.py \
component/test_routers_iptables_default_policy.py \
component/test_routers_network_ops.py \
component/test_vpc_router_nics.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py \
smoke/test_network.py

Result:
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_isolate_network_FW_PF_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_isolate_network_FW_PF_default_routes_egress_false | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Check that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 33 tests in 17014.931s

OK

And:
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_privategw_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_vpc_vpn.py \
smoke/test_service_offerings.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

Result:
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 42 tests in 8513.577s

OK

Saw the changes, LGTM

authors: DaanHoogland bhaisaab koushik-das remibergsma


text:
CLOUDSTACK-9887 ignore bogus default gateway

when a shared network is secondary the default gateway gets overwritten by a bogus one
dnsmasq does the right thing and replaces it with its own default which is not good for us
so check for '0.0.0.0'

@DaanHoogland can you fix the commit message to include the JIRA id
@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-662

@rhtyd the PR has the ticket id so the merge commit will as well. Is that not enough?

LGTM.
@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1021)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 26810 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2055-t1021-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Test completed. 47 look ok, 2 have error(s)

Test
Result
Time (s)
Test File

test_04_rvpc_privategw_static_routes
Failure
310.28
test_privategw_acl.py

test_02_list_snapshots_with_removed_data_store
Error
0.03
test_snapshots.py

test_01_vpc_site2site_vpn
Success
149.21
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
60.92
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
230.23
test_vpc_vpn.py

test_02_VPC_default_routes
Success
253.46
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
524.03
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
483.31
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1386.20
test_vpc_redundant.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Success
526.57
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Success
741.73
test_vpc_redundant.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1289.23
test_vpc_redundant.py

test_09_delete_detached_volume
Success
156.22
test_volumes.py

test_08_resize_volume
Success
156.11
test_volumes.py

test_07_resize_fail
Success
161.24
test_volumes.py

test_06_download_detached_volume
Success
156.16
test_volumes.py

test_05_detach_volume
Success
150.67
test_volumes.py

test_04_delete_attached_volume
Success
145.91
test_volumes.py

test_03_download_attached_volume
Success
150.98
test_volumes.py

test_02_attach_volume
Success
83.99
test_volumes.py

test_01_create_volume
Success
615.95
test_volumes.py

test_deploy_vm_multiple
Success
267.04
test_vm_life_cycle.py

test_deploy_vm
Success
0.02
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.02
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
26.50
test_vm_life_cycle.py

test_09_expunge_vm
Success
125.21
test_vm_life_cycle.py

test_08_migrate_vm
Success
35.65
test_vm_life_cycle.py

test_07_restore_vm
Success
0.09
test_vm_life_cycle.py

test_06_destroy_vm
Success
130.69
test_vm_life_cycle.py

test_03_reboot_vm
Success
125.69
test_vm_life_cycle.py

test_02_start_vm
Success
5.10
test_vm_life_cycle.py

test_01_stop_vm
Success
35.22
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
70.56
test_templates.py

test_08_list_system_templates
Success
0.02
test_templates.py

test_07_list_public_templates
Success
0.03
test_templates.py

test_05_template_permissions
Success
0.04
test_templates.py

test_04_extract_template
Success
5.13
test_templates.py

test_03_delete_template
Success
5.08
test_templates.py

test_02_edit_template
Success
90.11
test_templates.py

test_01_create_template
Success
25.25
test_templates.py

test_10_destroy_cpvm
Success
161.27
test_ssvm.py

test_09_destroy_ssvm
Success
164.85
test_ssvm.py

test_08_reboot_cpvm
Success
101.20
test_ssvm.py

test_07_reboot_ssvm
Success
163.50
test_ssvm.py

test_06_stop_cpvm
Success
131.40
test_ssvm.py

test_05_stop_ssvm
Success
163.47
test_ssvm.py

test_04_cpvm_internals
Success
0.94
test_ssvm.py

test_03_ssvm_internals
Success
3.25
test_ssvm.py

test_02_list_cpvm_vm
Success
0.09
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.10
test_ssvm.py

test_01_snapshot_root_disk
Success
10.93
test_snapshots.py

test_04_change_offering_small
Success
209.36
test_service_offerings.py

test_03_delete_service_offering
Success
0.03
test_service_offerings.py

test_02_edit_service_offering
Success
0.04
test_service_offerings.py

test_01_create_service_offering
Success
0.07
test_service_offerings.py

test_02_sys_template_ready
Success
0.10
test_secondary_storage.py

test_01_sys_vm_start
Success
0.13
test_secondary_storage.py

test_09_reboot_router
Success
35.27
test_routers.py

test_08_start_router
Success
25.21
test_routers.py

test_07_stop_router
Success
10.12
test_routers.py

test_06_router_advanced
Success
0.04
test_routers.py

test_05_router_basic
Success
0.03
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.56
test_routers.py

test_03_restart_network_cleanup
Success
45.33
test_routers.py

test_02_router_internal_adv
Success
1.02
test_routers.py

test_01_router_internal_basic
Success
0.56
test_routers.py

test_router_dns_guestipquery
Success
73.94
test_router_dns.py

test_router_dns_externalipquery
Success
0.07
test_router_dns.py

test_router_dhcphosts
Success
237.14
test_router_dhcphosts.py

test_router_dhcp_opts
Success
21.78
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
0.09
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
161.36
test_reset_vm_on_reboot.py

test_createRegion
Success
0.03
test_regions.py

test_create_pvlan_network
Success
5.21
test_pvlan.py

test_dedicatePublicIpRange
Success
0.34
test_public_ip_range.py

test_03_vpc_privategw_restart_vpc_cleanup
Success
443.45
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Success
338.70
test_privategw_acl.py

test_01_vpc_privategw_acl
Success
76.56
test_privategw_acl.py

test_01_primary_storage_nfs
Success
35.75
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.14
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
15.33
test_portable_publicip.py

test_isolate_network_password_server
Success
89.02
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.09
test_over_provisioning.py

test_oobm_zchange_password
Success
30.52
test_outofbandmanagement.py

test_oobm_multiple_mgmt_server_ownership
Success
16.28
test_outofbandmanagement.py

test_oobm_issue_power_status
Success
10.20
test_outofbandmanagement.py

test_oobm_issue_power_soft
Success
15.25
test_outofbandmanagement.py

test_oobm_issue_power_reset
Success
15.28
test_outofbandmanagement.py

test_oobm_issue_power_on
Success
15.26
test_outofbandmanagement.py

test_oobm_issue_power_off
Success
15.26
test_outofbandmanagement.py

test_oobm_issue_power_cycle
Success
15.25
test_outofbandmanagement.py

test_oobm_enabledisable_across_clusterzones
Success
87.27
test_outofbandmanagement.py

test_oobm_enable_feature_valid
Success
5.12
test_outofbandmanagement.py

test_oobm_enable_feature_invalid
Success
0.08
test_outofbandmanagement.py

test_oobm_disable_feature_valid
Success
5.15
test_outofbandmanagement.py

test_oobm_disable_feature_invalid
Success
0.10
test_outofbandmanagement.py

test_oobm_configure_invalid_driver
Success
0.06
test_outofbandmanagement.py

test_oobm_configure_default_driver
Success
0.07
test_outofbandmanagement.py

test_oobm_background_powerstate_sync
Success
23.33
test_outofbandmanagement.py

test_extendPhysicalNetworkVlan
Success
15.25
test_non_contigiousvlan.py

test_01_nic
Success
418.11
test_nic.py

test_releaseIP
Success
141.72
test_network.py

test_reboot_router
Success
407.71
test_network.py

test_public_ip_user_account
Success
10.18
test_network.py

test_public_ip_admin_account
Success
40.22
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
67.10
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.57
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
124.79
test_network.py

test_delete_account
Success
287.09
test_network.py

test_02_port_fwd_on_non_src_nat
Success
55.57
test_network.py

test_01_port_fwd_on_src_nat
Success
108.64
test_network.py

test_nic_secondaryip_add_remove
Success
237.09
test_multipleips_per_nic.py

test_list_zones_metrics
Success
0.17
test_metrics_api.py

test_list_volumes_metrics
Success
5.36
test_metrics_api.py

test_list_vms_metrics
Success
176.25
test_metrics_api.py

test_list_pstorage_metrics
Success
0.22
test_metrics_api.py

test_list_infrastructure_metrics
Success
0.30
test_metrics_api.py

test_list_hosts_metrics
Success
0.24
test_metrics_api.py

test_list_clusters_metrics
Success
0.24
test_metrics_api.py

login_test_saml_user
Success
17.89
test_login.py

test_assign_and_removal_lb
Success
133.39
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
187.26
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
208.09
test_loadbalance.py

test_03_list_snapshots
Success
0.05
test_list_ids_parameter.py

test_02_list_templates
Success
0.03
test_list_ids_parameter.py

test_01_list_volumes
Success
0.02
test_list_ids_parameter.py

test_07_list_default_iso
Success
0.05
test_iso.py

test_05_iso_permissions
Success
0.05
test_iso.py

test_04_extract_Iso
Success
5.19
test_iso.py

test_03_delete_iso
Success
95.13
test_iso.py

test_02_edit_iso
Success
0.06
test_iso.py

test_01_create_iso
Success
20.73
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
183.03
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
142.86
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
493.07
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
404.75
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.21
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.20
test_global_settings.py

test_rolepermission_lifecycle_update
Success
5.79
test_dynamicroles.py

test_rolepermission_lifecycle_list
Success
5.67
test_dynamicroles.py

test_rolepermission_lifecycle_delete
Success
5.58
test_dynamicroles.py

test_rolepermission_lifecycle_create
Success
5.60
test_dynamicroles.py

test_rolepermission_lifecycle_concurrent_updates
Success
5.70
test_dynamicroles.py

test_role_lifecycle_update_role_inuse
Success
5.62
test_dynamicroles.py

test_role_lifecycle_update
Success
10.68
test_dynamicroles.py

test_role_lifecycle_list
Success
5.62
test_dynamicroles.py

test_role_lifecycle_delete
Success
10.67
test_dynamicroles.py

test_role_lifecycle_create
Success
5.62
test_dynamicroles.py

test_role_inuse_deletion
Success
5.60
test_dynamicroles.py

test_role_account_acls_multiple_mgmt_servers
Success
7.03
test_dynamicroles.py

test_role_account_acls
Success
7.18
test_dynamicroles.py

test_default_role_deletion
Success
5.68
test_dynamicroles.py

test_04_create_fat_type_disk_offering
Success
0.06
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.03
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.05
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.06
test_disk_offerings.py

test_01_create_disk_offering
Success
0.08
test_disk_offerings.py

test_deployvm_userdispersing
Success
45.54
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
20.43
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
45.43
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
10.35
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
45.54
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
5.80
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
5.67
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
201.84
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
201.84
test_deploy_vm_iso.py

test_DeployVmAntiAffinityGroup
Success
55.64
test_affinity_groups.py

test_03_delete_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_02_revert_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_01_create_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_static_role_account_acls
Skipped
0.01
test_staticroles.py

test_01_scale_vm
Skipped
0.00
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.03
test_primary_storage.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
0.00
test_deploy_vgpu_enabled_vm.py

@karuturi can this be merged?

LGTM, based on logic and code but could not test

LGTM

I think it is fixed with CLOUDSTACK-9598

@ustcweizhou #1766 doesn't fix the issue completely as it still remains in 4.9

@DaanHoogland are you testing on 4.9.2 or 4.9 branch ?
This issue is related to commit c81b3380dffa06a7134845ea97438619018b520f
I think we'd better fix it in java side.

@ustcweizhou I am looking into this for a customer running 4.9.1
We can not do everything on the java side as the line must not be present when the gateway is not explicitely something other then 0.0.0.0. When we change that to Null we still need to not add the line. Any line of the form
dhcp-option=tag:interface-%s-%s,3,...
fot the tagged interface must be absent.

@DaanHoogland
To be honest, I have no idea why the gateway ip is '0.0.0.0' when the first ip in the network is applied to a non-default nic of a non-windows vm.
However, I do not think this PR fixes the issue. If the gateway ip is not set in dhcp-option, other vms in this network will not get the gateway, even the ip is on the default nic.

@DaanHoogland
FYI, we removed the lines to set gatewayip to 0.0.0.0 in java code in our fork. It does not bring any issue on our production in the past about one year.

@ustcweizhou It is worth a try. Can you confirm that no config line 3 is present for the tagged interface?
i.e. a line containing the pattern
"dhcp-option=tag:interface-%s-%s,3,"

@ustcweizhou do you have a link to that patch, you refer to?

@DaanHoogland The gateway is set after the change, so the dhcp-option you mentioned is configured with correct gateway.
here you are
diff --git a/server/src/com/cloud/network/router/CommandSetupHelper.java b/server/src/com/cloud/network/router/CommandSetupHelper.java
index 54e70b4..402fe3c 100644
--- a/server/src/com/cloud/network/router/CommandSetupHelper.java
+++ b/server/src/com/cloud/network/router/CommandSetupHelper.java
@@ -108,8 +108,6 @@ import com.cloud.offering.NetworkOffering;
 import com.cloud.offerings.NetworkOfferingVO;
 import com.cloud.offerings.dao.NetworkOfferingDao;
 import com.cloud.service.dao.ServiceOfferingDao;
-import com.cloud.storage.GuestOSVO;
-import com.cloud.storage.dao.GuestOSDao;
 import com.cloud.user.Account;
 import com.cloud.uservm.UserVm;
 import com.cloud.utils.Pair;
@@ -174,8 +172,6 @@ public class CommandSetupHelper {
     private VlanDao _vlanDao;
     @Inject
     private IPAddressDao _ipAddressDao;
-    @Inject
-    private GuestOSDao _guestOSDao;

     @Inject
     private RouterControlHelper _routerControlHelper;
@@ -225,12 +221,6 @@ public class CommandSetupHelper {
                 _networkModel.getExecuteInSeqNtwkElmtCmd());

         String gatewayIp = nic.getIPv4Gateway();
-        if (!nic.isDefaultNic()) {
-            final GuestOSVO guestOS = _guestOSDao.findById(vm.getGuestOSId());
-            if (guestOS == null || !guestOS.getDisplayName().toLowerCase().contains("windows")) {
-                gatewayIp = "0.0.0.0";
-            }
-        }

         final DataCenterVO dcVo = _dcDao.findById(router.getDataCenterId());

@ustcweizhou thanks for the patch, i'll look into the replace code. if the gateway line is removed completely this is good. I'll test your patch as well.

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-716

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

LGTM

Trillian test result (tid-1085)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31242 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2055-t1085-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Test completed. 47 look ok, 2 have error(s)

Test
Result
Time (s)
Test File

test_04_rvpc_privategw_static_routes
Failure
346.06
test_privategw_acl.py

test_02_list_snapshots_with_removed_data_store
Error
0.06
test_snapshots.py

test_01_vpc_site2site_vpn
Success
176.06
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
81.32
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
281.07
test_vpc_vpn.py

test_02_VPC_default_routes
Success
315.12
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
614.43
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
529.09
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1456.84
test_vpc_redundant.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Success
577.42
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Success
776.98
test_vpc_redundant.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1322.02
test_vpc_redundant.py

test_09_delete_detached_volume
Success
156.68
test_volumes.py

test_08_resize_volume
Success
156.50
test_volumes.py

test_07_resize_fail
Success
161.57
test_volumes.py

test_06_download_detached_volume
Success
156.37
test_volumes.py

test_05_detach_volume
Success
241.40
test_volumes.py

test_04_delete_attached_volume
Success
151.47
test_volumes.py

test_03_download_attached_volume
Success
156.39
test_volumes.py

test_02_attach_volume
Success
126.38
test_volumes.py

test_01_create_volume
Success
712.39
test_volumes.py

test_deploy_vm_multiple
Success
333.36
test_vm_life_cycle.py

test_deploy_vm
Success
0.03
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.03
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
26.85
test_vm_life_cycle.py

test_09_expunge_vm
Success
125.22
test_vm_life_cycle.py

test_08_migrate_vm
Success
41.03
test_vm_life_cycle.py

test_07_restore_vm
Success
0.14
test_vm_life_cycle.py

test_06_destroy_vm
Success
125.89
test_vm_life_cycle.py

test_03_reboot_vm
Success
125.92
test_vm_life_cycle.py

test_02_start_vm
Success
10.19
test_vm_life_cycle.py

test_01_stop_vm
Success
35.31
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
141.18
test_templates.py

test_08_list_system_templates
Success
0.03
test_templates.py

test_07_list_public_templates
Success
0.04
test_templates.py

test_05_template_permissions
Success
0.07
test_templates.py

test_04_extract_template
Success
5.17
test_templates.py

test_03_delete_template
Success
5.15
test_templates.py

test_02_edit_template
Success
90.15
test_templates.py

test_01_create_template
Success
65.60
test_templates.py

test_10_destroy_cpvm
Success
191.78
test_ssvm.py

test_09_destroy_ssvm
Success
168.82
test_ssvm.py

test_08_reboot_cpvm
Success
101.58
test_ssvm.py

test_07_reboot_ssvm
Success
133.88
test_ssvm.py

test_06_stop_cpvm
Success
161.86
test_ssvm.py

test_05_stop_ssvm
Success
139.04
test_ssvm.py

test_04_cpvm_internals
Success
1.30
test_ssvm.py

test_03_ssvm_internals
Success
3.77
test_ssvm.py

test_02_list_cpvm_vm
Success
0.15
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.15
test_ssvm.py

test_01_snapshot_root_disk
Success
11.23
test_snapshots.py

test_04_change_offering_small
Success
209.88
test_service_offerings.py

test_03_delete_service_offering
Success
0.04
test_service_offerings.py

test_02_edit_service_offering
Success
0.11
test_service_offerings.py

test_01_create_service_offering
Success
0.11
test_service_offerings.py

test_02_sys_template_ready
Success
0.14
test_secondary_storage.py

test_01_sys_vm_start
Success
0.20
test_secondary_storage.py

test_09_reboot_router
Success
45.40
test_routers.py

test_08_start_router
Success
35.36
test_routers.py

test_07_stop_router
Success
10.17
test_routers.py

test_06_router_advanced
Success
0.06
test_routers.py

test_05_router_basic
Success
0.04
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.74
test_routers.py

test_03_restart_network_cleanup
Success
65.57
test_routers.py

test_02_router_internal_adv
Success
1.09
test_routers.py

test_01_router_internal_basic
Success
0.62
test_routers.py

test_router_dns_guestipquery
Success
76.96
test_router_dns.py

test_router_dns_externalipquery
Success
0.07
test_router_dns.py

test_router_dhcphosts
Success
272.47
test_router_dhcphosts.py

test_router_dhcp_opts
Success
21.82
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
0.12
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
131.02
test_reset_vm_on_reboot.py

test_createRegion
Success
0.04
test_regions.py

test_create_pvlan_network
Success
5.24
test_pvlan.py

test_dedicatePublicIpRange
Success
0.73
test_public_ip_range.py

test_03_vpc_privategw_restart_vpc_cleanup
Success
555.95
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Success
406.23
test_privategw_acl.py

test_01_vpc_privategw_acl
Success
97.35
test_privategw_acl.py

test_01_primary_storage_nfs
Success
35.89
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.24
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
15.49
test_portable_publicip.py

test_isolate_network_password_server
Success
89.71
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.15
test_over_provisioning.py

test_oobm_zchange_password
Success
30.75
test_outofbandmanagement.py

test_oobm_multiple_mgmt_server_ownership
Success
16.39
test_outofbandmanagement.py

test_oobm_issue_power_status
Success
10.29
test_outofbandmanagement.py

test_oobm_issue_power_soft
Success
15.39
test_outofbandmanagement.py

test_oobm_issue_power_reset
Success
15.42
test_outofbandmanagement.py

test_oobm_issue_power_on
Success
10.34
test_outofbandmanagement.py

test_oobm_issue_power_off
Success
15.37
test_outofbandmanagement.py

test_oobm_issue_power_cycle
Success
15.40
test_outofbandmanagement.py

test_oobm_enabledisable_across_clusterzones
Success
87.86
test_outofbandmanagement.py

test_oobm_enable_feature_valid
Success
5.21
test_outofbandmanagement.py

test_oobm_enable_feature_invalid
Success
0.17
test_outofbandmanagement.py

test_oobm_disable_feature_valid
Success
5.21
test_outofbandmanagement.py

test_oobm_disable_feature_invalid
Success
0.15
test_outofbandmanagement.py

test_oobm_configure_invalid_driver
Success
0.11
test_outofbandmanagement.py

test_oobm_configure_default_driver
Success
0.13
test_outofbandmanagement.py

test_oobm_background_powerstate_sync
Success
23.53
test_outofbandmanagement.py

test_extendPhysicalNetworkVlan
Success
15.40
test_non_contigiousvlan.py

test_01_nic
Success
485.01
test_nic.py

test_releaseIP
Success
293.33
test_network.py

test_reboot_router
Success
434.06
test_network.py

test_public_ip_user_account
Success
10.28
test_network.py

test_public_ip_admin_account
Success
40.30
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
66.97
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.91
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
124.56
test_network.py

test_delete_account
Success
282.98
test_network.py

test_02_port_fwd_on_non_src_nat
Success
55.82
test_network.py

test_01_port_fwd_on_src_nat
Success
111.79
test_network.py

test_nic_secondaryip_add_remove
Success
217.84
test_multipleips_per_nic.py

test_list_zones_metrics
Success
0.28
test_metrics_api.py

test_list_volumes_metrics
Success
5.51
test_metrics_api.py

test_list_vms_metrics
Success
191.89
test_metrics_api.py

test_list_pstorage_metrics
Success
0.29
test_metrics_api.py

test_list_infrastructure_metrics
Success
0.49
test_metrics_api.py

test_list_hosts_metrics
Success
0.44
test_metrics_api.py

test_list_clusters_metrics
Success
0.35
test_metrics_api.py

login_test_saml_user
Success
19.45
test_login.py

test_assign_and_removal_lb
Success
148.22
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
187.61
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
218.63
test_loadbalance.py

test_03_list_snapshots
Success
0.08
test_list_ids_parameter.py

test_02_list_templates
Success
0.04
test_list_ids_parameter.py

test_01_list_volumes
Success
0.03
test_list_ids_parameter.py

test_07_list_default_iso
Success
0.06
test_iso.py

test_05_iso_permissions
Success
0.07
test_iso.py

test_04_extract_Iso
Success
5.21
test_iso.py

test_03_delete_iso
Success
95.15
test_iso.py

test_02_edit_iso
Success
0.06
test_iso.py

test_01_create_iso
Success
21.06
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
253.66
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
180.40
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
559.12
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
443.30
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.30
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.18
test_global_settings.py

test_rolepermission_lifecycle_update
Success
6.22
test_dynamicroles.py

test_rolepermission_lifecycle_list
Success
6.04
test_dynamicroles.py

test_rolepermission_lifecycle_delete
Success
5.88
test_dynamicroles.py

test_rolepermission_lifecycle_create
Success
5.94
test_dynamicroles.py

test_rolepermission_lifecycle_concurrent_updates
Success
6.02
test_dynamicroles.py

test_role_lifecycle_update_role_inuse
Success
5.92
test_dynamicroles.py

test_role_lifecycle_update
Success
10.99
test_dynamicroles.py

test_role_lifecycle_list
Success
5.95
test_dynamicroles.py

test_role_lifecycle_delete
Success
10.96
test_dynamicroles.py

test_role_lifecycle_create
Success
5.92
test_dynamicroles.py

test_role_inuse_deletion
Success
5.89
test_dynamicroles.py

test_role_account_acls_multiple_mgmt_servers
Success
8.38
test_dynamicroles.py

test_role_account_acls
Success
8.72
test_dynamicroles.py

test_default_role_deletion
Success
6.14
test_dynamicroles.py

test_04_create_fat_type_disk_offering
Success
0.09
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.05
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.05
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.07
test_disk_offerings.py

test_01_create_disk_offering
Success
0.11
test_disk_offerings.py

test_deployvm_userdispersing
Success
20.61
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
45.79
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
55.66
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
10.47
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
60.83
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
6.00
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
6.07
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
282.99
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
212.90
test_deploy_vm_iso.py

test_DeployVmAntiAffinityGroup
Success
66.00
test_affinity_groups.py

test_03_delete_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_02_revert_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_01_create_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_static_role_account_acls
Skipped
0.02
test_staticroles.py

test_01_scale_vm
Skipped
0.00
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.04
test_primary_storage.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
0.01
test_deploy_vgpu_enabled_vm.py

LGTM, based on test results and review.

@ustcweizhou @DaanHoogland you're both LGTM on this as well I suppose?

source: gharchive/pull-request

authors: DaanHoogland blueorangutan resmo rhtyd ustcweizhou

repo: apache/cloudstack

license_type: permissive

license_source: github-api


text:
Remove UserVmDetailsDao duplicate field

Description

Removing UserVmDetailsDao duplicate field;
Found the following repeated field in the UserVmManagerImpl class
@Inject
private UserVmDetailsDao _vmDetailsDao;
@Inject
private UserVmDetailsDao _uservmDetailsDao;
Refactored to a single field;
@Inject
private UserVmDetailsDao userVmDetailsDao;
Similar to this PR: https://github.com/apache/cloudstack/pull/2750/files

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[x] Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs

Screenshots (if appropriate):
How Has This Been Tested?

Checklist:

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
Testing
[ ] I have added tests to cover my changes.
[ ] All relevant new and existing integration tests have passed.
[ ] A full integration testsuite with all test that can run on my environment has passed.

@blueorangutan package

@dhlaluku a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2251

@blueorangutan test

@dhlaluku a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-2929)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42884 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2807-t2929-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermitten failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_usage.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 6 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_03_vpc_privategw_restart_vpc_cleanup
Failure
177.84
test_privategw_acl.py

test_04_extract_template
Failure
128.42
test_templates.py

ContextSuite context=TestISOUsage>:setup
Error
0.00
test_usage.py

test_06_download_detached_volume
Failure
139.67
test_volumes.py

test_04_rvpc_network_garbage_collector_nics
Failure
293.90
test_vpc_redundant.py

test_hostha_enable_ha_when_host_in_maintenance
Error
1.66
test_hostha_kvm.py

@DaanHoogland the failures are known and need to be addressed in master. I will have a look and see if I can provide fixes for some of them starting with the test_usage.py.

LGTM, failures not related to the changes in the PR. Merging.

authors: blueorangutan dhlaluku rhtyd


text:
Fix PowerReportMissing for new VRs

Description
Fallback to update (and creating) time when power state update time is null.
Follow up of CLOUDSTACK-8848
(also see https://github.com/apache/cloudstack/commit/542880ae76c5e6eefbf61d42364e881495e6f3af#diff-dd162c545fd70c5cf2c4b2fa599c13c9L114)
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs

Fixes #2845
Screenshots (if appropriate):
N/A
How Has This Been Tested?

Needs testing
Checklist:

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
Testing
[ ] I have added tests to cover my changes.
[ ] All relevant new and existing integration tests have passed.
[ ] A full integration testsuite with all test that can run on my environment has passed.

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2298

@blueorangutan test centos7 vmware-65

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-65) has been kicked to run smoke tests

Trillian test result (tid-3012)
Environment: vmware-65 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31172 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2846-t3012-vmware-65.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vgpu_enabled_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_outofbandmanagement.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 66 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

ContextSuite context=TestDeployVirtioSCSIVM>:teardown
Error
0.00
test_deploy_virtio_scsi_vm.py

authors: blueorangutan resmo rhtyd


text:
[WIP DO NOT MERGE] ui: add memory used column in instance metrics view

Problem: The metrics view for instances does not show used memory.
Root Cause: Used memory was not added as a column in the metrics view for instances.
Solution: Metrics view for instances makes ‘listVirtualMachineMetrics’ API call to gather metrics. The response of this API call contains total and free memory with the instance. These have been used to compute used memory as a percentage of total memory and displayed as a new column in UI. Also fixes
a bug for VMware, due to which incorrect memory usage was returned.
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2658

@blueorangutan test

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-3455)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32818 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3243-t3455-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 69 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_05_rvpc_multi_tiers
Failure
429.66
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
460.46
test_vpc_redundant.py

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2776

Here's a screenshot

LGTM

@rhtyd I think we're OK for merging this

LGTM

Thanks all.

authors: DagSonsteboSB andrijapanicsb blueorangutan borisstoyanov rhtyd


text:
UI: fix bulk delete project with cleanup

Description
This PR fixes an UI issue that cleanup parameter is not passed when bulk-delete projects.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[x] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

@shwstppr a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

authors: blueorangutan weizhouapache


text:
Upgrade org.json:json version

Description
This PR fixes #8696

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

How did you try to break this feature and the system with this change?

@blueorangutan package

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el7 ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 8867

@blueorangutan test matrix

@vishesh92 a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[SF] Trillian test result (tid-9416)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38528 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8750-t9416-xenserver-71.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

[SF] Trillian test result (tid-9418)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39467 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8750-t9418-kvm-centos7.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

[SF] Trillian test result (tid-9417)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 45660 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8750-t9417-vmware-67u3.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

@blueorangutan test alma9 kvm-alma9

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

[SF] Trillian Build Failed (tid-9452)

[SF] Trillian Build Failed (tid-9461)

@blueorangutan test ol9 kvm-ol9

@rohityadavcloud [SL] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, alma9. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

[SF] Trillian test result (tid-9454)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 40392 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8750-t9454-kvm-rocky8.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

@blueorangutan test alma9 kvm-alma9

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

[SF] Trillian test result (tid-9506)
Environment: kvm-alma9 (x2), Advanced Networking with Mgmt server a9
Total time taken: 43307 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8750-t9506-kvm-alma9.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

@sureshanaparti @vishesh92 , do you think we need more testing for this (cc @borisstoyanov )

authors: DaanHoogland blueorangutan rohityadavcloud vishesh92


text:
Persist IP addresses related to VM access via CPVM

Description
This PR proposes to persist IP addresses related to VM access via CPVM. Specifically, it stores the IP address of the client accessing a VM and the IP address of the console endpoint creator. To achieve this, the cloud.console_session table was extended with two new columns: client_address, which stores the client's IP address, and console_endpoint_creator_address, which captures the IP address of the console endpoint creator.
These IP addresses were already being captured for logging and validation purposes. The console endpoint creator's IP is captured here, at the start of the CreateConsoleEndpointCmd execution:
https://github.com/apache/cloudstack/blob/47a6b7011d7e42be86a748af50b87633147b5a90/api/src/main/java/org/apache/cloudstack/api/command/user/consoleproxy/CreateConsoleEndpointCmd.java#L65-L68
The client address is captured by calling session.getRemoteAddress().getAddress().getHostAddress() in this method, that gets executed when the client connects to the console:
https://github.com/apache/cloudstack/blob/47a6b7011d7e42be86a748af50b87633147b5a90/services/console-proxy/server/src/main/java/com/cloud/consoleproxy/ConsoleProxyNoVNCHandler.java#L157-L160
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[X] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI
[ ] test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[X] Minor

Screenshots (if appropriate):
How Has This Been Tested?

I manually added the client_address and console_endpoint_creator_address columns to the cloud.console_session table.
I generated a VM console endpoint.
I executed the following query in order to validate whether the stored IP was correct.

SELECT * FROM `cloud`.`console_session` ORDER BY created DESC LIMIT 1;

id
uuid
created
account_id
user_id
instance_id
host_id
acquired
removed
client_address
console_endpoint_creator_address

2
77f08cea-8068-471f-9464-44810e1ef545
2024-08-06 17:57:05
2
2
14
1
NULL
NULL
NULL
192.168.122.1

As can be noticed, the console_endpoint_creator_address column was populated accordingly, whereas the client_adress was still empty since the VM had not been accessed yet.

I accessed the VM via CPVM.
I executed the following query in order to validate whether the client stored IP was correct.

SELECT * FROM `cloud`.`console_session` ORDER BY created DESC LIMIT 1;

id
uuid
created
account_id
user_id
instance_id
host_id
acquired
removed
client_address
console_endpoint_creator_address

2
77f08cea-8068-471f-9464-44810e1ef545
2024-08-06 17:57:05
2
2
14
1
NULL
NULL
192.168.122.1
192.168.122.1

As can be noticed, the IP address of the client that accessed the VM through CPVM was persisted correctly.

@blueorangutan package

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 10674

@bernardodemarco , I understand what and how, but have a hard time to get the why of this PR. Can you expand on that please?

@bernardodemarco , I understand what and how, but have a hard time to get the why of this PR. Can you expand on that please?

The main idea behind the proposed enhancement is to help operators better identify who accessed a virtual machine console.
The cloud.console_session table already stores data relating to who (account_id and user_id) accessed a given virtual machine (instance_id), as well as when that happened (created and removed). By adding the client_address and console_endpoint_creator_address fields, it'll be easier to identify who accessed the console, which can be extremely helpful for security and auditing purposes.

thanks, makes sense.

@nvazquez can you have a look at this one?

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 11055

@blueorangutan package

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

client_address is always NULL in my testing.
can you check ? @bernardodemarco  cc @DaanHoogland

client_address is always NULL in my testing.

I tested the PR again, following the steps provided on the description, and the IP addresses were stored correctly.
It's important to remember that if there were any running CPVMs prior to the environment update, then they would need to be recreated in order to test if the client_address column is being populated correctly.
Furthermore, it's interesting taking a look at the CPVM logs. The following logs were generated during my tests, which indicates that the client IP 192.168.122.1 was captured correctly.
2024-09-10T17:19:38,330 INFO  [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp2090537387-40:[]) Verifying session
source IP 192.168.122.1 from WebSocket connection request.

2024-09-10T17:19:38,331 DEBUG [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp2090537387-40:[]) Session source IP
192.168.122.1 has been verified successfully.

@weizhouapache, if the problem continues happening, could you please describe your tests steps and share the CPVM logs?

client_address is always NULL in my testing.

I tested the PR again, following the steps provided on the description, and the IP addresses were stored correctly.
It's important to remember that if there were any running CPVMs prior to the environment update, then they would need to be recreated in order to test if the client_address column is being populated correctly.
Furthermore, it's interesting taking a look at the CPVM logs. The following logs were generated during my tests, which indicates that the client IP 192.168.122.1 was captured correctly.
2024-09-10T17:19:38,330 INFO  [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp2090537387-40:[]) Verifying session
source IP 192.168.122.1 from WebSocket connection request.

2024-09-10T17:19:38,331 DEBUG [cloud.consoleproxy.ConsoleProxyNoVNCHandler] (qtp2090537387-40:[]) Session source IP
192.168.122.1 has been verified successfully.

@weizhouapache, if the problem continues happening, could you please describe your tests steps and share the CPVM logs?

Thanks @bernardodemarco
It works after patching the cpvm

[SF] Trillian test result (tid-11441)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 50973 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9534-t11441-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_01_migrate_VM_and_root_volume
Error
97.32
test_vm_life_cycle.py

test_02_migrate_VM_with_two_data_disks
Error
50.60
test_vm_life_cycle.py

test_08_migrate_vm
Error
0.08
test_vm_life_cycle.py

test_01_migrate_vm_strict_tags_success
Error
0.23
test_vm_strict_host_tags.py

test_02_migrate_vm_strict_tags_failure
Error
0.23
test_vm_strict_host_tags.py

@JoaoJandre , it looks like this is ready. should we merge it before 4.20?

@JoaoJandre , it looks like this is ready. should we merge it before 4.20?

I think we should leave this for 4.21. We're only accepting bugfixes currently.

@blueorangutan package

@BryanMLima a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 11658

@blueorangutan package

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result [SF]: ✔ el8 ✔ el9 ✔ debian ✔ suse15. SL-JID 11715

@weizhouapache should we run the CI once again for this one? The tests failures here https://github.com/apache/cloudstack/pull/9534#issuecomment-2342555093 don't seem to be related

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[SF] Trillian test result (tid-11875)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51503 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9534-t11875-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test
Result
Time (s)
Test File

test_01_secure_vm_migration
Error
403.97
test_vm_life_cycle.py

authors: BryanMLima DaanHoogland JoaoJandre bernardodemarco blueorangutan weizhouapache


text:
Update IOUtils.java

Added a new public static function closeQuietly that takes an indefinite number of Closeable objects as input parameters.

With the recent move to java7 features and deprecation of all the closeQuitely, this pr will not be merged. Please close it.

Understood. Putting closeable objects in the try ( .. ) { .. } lines will handle closing for us.

repo: apache/commons-io

license_source: bigquery


text:
Add characterization test for copying a symlinked directory

See what happens when copyDirectory copies a directory that is a symlink to another directory containing non-symlinked files. This is a characterization test to explore current behavior, and arguably represents a bug. This behavior, and the test, is likely to change and should not be relied on.
@garydgregory @eamonnmcmanus

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (546a4d2) 85.99% compared to head (ef08c77) 85.99%.

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #570   +/-   ##
=========================================
  Coverage     85.99%   85.99%
  Complexity     3497     3497
=========================================
  Files           231      231
  Lines          8253     8253
  Branches        964      964
=========================================
  Hits           7097     7097
  Misses          866      866
  Partials        290      290

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Interesting: Adding these two tests add 4 seconds to running FileUtilsTest with mvn test -Dtest=FileUtilsTest.

authors: codecov-commenter elharo garydgregory


text:
For-Remove statement allows to remove current element from underlying…

… iterator collection in for-loops

Just noticed the pull-request, sorry. It seems it conflates everything (this, remove, regex, in...).
Would be easier to review if these were separate requests.

authors: dmitri-blinov henrib

repo: apache/commons-jexl


text:
Discrete maths

beginnings for doing a bit of discrete mathematics

This patch looks interesting.
Would you mind discussing about it on the Commons Developers list? This is where all technical discussions about Apache Commons Math occur. You can find instructions here: http://commons.apache.org/proper/commons-math/mail-lists.html
Do not forget to put the [math] marker on the subject line as the list is shared among all Commons components, this helps people filtering messages.
best regards,
Luc

authors: luca-vercelli maisonobe

repo: apache/commons-math


text:
ci: update codecov@v4 w/ token

Platforms affected
n/a
Motivation and Context

Update Codecov
Description

Bump codecov to version 4
Run on success
Add better run naming
Do not fail if codecov fails

Testing

n/a
Checklist

[ ] I've run the tests to see all new and existing tests pass
[ ] I added automated test coverage as appropriate for this change
[ ] Commit is prefixed with (platform) if this change only applies to one platform (e.g. (android))
[ ] If this Pull Request resolves an issue, I linked to the issue in the text above (and used the correct keyword to close issues using keywords)
[ ] I've updated the documentation if necessary

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.97%. Comparing base (e594cd4) to head (5241040).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #142   +/-   ##
=======================================
  Coverage   89.97%   89.97%
=======================================
  Files           6        6
  Lines        2374     2374
=======================================
  Hits         2136     2136
  Misses        238      238

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter erisu

repo: apache/cordova-node-xcode


text:
1.1.x

I am getting DeviceSerial = unknown
function OnDeviceReady()
{
        $localStorage.DeviceSerival = window.device.serial;
}

Device = iPhone | OS = iOS 11.4.1 | Browser = Mobile Safari UIWebView 11.4.1 | DeviceSerial = unknown
for android it is working but for ios is giving me unknown.

This PR has no content.
Is this really an issue? Then please open it over at issues.cordova.io.

And to make this issue unnecessary:
device.serial is not supported on iOS as the documentation also says: https://github.com/apache/cordova-plugin-device#deviceserial

repo: apache/cordova-plugin-device


text:
Erlang/OTP 23 Support

Summary
Would it be possible to add support for OTP 23?
Desired Behaviour
Support for building/running CouchDB with OTP 23.
Possible Solution
I tried just adding the appropriate version numbers to rebar.config.script as was done in #2070, but the build fails when attempting to link b64url.
==> b64url (compile)
Compiled src/b64url.erl
Compiling c_src/b64url.c
/usr/bin/ld: cannot find -lerl_interface
collect2: error: ld returned 1 exit status
ERROR: sh(cc c_src/b64url.o $LDFLAGS -shared  -L"/usr/lib/erlang/lib/erl_interface-4.0/lib" -lerl_interface -lei -o priv/b64url.so)
failed with return code 1 and the following output:
/usr/bin/ld: cannot find -lerl_interface
collect2: error: ld returned 1 exit status

Some deprecated parts of erl_interface were removed in OTP 23 (https://www.erlang.org/news/140).
There may, or course, be further incompatibilities.
Additional context
Void Linux has moved on to OTP 23.

Another problem is that we currently have a lot of this kind of compiler warning:
/proj/couchdb-3/src/chttpd/src/chttpd.erl:279: Warning: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace

But we can't replace those with the new try/catch syntax while we need to support versions previous to OTP 21. However, that same thread shows some workarounds that we could presumably use to patch all the existing incompatible code.

Having to stick macros in all over the code does sound like a bit of a pain, unless it can be done with a find/replace without breaking everything.
I have basically zero knowledge of Erlang or how it works, but I do find it a bit strange that they'd remove existing syntax, and expect dependant projects to keep up. Unless it's common for Erlang projects to only support specific OTP versions?

Another option is to deprecate erlang prior to 21. However it could be a tougher call because I think we have platforms which do not support 21 yet. Our current position is expressed here https://github.com/apache/couchdb/issues/2934#issuecomment-641473765.

@iilyak My understanding from IBM is that there have been issues in production with Erlang 21+, which is why the official binary packages we have on couchdb.org are built against 20.3.8.latest. If you or @rnewson or @davisp have updated info as to what's safe to move to, we could consider moving ahead, but the earliest would be 3.2.0. The impending 3.1.1 patch release would be too minor to change Erlang versions.

The only known bad versions I know of were explicitly excluded in the
rebar.config.script.
On Mon, Aug 31, 2020 at 10:25 AM Joan Touzet notifications@github.com
wrote:

@iilyak https://github.com/iilyak My understanding from IBM is that
there have been issues in production with Erlang 21+, which is why the
official binary packages we have on couchdb.org are built against
20.3.8.latest. If you or @rnewson https://github.com/rnewson or @davisp
https://github.com/davisp have updated info as to what's safe to move
to, we could consider moving ahead, but the earliest would be 3.2.0.
The impending 3.1.1 patch release would be too minor to change Erlang
versions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/couchdb/issues/3115#issuecomment-683849187,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAE3WLVBFJTVHIAXZCQXUDSDO6GZANCNFSM4QPXHR7A
.

FYI:
16:59 <+rnewson> couchdb goes quite wrong on 21/22
16:59 <+rnewson> the mochiweb modules thingy
16:59 <+rnewson> (at least)
16:59 <+rnewson> Wohali: I think it's all fixable but hard to keep 18/19 support
16:59 <+rnewson> I didn't get deep into it, but it bites me whenever I forget to add
                 erlang 20 to my path
17:00 <+rnewson> cloudant is on 20 all over afaik

Keep in mind that couch_log need to be updated as well to match the changes in OTP logging infrastructure. I did a test and logging of crashes from gen_server doesn't work on otp 22 and I am sure wouldn't be working on 23 without changes.

I’ve started digging through these.

[x] fixing -Ilerl_interface is relatively easy, we are not using it anywhere, so we can just hotwire our local rebar copy:

diff --git a/src/rebar_port_compiler.erl b/src/rebar_port_compiler.erl
index 9679c80..44f7b0d 100644
--- a/src/rebar_port_compiler.erl
+++ b/src/rebar_port_compiler.erl
@@ -673,7 +673,7 @@ default_env() ->
                        "\" "
                       ])},
      {"ERL_EI_LIBDIR", lists:concat(["\"", erl_interface_dir(lib), "\""])},
-     {"ERL_LDFLAGS"  , " -L$ERL_EI_LIBDIR -lerl_interface -lei"},
+     {"ERL_LDFLAGS"  , " -L$ERL_EI_LIBDIR -lei"},
      {"ERLANG_ARCH"  , rebar_utils:wordsize()},
      {"ERLANG_TARGET", rebar_utils:get_arch()},

[x] get_stacktrace() is not getting removed until 24, so we have one more release to go, but the diff is totally bearable IMHO: https://github.com/apache/couchdb/pull/new/feat/3.x/otp23

[x] snappy needs to allow 23:

diff --git a/rebar.config b/rebar.config
index 1785295..250cd2a 100644
--- a/rebar.config
+++ b/rebar.config
@@ -1,4 +1,4 @@
-{require_otp_vsn, "R14|R15|R16|17|18|19|20|21|22"}.
+{require_otp_vsn, "R14|R15|R16|17|18|19|20|21|22|23"}.

 {erl_opts, [debug_info, warn_unused_vars, warn_shadow_vars, warn_unused_import]}.
 {port_sources, ["c_src/*.cc",

[ ] jiffy needs…I don’t know ;D @davisp help:

escript enc compile
Uncaught error in rebar_core: {'EXIT',
                               {undef,
                                [{rebar_utils,get_cwd,[],[]},
                                 {rebar_config,new,0,[]},
                                 {rebar,init_config,1,[]},
                                 {rebar,run,1,[]},
                                 {rebar,main,1,[]},
                                 {escript,run,2,
                                  [{file,"escript.erl"},{line,758}]},
                                 {escript,start,1,
                                  [{file,"escript.erl"},{line,277}]},
                                 {init,start_em,1,[]}]}}
=ERROR REPORT==== 13-Mar-2021::18:16:56.267938 ===
Loading of /Users/jan/Work/asf/couchdb/src/jiffy/enc/enc/ebin/rebar_utils.beam failed: badfile

=ERROR REPORT==== 13-Mar-2021::18:16:56.267930 ===
beam/beam_load.c(1624): Error loading module rebar_utils:
  please re-compile this module with an 23 compiler (old-style fun with indices: 1/4)

=ERROR REPORT==== 13-Mar-2021::18:16:56.278168 ===
Loading of /Users/jan/Work/asf/couchdb/src/jiffy/enc/enc/ebin/rebar_utils.beam failed: badfile

=ERROR REPORT==== 13-Mar-2021::18:16:56.278160 ===
beam/beam_load.c(1624): Error loading module rebar_utils:
  please re-compile this module with an 23 compiler (old-style fun with indices: 1/4)

escript: exception error: undefined function rebar_utils:delayed_halt/1
  in function  escript:run/2 (escript.erl, line 758)
  in call from escript:start/1 (escript.erl, line 277)
  in call from init:start_em/1
  in call from init:do_boot/3

—
That’s as far as I got this time around.

I gave this a spin and with the changes outlined here on 3.x, I got make check to pass on Erlang 23.
@iilyak Do you have more pointers on the logging behaviour. I found this reference in the Erlang 22 release notes:

gen_* behaviours: If logging of the last N messages through sys:log/2,3 is active for the server, this log is included in the terminate report.

— https://www.erlang.org/news/132
But I’m not sure if that’s what you mean.

Erlang 20 is now fixed: https://github.com/apache/couchdb/pull/3422/commits/70334b3f9e85dd558c2a64ca222ceebe8f33a9c3#diff-a07a3d3b357a34c4acb80b2e178e9a19e6c2ed9d91cb0aa8d53cadabb1ae8bd0R95
This is mainly a draft to show how it can be done. I’m happy to see a bike shed about making this cleaner.
I’m a bit confused tho, this broke builds tagged with 21 and 22, but I didn’t dig too deep in it. Are we sure our per-version builds are actually using the versions they say they do?

I don't see it, our PRs currently build against 20.3.8.11, 20.3.8.25 and 22.2.3. We don't build against any 21 in Jenkins and we don't have 23 in Jenkins yet.

@wohali agreed that the logs show that now, there was just some intermittent state there when they were failing. The UI there is also not very intuitive, so I may have turned myself around.

It looks like @janl fixed this with #3422, but I don't see those commits on main yet.
@nickva can you cherry-pick #3422 onto main, then we can close this? Thanks!

@wohali good idea. I will do that. Beforehand I think we may want to clean up some of the applications in main which only work in 3.x (rexi, mem3, etc). I have a head-start on that already and will see how it looks in the next few days, then will port Jan's Erlang 23 support to main.

@nickva This is now blocking my CI work, I'm going to go ahead and cherry pick to main (with a PR) unless you feel strongly the cleanup should happen first.

We have OTP 23 support now in both main and 3.x branches. I think we can close this

source: gharchive/issue

authors: davisp iilyak janl jaydoane maciozo nickva wohali

repo: apache/couchdb


text:
WIP: Attempt to compile with scala 2.13.0

This branch contains changes to enable cross compilation with scala 2.13.0, but currently has quiet a few compilation errors if attempted to cross compile with scala 2.13.0.
Two major changes noted with the new version of scala is that:

the returned type of every method must explicitly be declared else it produces a syntax error that’s including void methods. Currently there are 92 compilations errors and almost if not half of them has to do with this new syntax requirements of void methods.
scala has a predefined Node class which has a child method which previously used to return a mutable seq of all the child of that particular node, but in scala 2.13.0 that returned type is now immutable which causes type mismatch in at least one instance so far so to get around “toSeq” function on the child call to turn that seq back to a mutable seq as it was in the previous version.
Other changes noted with scala 2.13.0 include:
a)	Deprecation of the any2Stringadd object predefinition as well as the implicit injection of the + sign. This now causes problem when adding a file name which is a string to a Path without calling the toString method on the path because it’s no longer implied now.
b)	Stricter type checking when adding a character to a string you need to either use the Character.toString() method on the character or use string interpolation to add the two. In some instances, the string interpolation causes some issues when that value is passed to a regex and cause some illegalArgumentException.

Re 2) .toSeq returns an immutable Seq.
Looking at the scala changelog https://github.com/scala/scala/releases/tag/v2.13.0 I believe what actually changed is the build it "Seq" type is now an alias for collection.immutable.Seq. Previously, it had meant collections.Seq, which includes mutable implementations.

Closed as Inactive. DAFFODIL-2152 has a link here for future reference.

authors: bsloane1650 gdesrosi mbeckerle

repo: apache/daffodil


text:
fix: SortMergeJoin with unsupported key type should fall back to Spark

Which issue does this PR close?

Closes #354.
Rationale for this change

What changes are included in this PR?

How are these changes tested?

As an idea, I was thinking that we could put in tests to check the supported types in Datafusion, so that when we upgrade to a version of datafusion that supports these other types (like timestamp with timezone for example) we are aware that we should change the list of allowed types.

Merged. Thanks.

@planga82 Thanks for the suggestion. I created an issue to track it.

repo: apache/datafusion-comet


text:
Add expected answers to DataFrame method examples

I feel it would benefit understanding for Dataframe methods if we also include the expected answer of calling the method in datafusion/core/src/dataframe/mod.rs
Currently we have a usage example and it would be great adding the expected answer
Originally posted by @comphead in https://github.com/apache/datafusion/pull/12520#discussion_r1765376444

Hi, I haven't contribute here before and would like to take this issue.

I have added expected result in my fork.
Could I also change doctest data to make methods' usage more apparent?

repo: apache/datafusion


text:
Add standalone AnalyzerRule example that implements row level access control

Which issue does this PR close?
Closes https://github.com/apache/datafusion/issues/10855
Rationale for this change
It is common to use DataFusion to analyze / modify queries, so it would be good to have some more full featured examples of doing so.
Additionally,  the current rewrite_expr.rs example has three distinct examples:

Using DataFusion LogicalPlanning without execution
Implementing an AnalyzerRule
Implementing an OptimizerRule

Each I think should be its own example to make them easier to find and more full featured
What changes are included in this PR?

Add example analyzer_rule.rs that implements "row level access control". I was inspired by @goldmedal 's description of Wren's usecase in https://github.com/apache/datafusion/issues/10855#issuecomment-2171124147 to pick the row level access control example

Are these changes tested?
By CI

Are there any user-facing changes?

Thank you for the review @jonahgao and @goldmedal