text:
git source element uses inconsistent remote names and cannot update local repo

See original issue on GitLab
In GitLab by [Gitlab user @bwh-ct] on Jul 22, 2020, 19:07
Summary
In some configurations, the git source element uses different remote names when cloning a repository and when trying to update the local repository later. I was able to reproduce this only when there is a mirror configured on the same HTTP host as the canonical URL.
Steps to reproduce

Start an HTTP server serving static files from some directory you can write to.

Define shell variables $http_dir as the directory served over HTTP, and $http_url as its HTTP URL.

In a temporary directory, run the following commands.

cat >project.conf <<EOT
name: build
element-path: .
aliases:
  local: $http_url
mirrors:
- name: my_mirror
  aliases:
    local:
    - $http_url/fake-mirror/
EOT

git init $http_dir/upstream
(cd $http_dir/upstream && echo foo > foo && git add foo && git commit -m foo)
GIT_DIR=$http_dir/upstream/.git git update-server-info

upstream_ref=$(GIT_DIR=$http_dir/upstream/.git git rev-parse HEAD)
cat >upstream.bst <<EOT
kind: import
sources:
- kind: git
  url: local:upstream/.git
  ref: $upstream_ref
config:
  target: /
EOT
bst build upstream.bst

(cd $http_dir/upstream && echo bar > foo && git add foo && git commit -m bar)
GIT_DIR=$http_dir/upstream/.git git update-server-info

new_upstream_ref=$(GIT_DIR=$http_dir/upstream/.git git rev-parse HEAD)
sed -i "s/$upstream_ref/$new_upstream_ref/" upstream.bst
bst build upstream.bst

What is the current bug behavior?
The second build fails, with error messages like:
    Running host command /home/bwh/.cache/buildstream/sources/git/local_upstream__git: /usr/bin/git fetch http___192_168_122_1__bwh_bst_test_ --prune --force --tags
    [--:--:--] STATUS  upstream.bst-0: Running host command

        /usr/bin/git fetch http___192_168_122_1__bwh_bst_test_ --prune --force --tags
    fatal: 'http___192_168_122_1__bwh_bst_test_' does not appear to be a git repository
    fatal: Could not read from remote repository.

The repository in /home/bwh/.cache/buildstream/sources/git/local_upstream__git only has the remotes:
http___192_168_122_1__bwh_bst_test__fake_mirror_
origin

What is the expected correct behavior?
Either the first build should have created the local repo with a remote named after the canonical URL, or the second build should fetch from the origin remote.
Relevant logs and/or screenshots
Possible fixes
Other relevant information

BuildStream version affected: /milestone %BuildStream_v1.4

I'm running BuildStream 1.4.3 and have not tested the master branch.

In GitLab by [Gitlab user @bwh-ct] on Jul 22, 2020, 19:07
changed the description

In GitLab by [Gitlab user @juergbi] on Jul 22, 2020, 21:27
Thanks for clearly listing the steps to reproduce this issue. This seems to be fixed in master by commit c7346b1d from !1808, which drops the use of git remotes and always passes full URLs to git on the command line. It could make sense to backport at least the part that switches from remotes to URLs.

In GitLab by [Gitlab user @bwh-ct] on Jul 29, 2020, 15:36
mentioned in commit CodethinkLabs/lorry/lorry-buildstream[Gitlab user @011e771f0cd65918c0bba74bb0a18774736dab11]

In GitLab by [Gitlab user @bwh-ct] on Jul 29, 2020, 15:37
mentioned in commit CodethinkLabs/lorry/lorry-buildstream[Gitlab user @f6b1f8c7af0c7f1e8912737d282ec8d0638878a2]

In GitLab by [Gitlab user @bwh-ct] on Jul 29, 2020, 15:37
mentioned in merge request CodethinkLabs/lorry/lorry-buildstream!17

In GitLab by [Gitlab user @bwh-ct] on Jul 29, 2020, 15:59
mentioned in merge request !2005

In GitLab by [Gitlab user @bwh-ct] on Jul 29, 2020, 17:04
mentioned in merge request !2006

In GitLab by [Gitlab user @bwh-ct] on Aug 14, 2020, 17:23
mentioned in commit aca3f31d989b00fba8aabc2a0c179c87f787b4e4

In GitLab by [Gitlab user @bwh-ct] on Aug 14, 2020, 17:24
mentioned in commit 396657cb77af947cc82ba792deec7df6b77cf68b

In GitLab by [Gitlab user @bwh-ct] on Aug 14, 2020, 17:27
mentioned in merge request !2030

In GitLab by [Gitlab user @bwh-ct] on Aug 14, 2020, 17:29
mentioned in merge request !2031

In GitLab by [Gitlab user @bwh-ct] on Aug 19, 2020, 15:45
mentioned in commit 968cd30898333e064aaa28aead8da119e6b00b33

In GitLab by [Gitlab user @bwh-ct] on Aug 20, 2020, 05:34
mentioned in commit f5784d0b123ea3dabde07c68e0e37f6be0b69c41

In GitLab by [Gitlab user @tristanvb] on Nov 6, 2020, 10:02
This is fixed by !2030


text:
Release 1.0.0-CR1

Now that Camel 3.0.0 is out we should converge towards a final release. We can skip the M5 release and instead release our RC1 before christmas.

+1

+1

New Features

IntegrationPlatform specification does no longer contain default values in order to ease upgrades (#1127, @nicolaferraro)
All traits are automatically documented on the Camel K website (see https://camel.apache.org/camel-k/latest/) (#1068, @nicolaferraro)
feat(build): Task-based builds
fix(build): Use EmptyDir volumes for Kaniko builds when caching is disabled
fix(build): Disable Kaniko cache by default (#1121, @astefanutti)
Default values for the kamel CLI can now be configured through a configuration file and/or environment variables. (#1133, @lburgazzoli)
Camel K Kubernetes Custom Resource APIs promoted to "v1" (#1160, @nicolaferraro)

Bug Fixes

Knative profile detection has been deferred to when the integration runs, in order to allow Camel K and Knative to be installed in any order (#1151, @nicolaferraro)

Other Notable Changes

Added support for pushing images to authenticated external docker registries (such as Docker Hub or Github Packages) (#1084, @nicolaferraro)
fix(OpenAPI): Use catalog runtime version to generate OpenAPI project (#1096, @astefanutti)
Upgrade Operator SDK to version 0.13.0 (#1153, @astefanutti)
Use JSON merge patch to reconcile Camel K custom resource statuses (#1110, @astefanutti)
Better report build progress (#1064, @astefanutti)
Support for Knative 0.9+ (codebase compiled with Knative 0.10.0 API). Dropped compatibility with Knative 0.8.x. (#1123, @nicolaferraro)
If an integration platform is not provided, a default one will be created whenever possible (OpenShift) (#1117, @nicolaferraro)
feat: Generic patching of integration resources in running phase (#1100, @astefanutti)
Upgrade to Camel K Runtime 1.0.9
Upgrade to Camel Quarkus 1.0.0-M1
Upgrade to Kaniko 0.14.0 (#1167, @nicolaferraro)

source: gharchive/issue

authors: abouchama nicolaferraro oscerd

repo: apache/camel-k

license_type: permissive

license_source: github-api


text:
fix(build): Upgrade container image to golang 1.21

Closes #5231
Release Note
fix(build): Upgrade container image to golang 1.21

We can safely skip test for this change as it only affect the Golang debug feature that is not checked in our suite.

source: gharchive/pull-request

authors: gansheer squakez


text:
Merge pull request #1 from apache/master

Merge from Master

Sorry wrong pull request.

repo: apache/camel


text:
CAMEL-10085 UnsafeUriCharactersEncoder.checkRAW...

CAMEL-10085 UnsafeUriCharactersEncoder.checkRAW compiles regex pattern every call.
This PR enables UnsafeUriCharactersEncoder to use static, precompiled pattern for checking if it matches given string. Thanks to it, performance may be improved.

Thanks looks good.

I'm glad I could help :)
Regards,
Marcin Lachowicz
2016-07-04 20:02 GMT+02:00 Claus Ibsen notifications@github.com:

Thanks looks good.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/camel/pull/1059#issuecomment-230335171, or mute
the thread
https://github.com/notifications/unsubscribe/AS3hkSaB6YzpmnBqHCD-FgUVm7q2BCcaks5qSUrSgaJpZM4JEYLL
.

I'll merge tomorrow morning :-)
Many thanks Marcin :-)
Il 04/lug/2016 21:24, "MarcinLachowicz" notifications@github.com ha
scritto:

I'm glad I could help :)
Regards,
Marcin Lachowicz
2016-07-04 20:02 GMT+02:00 Claus Ibsen notifications@github.com:

Thanks looks good.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/camel/pull/1059#issuecomment-230335171, or
mute
the thread
<
https://github.com/notifications/unsubscribe/AS3hkSaB6YzpmnBqHCD-FgUVm7q2BCcaks5qSUrSgaJpZM4JEYLL
.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/camel/pull/1059#issuecomment-230344725, or mute
the thread
https://github.com/notifications/unsubscribe/AE3r141YiU3KG2o38AhyFDl4iqyZyPuRks5qSV35gaJpZM4JEYLL
.

The PR has been merged. Can you close the PR?
Thanks for contributing :-)

Sure :) Thank you all for cooperation, cheers :)

authors: MarcinLachowicz davsclaus oscerd


text:
CAMEL-20181: camel-openapi-java - refactor tests to support both Open…

…API 3.0 and 3.1
Description

Target

[ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ ] I checked that each commit in the pull request has a meaningful subject line and body.

[ ] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

/component-test camel-openapi-java


text:
rdar://problem/23335186 Cass: Implement compaction for a specific t…

…oken range

What version of c* are you targeting? Looks like you've based it on whatever our current trunk is. For the OSS c*, you are probably best targeting > 2.2, and the back port to our internal versions probably won't be too hard.
I have further questions, as well, but waiting on this first piece of info.

This is based on OSS c* trunk.

I thought this was an internal repo, hence my comments. We don't accept PRs for the Apache Cassandra project, so please upload a patch to the JIRA ticket or add a link to your guthub branch in the ticket

authors: jasobrown vkasar

repo: apache/cassandra


text:
ui: Simplify code

Optioinal chainging
The optional chaining ?. is a safe way to access nested object properties, even if an intermediate property doesn’t exist.
(error.response && error.response.headers && error.response.headers['x-description']) || error.message
using optional chaining it reduces to
error.response?.headers?.['x-description'] || error.message
Ref : https://javascript.info/optional-chaining
This can also be used to invoke methods on objects, etc
This improves readability as well as simplifies the existing code
Nullish coalescing
Nullish coalescing is an improvement on the || operator where it only checks if an object is null or undefined.
Using the || operator returns false if the object is zero or an empty string which can cause issues in the code when the value is zero but incorrectly treated as false
let height = 0;

alert(height || 100); // 100
alert(height ?? 100); // 0

Ref: https://javascript.info/nullish-coalescing-operator
Object shorthand
Provides a simpler way when creating an object who's keys have the same name as the variables passed-in as properties
let cat = 'Miaow';
let dog = 'Woof';
let bird = 'Peet peet';

let someObject = {
  cat,
  dog,
  bird
}

Ref https://alligator.io/js/object-property-shorthand-es6/
ISSUE TYPE

Enhancement Request

Hi @davidjumani @utchoang - can you please confirm if PR #5167 which has been merged fixes this issue?

@nvazquez This is a code wide change, the PR only fixes it in one place

Thanks @davidjumani, please advise if/when there are more PRs about this issue. Perhaps we can also track them on the issue description using a task list

Hello, I am Yashika Sarkar and I am new to open source but since ACS projects(2021) interests me I was going through the issues and I came across title "Optioinal chainging", I am not sure if it's an error but shouldn't it be optional chaining?
#4956
Thank you.

cc @borisstoyanov

@utchoang  Hello, I have come across certain lines of codes where optional chaining may be used .
Please let me know if it is okay to modify them or not
These are the screenshots-
 last line
 first and last line
 fourth line

@Yashikaone These sections may be subject to optional code.

Looks like these have been fixed, if not pl re-reopen @davidjumani

authors: Yashikaone davidjumani nvazquez rohityadavcloud utchoang

repo: apache/cloudstack


text:
CLOUDSTACK-8829 : Consecutive cold migration fails.

Issue - Consecutive VM cold migration fails.
Root Cause Analysis - In case of VMware, if VM is being cold migrated between clusters belonging to two different VMware DCs, Cloudstack unregisters the VM from the source host and cleans up the associated VM files. The check if a VM is being cold migrated across DCs is made using the source host id. In case of consecutive cold migrations since the source host id of a VM is NULL and no VM exists, Cloudstack should skip the check
Proposed Solution - Attempt to unregister a VM in another DC, only if there is a host associated with a VM.
This closes #797

Hi @borisstoyanov Bhai,
I need one help. Could you please run relevent test cases on this PR ?

@priyankparihar I can trigger smoketests for that.
@blueorangutan package

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-684

@blueorangutan test centos7 vmware5.5u3

@borisstoyanov unsupported parameters provided. Supported mgmt server os are: centos6, centos7, ubuntu. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-ubuntu, xenserver-65sp1, xenserver-62sp1, vmware-60u2, vmware-55u3, vmware-51u1, vmware-50u1

@blueorangutan test centos7 vmware-55u3

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + vmware-55u3) has been kicked to run smoke tests

Trillian test result (tid-1044)
Environment: vmware-55u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43632 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2056-t1044-vmware-55u3.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermitten failure detected: /marvin/tests/smoke/test_network_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Test completed. 48 look ok, 3 have error(s)

Test
Result
Time (s)
Test File

test_04_rvpc_privategw_static_routes
Failure
893.79
test_privategw_acl.py

test_03_vpc_privategw_restart_vpc_cleanup
Failure
197.32
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Failure
146.87
test_privategw_acl.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Error
242.31
test_vpc_redundant.py

test_06_verify_guest_lspci_again
Error
0.00
test_deploy_virtio_scsi_vm.py

test_05_change_vm_ostype_restart
Error
0.00
test_deploy_virtio_scsi_vm.py

test_04_verify_guest_lspci
Error
0.00
test_deploy_virtio_scsi_vm.py

test_03_verify_libvirt_attach_disk
Error
0.00
test_deploy_virtio_scsi_vm.py

test_02_verify_libvirt_after_restart
Error
0.00
test_deploy_virtio_scsi_vm.py

test_01_verify_libvirt
Error
0.00
test_deploy_virtio_scsi_vm.py

ContextSuite context=TestDeployVirtioSCSIVM>:teardown
Error
0.00
test_deploy_virtio_scsi_vm.py

test_01_vpc_site2site_vpn
Success
335.86
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
156.81
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
558.25
test_vpc_vpn.py

test_02_VPC_default_routes
Success
319.34
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
691.35
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
646.78
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1539.26
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Success
644.06
test_vpc_redundant.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1335.05
test_vpc_redundant.py

test_09_delete_detached_volume
Success
20.82
test_volumes.py

test_06_download_detached_volume
Success
65.57
test_volumes.py

test_05_detach_volume
Success
100.28
test_volumes.py

test_04_delete_attached_volume
Success
15.22
test_volumes.py

test_03_download_attached_volume
Success
10.28
test_volumes.py

test_02_attach_volume
Success
58.75
test_volumes.py

test_01_create_volume
Success
445.21
test_volumes.py

test_change_service_offering_for_vm_with_snapshots
Success
459.08
test_vm_snapshots.py

test_03_delete_vm_snapshots
Success
275.25
test_vm_snapshots.py

test_02_revert_vm_snapshots
Success
199.11
test_vm_snapshots.py

test_01_create_vm_snapshots
Success
163.69
test_vm_snapshots.py

test_deploy_vm_multiple
Success
293.12
test_vm_life_cycle.py

test_deploy_vm
Success
0.03
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.03
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
26.82
test_vm_life_cycle.py

test_09_expunge_vm
Success
125.23
test_vm_life_cycle.py

test_08_migrate_vm
Success
101.45
test_vm_life_cycle.py

test_07_restore_vm
Success
0.12
test_vm_life_cycle.py

test_06_destroy_vm
Success
5.13
test_vm_life_cycle.py

test_03_reboot_vm
Success
5.14
test_vm_life_cycle.py

test_02_start_vm
Success
20.25
test_vm_life_cycle.py

test_01_stop_vm
Success
5.12
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
236.69
test_templates.py

test_08_list_system_templates
Success
0.03
test_templates.py

test_07_list_public_templates
Success
0.06
test_templates.py

test_05_template_permissions
Success
0.07
test_templates.py

test_04_extract_template
Success
20.65
test_templates.py

test_03_delete_template
Success
5.13
test_templates.py

test_02_edit_template
Success
90.15
test_templates.py

test_01_create_template
Success
120.95
test_templates.py

test_10_destroy_cpvm
Success
241.82
test_ssvm.py

test_09_destroy_ssvm
Success
233.64
test_ssvm.py

test_08_reboot_cpvm
Success
156.69
test_ssvm.py

test_07_reboot_ssvm
Success
158.53
test_ssvm.py

test_06_stop_cpvm
Success
176.95
test_ssvm.py

test_05_stop_ssvm
Success
213.82
test_ssvm.py

test_04_cpvm_internals
Success
1.12
test_ssvm.py

test_03_ssvm_internals
Success
3.38
test_ssvm.py

test_02_list_cpvm_vm
Success
0.13
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.14
test_ssvm.py

test_02_list_snapshots_with_removed_data_store
Success
197.74
test_snapshots.py

test_01_snapshot_root_disk
Success
26.33
test_snapshots.py

test_04_change_offering_small
Success
98.24
test_service_offerings.py

test_03_delete_service_offering
Success
0.04
test_service_offerings.py

test_02_edit_service_offering
Success
0.08
test_service_offerings.py

test_01_create_service_offering
Success
0.10
test_service_offerings.py

test_02_sys_template_ready
Success
0.13
test_secondary_storage.py

test_01_sys_vm_start
Success
0.19
test_secondary_storage.py

test_09_reboot_router
Success
120.79
test_routers.py

test_08_start_router
Success
110.76
test_routers.py

test_07_stop_router
Success
25.25
test_routers.py

test_06_router_advanced
Success
0.06
test_routers.py

test_05_router_basic
Success
0.04
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.66
test_routers.py

test_03_restart_network_cleanup
Success
130.98
test_routers.py

test_02_router_internal_adv
Success
1.09
test_routers.py

test_01_router_internal_basic
Success
0.57
test_routers.py

test_router_dns_guestipquery
Success
77.40
test_router_dns.py

test_router_dns_externalipquery
Success
0.08
test_router_dns.py

test_router_dhcphosts
Success
141.15
test_router_dhcphosts.py

test_router_dhcp_opts
Success
21.67
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
0.10
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
30.35
test_reset_vm_on_reboot.py

test_createRegion
Success
0.04
test_regions.py

test_create_pvlan_network
Success
5.23
test_pvlan.py

test_dedicatePublicIpRange
Success
0.46
test_public_ip_range.py

test_01_vpc_privategw_acl
Success
193.07
test_privategw_acl.py

test_03_migration_options_storage_tags
Success
37.21
test_primary_storage.py

test_02_edit_primary_storage_tags
Success
0.12
test_primary_storage.py

test_01_primary_storage_nfs
Success
37.31
test_primary_storage.py

test_01_deploy_vms_storage_tags
Success
46.62
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.25
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
15.52
test_portable_publicip.py

test_isolate_network_password_server
Success
59.18
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.13
test_over_provisioning.py

test_oobm_zchange_password
Success
30.86
test_outofbandmanagement.py

test_oobm_multiple_mgmt_server_ownership
Success
16.43
test_outofbandmanagement.py

test_oobm_issue_power_status
Success
10.28
test_outofbandmanagement.py

test_oobm_issue_power_soft
Success
15.42
test_outofbandmanagement.py

test_oobm_issue_power_reset
Success
15.40
test_outofbandmanagement.py

test_oobm_issue_power_on
Success
15.41
test_outofbandmanagement.py

test_oobm_issue_power_off
Success
10.34
test_outofbandmanagement.py

test_oobm_issue_power_cycle
Success
15.35
test_outofbandmanagement.py

test_oobm_enabledisable_across_clusterzones
Success
82.71
test_outofbandmanagement.py

test_oobm_enable_feature_valid
Success
5.21
test_outofbandmanagement.py

test_oobm_enable_feature_invalid
Success
0.10
test_outofbandmanagement.py

test_oobm_disable_feature_valid
Success
5.22
test_outofbandmanagement.py

test_oobm_disable_feature_invalid
Success
0.10
test_outofbandmanagement.py

test_oobm_configure_invalid_driver
Success
0.08
test_outofbandmanagement.py

test_oobm_configure_default_driver
Success
0.08
test_outofbandmanagement.py

test_oobm_background_powerstate_sync
Success
29.45
test_outofbandmanagement.py

test_extendPhysicalNetworkVlan
Success
15.33
test_non_contigiousvlan.py

test_01_nic
Success
520.59
test_nic.py

test_releaseIP
Success
318.54
test_network.py

test_reboot_router
Success
604.90
test_network.py

test_public_ip_user_account
Success
10.33
test_network.py

test_public_ip_admin_account
Success
40.36
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
77.19
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.80
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
124.71
test_network.py

test_delete_account
Success
277.99
test_network.py

test_02_port_fwd_on_non_src_nat
Success
55.81
test_network.py

test_01_port_fwd_on_src_nat
Success
111.86
test_network.py

test_nested_virtualization_vmware
Success
424.40
test_nested_virtualization.py

test_nic_secondaryip_add_remove
Success
293.34
test_multipleips_per_nic.py

test_list_zones_metrics
Success
0.26
test_metrics_api.py

test_list_volumes_metrics
Success
5.45
test_metrics_api.py

test_list_vms_metrics
Success
187.01
test_metrics_api.py

test_list_pstorage_metrics
Success
0.35
test_metrics_api.py

test_list_infrastructure_metrics
Success
0.44
test_metrics_api.py

test_list_hosts_metrics
Success
0.48
test_metrics_api.py

test_list_clusters_metrics
Success
0.37
test_metrics_api.py

login_test_saml_user
Success
19.32
test_login.py

test_assign_and_removal_lb
Success
148.12
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
207.03
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
207.83
test_loadbalance.py

test_03_list_snapshots
Success
0.09
test_list_ids_parameter.py

test_02_list_templates
Success
0.04
test_list_ids_parameter.py

test_01_list_volumes
Success
0.03
test_list_ids_parameter.py

test_07_list_default_iso
Success
0.07
test_iso.py

test_05_iso_permissions
Success
0.06
test_iso.py

test_04_extract_Iso
Success
5.18
test_iso.py

test_03_delete_iso
Success
95.25
test_iso.py

test_02_edit_iso
Success
0.08
test_iso.py

test_01_create_iso
Success
21.00
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
490.08
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
359.02
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
968.72
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
807.22
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.27
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.14
test_global_settings.py

test_rolepermission_lifecycle_update
Success
6.19
test_dynamicroles.py

test_rolepermission_lifecycle_list
Success
6.03
test_dynamicroles.py

test_rolepermission_lifecycle_delete
Success
6.00
test_dynamicroles.py

test_rolepermission_lifecycle_create
Success
5.91
test_dynamicroles.py

test_rolepermission_lifecycle_concurrent_updates
Success
6.05
test_dynamicroles.py

test_role_lifecycle_update_role_inuse
Success
5.92
test_dynamicroles.py

test_role_lifecycle_update
Success
11.04
test_dynamicroles.py

test_role_lifecycle_list
Success
5.92
test_dynamicroles.py

test_role_lifecycle_delete
Success
10.97
test_dynamicroles.py

test_role_lifecycle_create
Success
5.92
test_dynamicroles.py

test_role_inuse_deletion
Success
5.87
test_dynamicroles.py

test_role_account_acls_multiple_mgmt_servers
Success
8.29
test_dynamicroles.py

test_role_account_acls
Success
8.60
test_dynamicroles.py

test_default_role_deletion
Success
6.04
test_dynamicroles.py

test_04_create_fat_type_disk_offering
Success
0.08
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.04
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.06
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.10
test_disk_offerings.py

test_01_create_disk_offering
Success
0.10
test_disk_offerings.py

test_deployvm_userdispersing
Success
81.02
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
55.83
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
166.37
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
30.53
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
126.31
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
5.87
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
5.85
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
6.00
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
187.85
test_deploy_vm_iso.py

test_3d_gpu_support
Success
445.43
test_deploy_vgpu_enabled_vm.py

test_DeployVmAntiAffinityGroup
Success
176.92
test_affinity_groups.py

test_08_resize_volume
Skipped
5.12
test_volumes.py

test_07_resize_fail
Skipped
10.28
test_volumes.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_static_role_account_acls
Skipped
0.03
test_staticroles.py

test_11_ss_nfs_version_on_ssvm
Skipped
0.02
test_ssvm.py

test_01_scale_vm
Skipped
63.32
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.04
test_primary_storage.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
1.34
test_deploy_vgpu_enabled_vm.py


text:
Cordova Docs Layout is Messed Up

Issue Type
[X ] Bug
[ ] Feature Request
Priority
[ ] Minor
[ ] Major
[ X] Critical
[ ] Blocker
Environment

OS: Mac
Browser: Chrome

Description
Recently the Cordova Plugin docs have a CSS layout issue. See screenshot for an example and a URL of the corresponding issue.

hmm, we did deploy an update to the site recently to adjust some styling for better readability, but none of my coworkers see this problem in Chrome. If you've visited this page in the past week and then visited it after we deployed the update, it's possible that you're seeing a cached version of the stylesheet. Could you try a force refresh or clearing your browser cache?
For reference, this is what I see at that URL:

That was it!  Thank you.

Ouch, we should have a cache busting mechanism (like appending resource hashes to resource names) in our build. This probably concerns more than only one user.
I guess any react app created from a recent boilerplate should have something like this.

No React in Cordova docs for sure ;)

@janpio I'm confused. I'm almost certain I edited a *.jsx file in #848 :confused:

... which is the plugins subpage/project, which is also part of the cordova-docs repo, but not the Cordova documentation pages. (cordova-docs could/should be called cordova-website, but isn't.)

@janpio I think 3 years ago someone tried to rename cordova-docs to cordova-website but the repo name was not updated. You can see in package.json that it was changed then from cordova-docs to cordova-website.

I stand by my statement that any part of the docs should have cache busting.

100% agreement here.

I think 3 years ago someone tried to rename cordova-docs to cordova-website but the repo name was not updated. You can see in package.json that it was changed then from cordova-docs to cordova-website.

It's very possible the repo was named that way some time ago... many strange things happened here already ;) Do you think it is worth to rename it now @erisu?

authors: GimpMaster dpogue erisu janpio raphinesse

repo: apache/cordova-docs

license_source: bigquery


text:
Document new management subsystems (smoosh, ioq, ken)

[x] smoosh
[x] ioq
[ ] ken

Smoosh and IOQ both have extensive operator guides in the actual code repos; these could probably be edited and ported over to docs.couchdb.org. Ken does not.

https://github.com/apache/couchdb-documentation/pull/438

https://github.com/apache/couchdb-documentation/pull/439

IOQ docs not merged. There's a ML thread to figure out what needs to be documented.

ioq merged, closing

authors: denib janl kocolosk

repo: apache/couchdb


text:
Improve binary scalars display

Which issue does this PR close?

Closes #.
Rationale for this change

Binary scalars might be large, which makes plan printing so long.
What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Sure, added a test.

Showing a binary as a collection of numbers from 0 to 255 is a bit uncommon. What about showing it as hex or base64 instead?

Showing a binary as a collection of numbers from 0 to 255 is a bit uncommon. What about showing it as hex or base64 instead?

I don't have strong opinion.

Then let's go for hex since base64 is a) another dependency and b) is maybe more opinionated and harder to recognize.

authors: crepererum lewiszlw

repo: apache/datafusion


text:
[Feature][Python] Python API CLI Enhancement

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Make pydolphinscheduler's CLI more powerful, make it can operate the model of DolphinScheduler, run pydolphinscheduler's code, visualize its DAG graph in the terminal.
Content
Project/Tenant/User Side

[ ] Add CURD to the project/tenant/user section of the API and add its tests.
[ ] Add Python CLI support for project/tenant/user

[ ] Add CURD calles from the python client to the python gateway on the project/tenant/user side.
[ ] Add Python CLI command for project/tenant/user and it's tests.
[ ] Add documentation for the project/tenant/user command.

WorkFlow Side

[ ] Add CLI support for running DAG files.

[ ] Add support for parsing /dags folders.
[ ] Add Python CLI command for running DAG files and it's tests.
[ ] Add documentation running DAG files.

[ ] Add CURD to the workflow section of the API and add its tests.
[ ] Add Python CLI support for workflow

[ ] Add CURD and other calles from the python client to the python gateway on the workflow side.
[ ] Add Python CLI command for workflow and it's tests.
[ ] Add documentation for the workflow command.

[ ] Add CLI support for task graph visualization and add its tests.

Use case
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Tips: if you already have exists sub-issue or sub-PR, you can change it to
  * [x] condition. #453
  * [x] switch. https://github.com/apache/dolphinscheduler/issues/6561

and it will reduce like

the convenient is when related issue or pr is be closed, you list will be auto-click by github itself

I suggest we separate Add CURD to the project/tenant/user section of the API and add its tests. into three PR, and combine the python api code into it. and then add CLI part, just like

Add CURD to the project for python gateway service and python api side its tests.
Add CLI for project

Only by completing those jobs, we can say completed a single submodule of this topic, whcih including java code, python code, UT and documentation. And we can estimate the future time job according to the project submodule we have just finish

authors: lyleshaw zhongjiajie

repo: apache/dolphinscheduler


text:
[Improvement][all] Field injection is not recommended Inspection info: Spring Team recommends: "Always use constructor based dependency injection in your beans. Always use assertions for mandatory dependencies".

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
I found out that our project uses @Autowired, spring Team recommends: "Always use constructor based dependency injection in your beans. Always use assertions for mandatory dependencies".
https://blog.marcnuri.com/field-injection-is-not-recommended.
So I think this is a needed optimization point.

[ ]  dolphinscheduler-api https://github.com/apache/dolphinscheduler/pull/13214
[ ]  dolphinscheduler-dao https://github.com/apache/dolphinscheduler/pull/13215
[ ]  dolphinscheduler-master https://github.com/apache/dolphinscheduler/pull/13216
[ ] dolphinscheduler-scheduler-plugin https://github.com/apache/dolphinscheduler/pull/13217
[ ] dolphinscheduler-service https://github.com/apache/dolphinscheduler/pull/13218
[ ] dolphinscheduler-tools https://github.com/apache/dolphinscheduler/pull/13219
[ ] dolphinscheduler-woker https://github.com/apache/dolphinscheduler/pull/13220

Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

This pr has changed a lot, so I split the task, and friends who need to participate can also claim it.

This pr has changed a lot, so I split the task, and friends who need to participate can also claim it.

@EricGao888 @SbloodyS do you have any suggestions?

Hi @liugddx , may I ask whether you plan to wipe out all the @Automwired annotations in this project, or just part of them?

Hi @liugddx , may I ask whether you plan to wipe out all the @Automwired annotations in this project, or just part of them?

I am going to replace all @Automwired annotations in this project.

Another good good first issue that I think I can contribute to is to replace logging with @Slf4j.You can contribute and don't need to be very familiar with the system. I think this is very friendly.I would also like to contribute in this way.

this is just a IDEA warnning, but i do think that construct will better than@Autowired, In terms of readability, @Autowired is better, and if we will inject many beans, o huge constuct may not be a good idea. WDYT

Thanks for the reply, if you write the constructor directly, it will make the code poorly readable, but we use lombok to solve this problem. In addition, this is not just a warning from IDEA, you can see https://blog.marcnuri.com/field-injection-is-not-recommended for details. Of course, the purpose of my mentioning this PR is to lower my contribution threshold, and I am happy to contribute my strength to dolphinscheduler.

Hi @liugddx , may I ask whether you plan to wipe out all the @Automwired annotations in this project, or just part of them?

Please tell me if this is a meaningful issue? If not, I will close the issue and related pr, thanks.

Hi @liugddx , may I ask whether you plan to wipe out all the @Automwired annotations in this project, or just part of them?

Please tell me if this is a meaningful issue? If not, I will close the issue and related pr, thanks.

I'm +1 to this.

not stale

authors: EricGao888 liugddx zhongjiajie


text:
[Feature][Metrics] Add metrics for workflow runtime

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
If most of times, we want to monitor the workflow at workflow level.
e.g.
workflowA:
        executeCount: 3
        successCount: 1
        failedCount:  2
        failoverCount: xx
        successTaskCount: xx

Can we output this to Prometheus?
At the workflow level, the label count should be acceptable.
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

cc @caishunfeng @EricGao888

cc @caishunfeng @EricGao888

@ruanwenjun I'm + 1 to this. However, it would be better if we have related retention policy. For example, we could add a scheduled job to periodically clean up the metrics for deleted workflow / tasks or those which haven't be triggered or scheduled for a long time.

cc @caishunfeng @EricGao888

@ruanwenjun I'm + 1 to this. However, it would be better if we have related retention policy. For example, we could add a scheduled job to periodically clean up the metrics for deleted workflow / tasks or those which haven't be triggered or scheduled for a long time.

Good catch, when we delete a workflow, the API server can broadcast an RPC message to all master to delete the label.

related: #10525


text:
[BUG] report an error when use python-plugin run as Debug on win10

Describe the bug
report an error when use python-plugin run as Debug on win10
See below for details.
Which version of Dolphin Scheduler:
-[dev-1.3]
Additional context
error-log:
[taskAppId=TASK-7-8-11]:[97] - python task failure
java.io.IOException: Cannot run program "sudo" (in directory "\tmp\dolphinscheduler\exec\process\2\7\8\11"): CreateProcess error=2, 系统找不到指定的文件。
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.buildProcess(AbstractCommandExecutor.java:118)
at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:146)
at org.apache.dolphinscheduler.server.worker.task.python.PythonTask.handle(PythonTask.java:90)
at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: CreateProcess error=2, 系统找不到指定的文件。
@liwenhe1993

This problem still exists in version 2.0.5

3.2.1 版本中本地执行 net user xxx  存在 CreateProcess error=5, 拒绝访问

authors: ITBOX-ITBOY glmapper torysuper


text:
[Feature][JsonSplit-api]processDefinition queryProcessDefinitionAllByProjectId interface

from #5498
Change the request parameter projectId to projectCode,including the front end and controller interface

I will implement the back-end java code logic

fixed by #5744


text:
Remove datasource in procedure task

Purpose of the pull request
close #13197
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Codecov Report

Merging #13198 (99396cc) into dev (43e6ac3) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##                dev   #13198      +/-   ##
============================================
- Coverage     39.37%   39.34%   -0.03%
+ Complexity     4278     4272       -6
============================================
  Files          1066     1066
  Lines         40479    40450      -29
  Branches       4657     4642      -15
============================================
- Hits          15937    15916      -21
+ Misses        22755    22744      -11
- Partials       1787     1790       +3

Impacted Files
Coverage Δ

...source/api/plugin/DataSourceProcessorProvider.java
0.00% <0.00%> (ø)

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...erver/master/processor/queue/TaskEventService.java
75.00% <0.00%> (-5.36%)
:arrow_down:

...e/dolphinscheduler/remote/NettyRemotingClient.java
50.00% <0.00%> (-2.78%)
:arrow_down:

...dolphinscheduler/remote/future/ResponseFuture.java
81.96% <0.00%> (-1.64%)
:arrow_down:

...hinscheduler/plugin/alert/script/ScriptSender.java
70.58% <0.00%> (-1.64%)
:arrow_down:

...cheduler/plugin/alert/dingtalk/DingTalkSender.java
34.13% <0.00%> (-0.78%)
:arrow_down:

...rver/master/runner/task/BlockingTaskProcessor.java
75.86% <0.00%> (-0.55%)
:arrow_down:

...inscheduler/plugin/registry/etcd/EtcdRegistry.java
50.69% <0.00%> (-0.35%)
:arrow_down:

.../dolphinscheduler/plugin/task/datax/DataxTask.java
36.62% <0.00%> (-0.26%)
:arrow_down:

... and 18 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@ruanwenjun  To avoid using connections from thread pool? create a separated connection in each task?

@ruanwenjun To avoid using connections from thread pool? create a separated connection in each task?

Yes, you can see the issue description.

@ruanwenjun Can we add this pr as an improvement? Will this change affect the existing usage?

@ruanwenjun To avoid using connections from thread pool? create a separated connection in each task?

Yes, you can see the issue description.

Is it possible to create too many connections if there are lots of tasks?

Yes, this will need to create connection for each processdure task, but right now, I don't find other solution. Since the temporary table is bound to the session, if we don't close the session the temporary will always exist.

@ruanwenjun Can we add this pr as an improvement? Will this change affect the existing usage?

Ok, I add improvement table, this PR will not affect the current usage

@ruanwenjun To avoid using connections from thread pool? create a separated connection in each task?

Yes, you can see the issue description.

Is it possible to create too many connections if there are lots of tasks?

Yes, this will need to create connection for each processdure task, but right now, I don't find other solution. Since the temporary table is bound to the session, if we don't close the session the temporary will always exist.

If we can't find another solution, then current  is the best one. Too many connections for a DB is a disaster absolutely, dropping a temporary table is just a flaw.

@ruanwenjun To avoid using connections from thread pool? create a separated connection in each task?

Yes, you can see the issue description.

Is it possible to create too many connections if there are lots of tasks?

Yes, this will need to create connection for each processdure task, but right now, I don't find other solution. Since the temporary table is bound to the session, if we don't close the session the temporary will always exist.

If we can't find another solution, then current is the best one. Too many connections for a DB is a disaster absolutely, dropping a temporary table is just a flaw.

This is a bug, will cause the procedure task cannot use, the temporary table is shared by multiple task,  if you create a template task in the procedure task, you will get an error, since there might already exist a temporary table with the same name created by another task.

If user drop it manually, is it OK?

If user drop it manually, is it OK?

It's OK for the temporary table if user drop in the end of procedure task, but we cannot make sure all user will drop their temporary table in their procedure task.

authors: caishunfeng codecov-commenter gabrywu ruanwenjun


text:
fix:Modify the datax path when Windows calls the datax script

Purpose of the pull request
This pull makes DataX Task could be used on WindowsOS
Brief change log
Modify the datax path when WindowsOS calls the datax script
Verify this pull request
This pull request is code cleanup without any test coverage.

do you already run dolphinscheudler in windows in production? I think we should ignore the windows env because all of other task without consider and will have simple issue

do you already run dolphinscheudler in windows in production? I think we should ignore the windows env because all of other task without consider and will have simple issue

Windows is not our production environment, but it needs to be developed on Windows. The production environment is on Ubuntu and has been successfully deployed. I have successfully run the tasks of Python and Datax based on the source code of version 3.1.7 in the Windows 11 environment. As the development needs to be on Windows, I have made modifications to the code for the Windows section. This submission of code solves the problem of the source code being unable to run datax tasks on Windows

do you already run dolphinscheudler in windows in production? I think we should ignore the windows env because all of other task without consider and will have simple issue

+1

closing

authors: Rontic SbloodyS zhongjiajie


text:
Remove unused caffeine cache

Purpose of the pull request
Since we have already removed cache, we need to remove this kind of config. And caffeine is also not needed.
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

(or)
If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 39.17%. Comparing base (2b5f843) to head (9674b78).

:exclamation: Current head 9674b78 differs from pull request most recent head dfa7bee. Consider uploading reports for the commit dfa7bee to get more accurate results

Additional details and impacted files
@@             Coverage Diff              @@
##                dev   #15830      +/-   ##
============================================
- Coverage     39.17%   39.17%   -0.01%
  Complexity     4889     4889
============================================
  Files          1323     1323
  Lines         45187    45152      -35
  Branches       4811     4788      -23
============================================
- Hits          17703    17689      -14
+ Misses        25603    25582      -21
  Partials       1881     1881

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter ruanwenjun


text:
[Bug] 2.02 streamload 异常

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
2.0.2
What's Wrong?
升级到2.0.2后， 将数据导入创建了物化视图的表， 当导入数据未完全覆盖所创建物化视图时， 导入失败。
表结构 ：
CREATE TABLE test (
a tinyint(4) NULL DEFAULT "0" ,
b int(11) DEFAULT "0" ,
c int(11) DEFAULT "0" ,
t DATETIMEV2 COMMENT "时间",
d bigint(20) SUM NULL DEFAULT "0",
e bigint(20) SUM NULL DEFAULT "0"
)
AGGREGATE KEY(a, b, c, t)
PARTITION BY RANGE(t) ()
DISTRIBUTED BY HASH(a) BUCKETS AUTO
PROPERTIES
(
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "DAY",
"dynamic_partition.start" = "-3",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.time_zone" = "Asia/Shanghai",
"dynamic_partition.create_history_partition" = "true",
"dynamic_partition.history_partition_num" = "3",
"dynamic_partition.replication_num" = "2",
"light_schema_change" = "true"
);
创建物化视图：
CREATE MATERIALIZED VIEW mv_test AS
SELECT a,b,t,SUM(d) FROM test GROUP BY 1,2,3
导入列 a,c,t,d,e, 发生异常 ：
{
"TxnId": 4160958,
"Label": "7b68ce49-2ad6-41d2-8706-a0b18cd4d897",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Fail",
"Message": "[ANALYSIS_ERROR]errCode = 2, detailMessage = unknown reference column, column=mv_b, reference=b\n0. StackTrace::tryCapture() @ 0x000000000ba70197 in /opt/apache-doris/be/lib/doris_be\n1. doris::get_stack_traceabi:cxx11 @ 0x000000000ba6e72d in /opt/apache-doris/be/lib/doris_be\n2. doris::Status doris::Status::Error(int, std::basic_string_view<char, std::char_traits >) @ 0x000000000af07e2b in /opt/apache-doris/be/lib/doris_be\n3. doris::Status::create(doris::TStatus const&) @ 0x000000000b747e3e in /opt/apache-doris/be/lib/doris_be\n4. doris::StreamLoadAction::_process_put(doris::HttpRequest*, std::shared_ptrdoris::StreamLoadContext) @ 0x000000000bec85d5 in /opt/apache-doris/be/lib/doris_be\n5. doris::StreamLoadAction::on_header(doris::HttpRequest*, std::shared_ptrdoris::StreamLoadContext) @ 0x000000000bec60aa in /opt/apache-doris/be/lib/doris_be\n6. doris::StreamLoadAction::on_header(doris::HttpRequest*) @ 0x000000000bec51f2 in /opt/apache-doris/be/lib/doris_be\n7. doris::EvHttpServer::on_header(evhttp_request*) @ 0x000000000bee35d1 in /opt/apache-doris/be/lib/doris_be\n8. ? @ 0x00000000131f3d8a in /opt/apache-doris/be/lib/doris_be\n9. bufferevent_run_readcb @ 0x00000000131d6421 in /opt/apache-doris/be/lib/doris_be\n10. ? @ 0x00000000131f82c3 in /opt/apache-doris/be/lib/doris_be\n11. ? @ 0x00000000131df229 in /opt/apache-doris/be/lib/doris_be\n12. ? @ 0x00000000131df8a7 in /opt/apache-doris/be/lib/doris_be\n13. ? @ 0x00000000131e1ed8 in /opt/apache-doris/be/lib/doris_be\n14. ? @ 0x000000000bee3aa7 in /opt/apache-doris/be/lib/doris_be\n15. doris::ThreadPool::dispatch_thread() @ 0x000000000baacdaf in /opt/apache-doris/be/lib/doris_be\n16. doris::Thread::supervise_thread(void*) @ 0x000000000baa2d3c in /opt/apache-doris/be/lib/doris_be\n17. start_thread @ 0x0000000000007dc5 in /usr/lib64/libpthread-2.17.so\n18. __clone @ 0x000000
What You Expected?
希望bug能得到修复
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

补充图片

I tested it on 2.0.3 and there was no problem. Please upgrade after the version is released.

2.0.2.1会有问题吗？ 2.0.3看起来有点遥远

this week will release

repo: apache/doris


text:
[typo] fix some typo in comments

Proposed changes

fix some typo in comments which will not affect the normal function.
Issue Number: close #xxx

Problem Summary:
Describe the overview of changes.
Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (No Need)
Has document been added or modified: (No)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

iff means if and only if so it's actually not a typo


text:
feature-wip explode support more sub types

Proposed changes
Issue Number: close #10052
Problem Summary:
refractor explode and explode_outer:

support more sub types;
support nullable elements of array;

Usage example:
mysql> set enable_vectorized_engine = true
mysql> set enable_array_type = true

mysql> select e1 from (select 1 k1) as t lateral view explode([1,2,3]) tmp1 as e1;
+------+
| e1   |
+------+
|    1 |
|    2 |
|    3 |
+------+

mysql> select e1 from (select 1 k1) as t lateral view explode_outer(null) tmp1 as e1;
+------+
| e1   |
+------+
| NULL |
+------+

mysql> select e1 from (select 1 k1) as t lateral view explode([]) tmp1 as e1;
Empty set (0.010 sec)

mysql> select e1 from (select 1 k1) as t lateral view explode([null,1,null]) tmp1 as e1;
+------+
| e1   |
+------+
| NULL |
|    1 |
| NULL |
+------+

mysql> select e1 from (select 1 k1) as t lateral view explode_outer([null,1,null]) tmp1 as e1;
+------+
| e1   |
+------+
| NULL |
|    1 |
| NULL |
+------+

Checklist(Required)

Does it affect the original behavior: (Yes/No/I Don't know)
Has unit tests been added: (Yes/No/No Need)
Has document been added or modified: (Yes/No/No Need)
Does it need to update dependencies: (Yes/No)
Are there any changes that cannot be rolled back: (Yes/No)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

maybe need rebase to fix p0

authors: cambyzju yangzhg


text:
fixchange defaultConcreteType function's return value for decimal

Proposed changes
Issue Number: close #xxx
<--Describe your changes.-->
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


text:
Opt short circuit probe for join node

Proposed changes
Support the _short_circuit_for_probe for join node:

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


text:
Enhancement Add HttpError to HttpClient::execute_with_retry

Proposed changes
Add HttpError to HttpClient::execute_with_retry
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 52.73 seconds
stream load tsv:          506 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17160851715 Bytes

authors: JackDrogon hello-stephen


text:
fixfix mysql catalog not check auth

Proposed changes
Issue Number: close #xxx
set needCheckColumnPriv=true
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

旧优化器改回之前的鉴权逻辑，就不会有现在的问题了


text:
[Bug](runtime filter) Fix runtime filter initialization

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
enhancement refine io error and report bad when disk is abnormal

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall


text:
FIX fix large int cast to decimalv3

Proposed changes
Issue Number: close #xxx
fix large int cast to decimalv3 , before if we make data more than max int64 cast to decimalv3 will make result overflow and incorrect
such as
mysql> select CAST(12345678901234567890123456789012345678 AS DECIMALV3(38, 0));
+------------------------------------------------------------------+
| CAST(12345678901234567890123456789012345678 AS DECIMALV3(38, 0)) |
+------------------------------------------------------------------+
|                                             -4302749291975740594 |
+------------------------------------------------------------------+

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fixfill up miss slot of order having project

Proposed changes
fill up miss slot of order having project such as
select a + 1 as c from t having by c > 2 order by a

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix fix wrong assert and cancel load error

Proposed changes

java.lang.IllegalArgumentException: null
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.qe.Coordinator.updateFragmentExecStatus(Coordinator.java:2590) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.QeProcessorImpl.reportExecStatus(QeProcessorImpl.java:253) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.service.FrontendServiceImpl.reportExecStatus(FrontendServiceImpl.java:960) ~[doris-fe.jar:1.2-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_352-352]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_352-352]
        at org.apache.doris.service.FeServer.lambda$start$0(FeServer.java:60) ~[doris-fe.jar:1.2-SNAPSHOT]
        at com.sun.proxy.$Proxy26.reportExecStatus(Unknown Source) ~[?:?]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3632) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.doris.thrift.FrontendService$Processor$reportExecStatus.getResult(FrontendService.java:3612) ~[fe-common-1.2-SNAPSHOT.jar:1.2-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.16.0.jar:0.16.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) ~[libthrift-0.16.0.jar:0.16.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_352-352]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_352-352]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_352-352]

Fix unable to cancel load

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
opt Purge all unused Jemalloc dirty pages for all arenas when exceed limit

add BE conf je_dirty_pages_mem_limit_percent, default 5% * mem_limit, purge all unused Jemalloc dirty pages for all arenas when exceed limit.

run buildall

run buildall

run buildall

run buildall


text:
COMPILE do not patch bitshuffle on macos

Proposed changes
This PR(https://github.com/apache/doris/pull/38378) introduces a patch, but the patch cannot be applied on macOS.
error info
mryange/doris/thirdparty/installed/lib/liblz4.a  /Users/mryange/doris/thirdparty/installed/lib/libzstd.a  bin/libclucene-core-static.a  bin/libclucene-shared-static.a && :
ld: warning: directory not found for option '-F/Applications/Xcode_15.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk/System/Library/Frameworks'

Temporary solution:
Revert this PR
cd thirdparty
rm -rf src/bitshuffle-0.5.1
rm installed/lib64/bitshuffle.a
./build-thirdparty.sh bitshuffle

Note, if you encounter this error ("download-thirdparty.sh: line 59: ${GIVEN_LIB,,}: bad substitution"), you can upgrade your bash version or manually modify the script.
Either pick the current PR,
cd thirdparty
rm -rf src/bitshuffle-0.5.1
rm installed/lib64/bitshuffle.a
./build-thirdparty.sh bitshuffle

If executed correctly, you will get:
===== Patching thirdparty archives...
MacOS. Skipping BITSHUFFLE patching.

or wait for the latest precompiled third-party library

run buildall

run buildall

run buildall


text:
featadd deadlock detection tool and monitored lock implementations #39015

Proposed changes
#39015
Description:
This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features:
AbstractMonitoredLock:
A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality:
Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg
2024-08-07 12:02:59  [ Thread-2:2006 ] - [ WARN ]  Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms
Thread stack trace:
	at java.lang.Thread.getStackTrace(Thread.java:1564)
	at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49)
	at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32)
	at org.example.ExampleService.timeout(ExampleService.java:17)
	at org.example.Main.lambda$test2$1(Main.java:39)
	at java.lang.Thread.run(Thread.java:750)

DeadlockCheckerTool:
Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces.
ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing. ##### eg
Thread Name: Thread-0
Thread State: WAITING
Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213
Lock Owner Name: Thread-1
Lock Owner Id: 12
Waited Time: -1
Blocked Time: -1
Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213
Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213
Stack Trace:
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22)
	at org.example.Main.lambda$testDeadLock$3(Main.java:79)
	at org.example.Main$$Lambda$1/1221555852.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)

2024-08-07 14:11:28  [ pool-1-thread-1:2001 ] - [ WARN ]  Deadlocks detected:
Thread Name: Thread-1
Thread State: WAITING
Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf
Lock Owner Name: Thread-0
Lock Owner Id: 11
Waited Time: -1
Blocked Time: -1
Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf
Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf
Stack Trace:
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
	at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22)
	at org.example.Main.lambda$testDeadLock$4(Main.java:93)
	at org.example.Main$$Lambda$2/1556956098.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:750)

benchmark
    @Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS)
    @Threads(1)

Benchmark                                                          Mode  Cnt       Score   Error   Units
LockBenchmark.testMonitoredLock                                   thrpt    2   15889.407          ops/ms
LockBenchmark.testMonitoredLock:·gc.alloc.rate                    thrpt    2     678.061          MB/sec
LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm               thrpt    2      56.000            B/op
LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space           thrpt    2     668.249          MB/sec
LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm      thrpt    2      55.080            B/op
LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space       thrpt    2       0.075          MB/sec
LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm  thrpt    2       0.006            B/op
LockBenchmark.testMonitoredLock:·gc.count                         thrpt    2      20.000          counts
LockBenchmark.testMonitoredLock:·gc.time                          thrpt    2       6.000              ms
LockBenchmark.testNativeLock                                      thrpt    2  103130.635          ops/ms
LockBenchmark.testNativeLock:·gc.alloc.rate                       thrpt    2      ≈ 10⁻⁴          MB/sec
LockBenchmark.testNativeLock:·gc.alloc.rate.norm                  thrpt    2      ≈ 10⁻⁶            B/op
LockBenchmark.testNativeLock:·gc.count                            thrpt    2         ≈ 0          counts

    @Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
    @Measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS)
    @Threads(100)

Benchmark                                                          Mode  Cnt       Score   Error   Units
LockBenchmark.testMonitoredLock                                   thrpt    2   10994.606          ops/ms
LockBenchmark.testMonitoredLock:·gc.alloc.rate                    thrpt    2     488.508          MB/sec
LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm               thrpt    2      56.002            B/op
LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space           thrpt    2     481.390          MB/sec
LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm      thrpt    2      55.163            B/op
LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space       thrpt    2       0.020          MB/sec
LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm  thrpt    2       0.002            B/op
LockBenchmark.testMonitoredLock:·gc.count                         thrpt    2      18.000          counts
LockBenchmark.testMonitoredLock:·gc.time                          thrpt    2       9.000              ms
LockBenchmark.testNativeLock                                      thrpt    2  558652.036          ops/ms
LockBenchmark.testNativeLock:·gc.alloc.rate                       thrpt    2       0.016          MB/sec
LockBenchmark.testNativeLock:·gc.alloc.rate.norm                  thrpt    2      ≈ 10⁻⁴            B/op
LockBenchmark.testNativeLock:·gc.count                            thrpt    2         ≈ 0          counts

Proposed changes
Issue Number: close #xxx

run buildall


text:
[fix] (Nereids) restrict the execution of match on the scan

Proposed changes
restrict the execution of match on the scan

run buildall


text:
fix fill default value for required field in TFrontendPingFrontendResult

Otherwise, if ping failed, some of required fields are not set, causing some unexpected error log like:

run buildall


text:
fix immediately return error when close wait failed

What problem does this PR solve?
Related PR: #38003
Problem Summary:
#38003 introduced a problem where the last sink node could report success even when close wait timeout, which may cause data loss.
Previously we made that change hoping to tolerate minority replica failure in this step.
However, it turns out the last sink node could miss tablet reports from downstreams in case of close wait failure.
This PR fixes the problem by return the close_wait error immediately.
The most common error in close wait is timeout, and it should not be fault tolerant on a replica basis anyways.
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[x] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[x] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[x] Yes.

Does this need documentation?

[x] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall


text:
Adding bitwise expressions

Related to #8560
This feature adds several bitwise expressions to be used on ingestion.
Use Case
Say you're doing CDC and your source table has a column of binary flags. For OLAP queries, it's extremely useful to have these flags extracted out into their own dimensions which can be done now on ingestion via the new expressions.
Example:

You might have a column in your source DB called flags that represents a series of binary flags. Bit 1 might mean an order was placed on mobile, bit 2 might mean it was purchased on web, bit 4 might mean it was the users first purchase.
Instead of ingesting the raw integer value of the flags column, you can break these columns out into something like mobile_ind, web_ind, first_purchase_ind where your expression can be bitwiseAnd(flags, 1), bitwiseAnd(flags, 2), bitwiseAnd(flags, 4).

Druid SQL currently does not support bitwise operations (#8560) which makes these even more valuable IMO.
Tested on a local cluster
Ingestion spec:
{
  "type": "index_parallel",
  "spec": {
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "inline",
        "data": "\"x\",\"y\"\n4,2\n8,4\n16,8\n3,2\n5,2"
      },
      "inputFormat": {
        "type": "csv",
        "findColumnsFromHeader": true
      }
    },
    "tuningConfig": {
      "type": "index_parallel",
      "partitionsSpec": {
        "type": "dynamic"
      }
    },
    "dataSchema": {
      "dataSource": "bitwise_test",
      "granularitySpec": {
        "type": "uniform",
        "queryGranularity": "NONE",
        "rollup": false,
        "segmentGranularity": "YEAR"
      },
      "timestampSpec": {
        "column": "!!!_no_such_column_!!!",
        "missingValue": "2010-01-01T00:00:00Z"
      },
      "transformSpec": {
        "transforms": [
          {
            "type": "expression",
            "name": "zBitwiseAnd",
            "expression": "bitwiseAnd(CAST(x, 'LONG'), CAST(y, 'LONG'))"
          },
          {
            "type": "expression",
            "name": "zBitwiseOr",
            "expression": "bitwiseOr(CAST(x, 'LONG'), CAST(y, 'LONG'))"
          },
          {
            "type": "expression",
            "name": "zBitwiseComplement",
            "expression": "bitwiseComplement(CAST(x, 'LONG'))"
          },
          {
            "type": "expression",
            "expression": "bitwiseShiftLeft(CAST(x, 'LONG'), CAST(y, 'LONG'))",
            "name": "zBitwiseShiftLeft"
          },
          {
            "type": "expression",
            "name": "zBitwiseShiftRight",
            "expression": "bitwiseShiftRight(CAST(x, 'LONG'), CAST(y, 'LONG'))"
          },
          {
            "type": "expression",
            "name": "zBitwiseXor",
            "expression": "bitwiseXor(CAST(x, 'LONG'), CAST(y, 'LONG'))"
          }
        ]
      },
      "dimensionsSpec": {
        "dimensions": [
          {
            "type": "long",
            "name": "x"
          },
          {
            "type": "long",
            "name": "y"
          },
          {
            "type": "long",
            "name": "zBitwiseAnd"
          },
          {
            "type": "long",
            "name": "zBitwiseComplement"
          },
          {
            "type": "long",
            "name": "zBitwiseOr"
          },
          {
            "type": "long",
            "name": "zBitwiseShiftLeft"
          },
          {
            "type": "long",
            "name": "zBitwiseShiftRight"
          },
          {
            "type": "long",
            "name": "zBitwiseXor"
          }
        ]
      }
    }
  }
}

This PR has:

[x] been self-reviewed.
[x] added documentation for new or modified features or behaviors.
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] been tested in a test Druid cluster.

Looks like the CI failed on spellcheck of the expression names. Let me know if you want me to change the names

Looks like the CI failed on spellcheck of the expression names. Let me know if you want me to change the names

Generally you'd just add them to the spell checker exclusions file: website/.spelling.

@kaplanmaxe thank you for the contribution!
There's another patch with a similar feature: #10084. I prefer the approach you took in this patch — of defining named functions — because it will be easier to integrate into the SQL layer. It is pretty quick to write a SQL operator that is backed by a native expr function. You also have a few more functions added (like bit shifts), which is great.
Some comments:

Please add some tests. You could start by copying the tests from #10084 and then adding new ones for the additional functions you have. That patch has a pretty good range of tests for a variety of cases. It's a bit more complicated because it also modified the grammar. Your tests would mostly be going in "FunctionTest".
The functions should be named like other native expr functions, using snake_case rather than camelCase.
If you're interested in adding SQL functions too, or as a follow up, check out IPv4AddressMatchOperatorConversion for an example of how to implement a SQL function that maps cleanly onto a native expr function. It is mostly just declaring the proper type and operand checking info for the SQL layer.

Thank you @gianm and @clintropolis for your very detailed reviews and feedback. As a first time contributor and someone unfamiliar with the code base, this was extremely helpful.
I will update this over the weekend likely. Unfortunately, i am effected by a major power outage atm, but hope it comes back soon. Druid has been amazing for our use case, and I'm really excited this is conceptually approved as it's the biggest thing affecting us atm. Expect to see some revisions soon within the next few days!

Looking forward to it, @kaplanmaxe.

I would love to see this functionality in druid. Any chance this could be fixed/merged?

Hey guys. Sorry I haven't responded recently.
I'm just going to be honest, our team found a workaround for this one, and I likely don't have the capacity to finish this up. I know this feature is going to be really valuable to end users, and I don't mind if someone else picks up this PR and finishes it up. Really sorry about that, but just want to be honest with intentions so you guys aren't waiting on me to pick it up. In the end, just want what is best for Druid

I'm just going to be honest, our team found a workaround for this one, and I likely don't have the capacity to finish this up. I know this feature is going to be really valuable to end users, and I don't mind if someone else picks up this PR and finishes it up. Really sorry about that, but just want to be honest with intentions so you guys aren't waiting on me to pick it up. In the end, just want what is best for Druid

@kaplanmaxe I should be able to pick this up sometime soon and get it finished up so we can get this feature in, thanks!

Let's keep this one open, stalebot.

Let's keep this one open, stalebot.

Ah, this one can be closed actually, i picked up the commit from here and added tests in #10605

Ah, this one can be closed actually, i picked up the commit from here and added tests in #10605

authors: clintropolis gianm kaplanmaxe teyeheimans

repo: apache/druid


text:
dubbo-2.7.18针对nacos 2.1.1版本的压测问题

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Ask your question here
因目前在做dubbo2.7.6->2.7.18 + nacos-client +nacos-server-2.1.1 的整体升级进行压测
报告显示在服务实例数均为一万的情况下
dubbo-2.7.6+nacos*-2.1.1 的cpu使用率为47%，9848端口建链数为4000左右
dubbo-2.7.18+nacos*-2.1.1 的cpu使用率为6%， 9848端口建链数为135左右
8848为注册中心端口，偏移量为9848
在相同服务器上，配置一样，想问题为何会有如此大的差别，是做什么对nacos的优化了吗？在报告上并没发现相关处理。

May related with https://github.com/apache/dubbo/pull/6571

请认真下看疑问点，我认为与此项修改并无关联

May related with #6571
请认真下看疑问点，我认为与此项修改并无关联

https://github.com/apache/dubbo/pull/6571 修改了注册中心的缓存，在这个提交之前，客户端发布了 n 个服务，就会建立 n 条连接，在这之后才会合并到同一个连接使用

authors: AlbumenJ anuserin

repo: apache/dubbo


text:
dubbo3目前是否有python的sdk，支持哪个版本的python

[ ] I have searched the issues of this repository and believe that this is not a duplicate.

Ask your question here
dubbo官网的python引导，目前的项目dubbo-python2，已经3年没有更新了

试试这个 https://gitee.com/happyplay/dubbo-python3


text:
[3.0]Triple protocol provider throw ClosedChannelException when invoke timeout

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

Dubbo version: 3.0.5-SNAPSHOT

Steps to reproduce this issue

Start provider and consumer use triple protocol.
Consumer invoke timeout interface.
Provider throw ClosedChannelException

java.nio.channels.ClosedChannelException
	at io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe.write(AbstractHttp2StreamChannel.java:916)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
	at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1015)
	at io.netty.handler.codec.http2.AbstractHttp2StreamChannel.write(AbstractHttp2StreamChannel.java:480)
	at org.apache.dubbo.rpc.protocol.tri.command.QueuedCommand$AbstractQueuedCommand.run(QueuedCommand.java:56)
	at org.apache.dubbo.rpc.protocol.tri.WriteQueue.flush(WriteQueue.java:67)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
[WARN][2021-12-22 15:40:12.099][NettyServerWorker-3-1][org.apache.dubbo.rpc.protocol.tri.ServerOutboundTransportObserver.lambda$onMetadata$0(ServerOutboundTransportObserver.java:42)ogger{50}]  [DUBBO] send header error endStream=true, dubbo version: 3.0.5-SNAPSHOT, current host: 10.128.49.130

When invoke timeout client will close channel.
https://github.com/apache/dubbo/blob/0765b612cb89fa22a40c3043e955552c8e941160/dubbo-rpc/dubbo-rpc-triple/src/main/java/org/apache/dubbo/rpc/protocol/tri/TripleClientRequestHandler.java#L57-L58

Pls. provide [GitHub address] to reproduce this issue.
Expected Behavior

Actual Behavior

If there is an exception, please attach the exception trace:
Just put your stack trace here!

This is to release client-local Http2StreamChannel when the server times out

authors: EarthChen haoyann


text:
optimize the same weight judgement

What is the purpose of the change
optimize the same weight judgement
Brief changelog
Move the logic to determine if the weights are the same before accumulation. So we can reduce addition operations
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 70.45%. Comparing base (afcf04d) to head (44d7627).

Additional details and impacted files
@@            Coverage Diff             @@
##              3.2   #13949      +/-   ##
==========================================
+ Coverage   70.43%   70.45%   +0.01%
==========================================
  Files        1607     1607
  Lines       70075    70075
  Branches    10100    10100
==========================================
+ Hits        49360    49373      +13
+ Misses      16080    16071       -9
+ Partials     4635     4631       -4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter mengle-yang


text:
[3.0] Improve dynamic configuration initialization

What is the purpose of the change

Initialize dynamic configuration in DubboBootstrap only
Get dynamic configuration from Environment only

Brief changelog
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #8294 (3df8d95) into 3.0 (3eff57a) will decrease coverage by 0.15%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                3.0    #8294      +/-   ##
============================================
- Coverage     64.03%   63.88%   -0.16%
- Complexity      310      311       +1
============================================
  Files          1077     1077
  Lines         45525    45482      -43
  Branches       6840     6810      -30
============================================
- Hits          29152    29054      -98
- Misses        13146    13198      +52
- Partials       3227     3230       +3

Impacted Files
Coverage Δ

...mmon/config/configcenter/DynamicConfiguration.java
72.22% <ø> (-7.78%)
:arrow_down:

...overnance/DefaultGovernanceRuleRepositoryImpl.java
100.00% <100.00%> (ø)

.../apache/dubbo/config/bootstrap/DubboBootstrap.java
70.68% <100.00%> (+0.08%)
:arrow_up:

...nfigcenter/nop/NopDynamicConfigurationFactory.java
50.00% <0.00%> (-50.00%)
:arrow_down:

...nfig/configcenter/nop/NopDynamicConfiguration.java
25.00% <0.00%> (-37.50%)
:arrow_down:

...ng/transport/dispatcher/all/AllChannelHandler.java
68.96% <0.00%> (-27.59%)
:arrow_down:

...figcenter/AbstractDynamicConfigurationFactory.java
75.00% <0.00%> (-25.00%)
:arrow_down:

...ng/transport/dispatcher/WrappedChannelHandler.java
47.82% <0.00%> (-15.22%)
:arrow_down:

...dubbo/common/status/support/LoadStatusChecker.java
50.00% <0.00%> (-11.54%)
:arrow_down:

...in/java/org/apache/dubbo/common/utils/JVMUtil.java
81.13% <0.00%> (-11.33%)
:arrow_down:

... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3eff57a...3df8d95. Read the comment docs.

authors: codecov-commenter kylixs


text:
Catch deploy error, improve module start waiting

What is the purpose of the change

Catch deploy error
Improve module start waiting, if no async export/refer services, do not wait for started on new thread.

Codecov Report

Merging #9204 (1b2a9d3) into 3.0 (eaf2b3d) will decrease coverage by 0.11%.
The diff coverage is 56.94%.

@@             Coverage Diff              @@
##                3.0    #9204      +/-   ##
============================================
- Coverage     65.11%   65.00%   -0.12%
+ Complexity      314      313       -1
============================================
  Files          1188     1188
  Lines         51399    51434      +35
  Branches       7679     7688       +9
============================================
- Hits          33471    33437      -34
- Misses        14319    14371      +52
- Partials       3609     3626      +17

Impacted Files
Coverage Δ

...ubbo/config/deploy/DefaultApplicationDeployer.java
73.25% <53.48%> (-1.94%)
:arrow_down:

...che/dubbo/config/deploy/DefaultModuleDeployer.java
66.81% <61.53%> (-2.43%)
:arrow_down:

...g/apache/dubbo/common/deploy/AbstractDeployer.java
73.84% <66.66%> (+8.76%)
:arrow_up:

.../protocol/tri/ServerOutboundTransportObserver.java
64.51% <0.00%> (-12.91%)
:arrow_down:

...in/java/org/apache/dubbo/common/utils/JVMUtil.java
79.59% <0.00%> (-12.25%)
:arrow_down:

.../common/threadpool/serial/SerializingExecutor.java
70.37% <0.00%> (-7.41%)
:arrow_down:

...e/dubbo/remoting/transport/netty/NettyChannel.java
55.68% <0.00%> (-4.55%)
:arrow_down:

...pache/dubbo/remoting/transport/AbstractServer.java
57.14% <0.00%> (-4.29%)
:arrow_down:

...e/dubbo/common/url/component/ServiceConfigURL.java
86.53% <0.00%> (-2.83%)
:arrow_down:

.../registry/zookeeper/ZookeeperServiceDiscovery.java
83.52% <0.00%> (-2.36%)
:arrow_down:

... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eaf2b3d...1b2a9d3. Read the comment docs.


text:
[Bug] With no data to scale, the Y-axis will disappear

Version
5.2.2
Link to minimal reproduction
No response
Steps To Reproduce
https://echarts.apache.org/examples/editor.html?c=area-time-axis
Current Behavior

Expected Behavior
When there is no data, scale and leave the Y-axis unchanged
Environment
- OS:
- Browser: Chrome
- Framework:

Any additional comments?
No response

This is the expected behavior unless you set min and max of yAxis.

This is the expected behavior unless you set min and max of yAxis.

sometimes, we don't know the max or min value of yAxis, so I will add filterMode: 'empty' to the data zoom options. For others, if you want to keep xAxis, you can try to add filterMode: 'none' to the data zoom options

authors: CharmingGeeker Ovilia xiaonaiquan

repo: apache/echarts


text:
[Bug] 堆积柱图，设置一个系列的markpoint：{type:max},标注只会显示在堆积后最大值的柱子上，而不是该系列最大值的上面

Version
5.0及以上版本
Link to Minimal Reproduction
https://echarts.apache.org/examples/zh/editor.html?c=line-simple&code=PYBwLglsB2AEC8sDeAoWsBEAPAgliAzhgFzJrqZgCeIApiZgMYCGYtA5sAE5UYA05dBgAmrZgwDagihgBsABgC0Adnn9pQgBxKAnGoEUZqxdozSAuuQC-BzFTyEGqQxmp0GGAG7MANgFd6WxtyDAJaLghaIlIpQ2dDTGhmAFt6UgxALrlAB1N1BMxRMHEYjQp4vKFvfzTYWPLDeVs69ABWEsNzRvKRQhAfZioANV8AyTaEuSVVXKahZrMZjrGhLmAAdwBJaGFaLFGZ9AalinkjxZmMRmAfTe3d0gBGMeC6sq7KkeL92HvOus1T37jYQ9PqDYbVWrnbSKPTTKHzJpnJoYFYbLY7Pb7H5HdCPBaAlyXa7ou7fJ4Esjnd4QnGwABMFMM9zxiMZ-RB_SGVUx52MpjZQmZCLqSLqKLWNwxn32DNpdIBRwuV0lpJZeSsJVFMjc1QwACNmFw4TICIVGABrDzbABmzD8PjAwtgz1KJQwSVSHkAhdE5QEiMQ8hKvPJecGBuqHL7y_GK1Eq8PlSNYhXnInx0jR8ouvLB8bUhN5bFfNXlLUhuMkgsJIvJmOp5WVjPksa5lz56UzWXFlPIiu3KuGLsyntitON-lPTV-nUeA1Gv2m5gWq20W32x1-5KG80ABWAEGgjtIrfyAY75RPLhn6S3uwFiRSusAAOaAck1ADwKTvVY0sX8MGvQlhWEAA
Steps to Reproduce
详细见下面的例子
https://echarts.apache.org/examples/zh/editor.html?c=line-simple&code=PYBwLglsB2AEC8sDeAoWsBEAPAgliAzhgFzJrqZgCeIApiZgMYCGYtA5sAE5UYA05dBgAmrZgwDagihgBsABgC0Adnn9pQgBxKAnGoEUZqxdozSAuuQC-BzFTyEGqQxmp0GGAG7MANgFd6WxtyDAJaLghaIlIpQ2dDTGhmAFt6UgxALrlAB1N1BMxRMHEYjQp4vKFvfzTYWPLDeVs69ABWEsNzRvKRQhAfZioANV8AyTaEuSVVXKahZrMZjrGhLmAAdwBJaGFaLFGZ9AalinkjxZmMRmAfTe3d0gBGMeC6sq7KkeL92HvOus1T37jYQ9PqDYbVWrnbSKPTTKHzJpnJoYFYbLY7Pb7H5HdCPBaAlyXa7ou7fJ4Esjnd4QnGwABMFMM9zxiMZ-RB_SGVUx52MpjZQmZCLqSLqKLWNwxn32DNpdIBRwuV0lpJZeSsJVFMjc1QwACNmFw4TICIVGABrDzbABmzD8PjAwtgz1KJQwSVSHkAhdE5QEiMQ8hKvPJecGBuqHL7y_GK1Eq8PlSNYhXnInx0jR8ouvLB8bUhN5bFfNXlLUhuMkgsJIvJmOp5WVjPksa5lz56UzWXFlPIiu3KuGLsyntitON-lPTV-nUeA1Gv2m5gWq20W32x1-5KG80ABWAEGgjtIrfyAY75RPLhn6S3uwFiRSusAAOaAck1ADwKTvVY0sX8MGvQlhWEAA
Current Behavior
一个堆积柱图，多个系列堆积，设置其中一个系列的markpoint：{type:max},标注只会显示在堆积后最大值的柱子上，即多个系列叠加后的最大值，而不是某个系列最大值的上面，不堆积则正常
Expected Behavior
希望标志放在哪个系列上，就显示在该系列最大值上，跟其他系列无关
Environment
- OS:
- Browser:
- Framework:

Any additional comments?
No response

bug confirmed. Workaround is to calculate data.coord instead of type for markPoint - Demo.

authors: WuEyuan helgasoft


text:
echarts lines effect loop报错(急)

问题简述 (One-line summary)
1、使用未公示的配置loop
                effect : {
                    symbol: 'arrow',
                    symbolSize: 2,
                    show: true,
                    period: 2,
                    loop: false,
                    callback: null,
                    trailLength: 0.1
                }

2、第一次往data里面添加数据是没有问题的，但是清空完数据，再次添加数据的时候，会报错
        symbol.setStyle('shadowColor', color);

这里直接报symbol为undefined 初步猜测是因为使用了loop的原因
因为loop:false后会在动画结束后移除symbol对象
                animator.done(function () {
                    self.remove(symbol);
                });

还发现有另外一个问题，就是 animator.done的回调不是每次都调用，当数据清空后，再次添加数据的时候，第一次会不走回调；
求解决办法

找到部分问题所在；
当lines的data默认有一条数据，然后通过setOption动态添加会导致animator.done异常；
结果只执行一次回调，有一次会被忽略；

@pissang  还有一个2个动画回调执行一次的问题没有解决 仍然存在 报错那个已经OK

loop为false时，在有残影的时候，动画播放完，但残影不会被消除，如何能解决？我也遇到了这个问题，有没有好的办法？

@pissang loop为false时，在有残影的时候，动画播放完，但残影不会被消除，如何能解决？我也遇到了这个问题，有没有好的办法？

我想问问那个effect执行完的回调是需要写到源码中吗?我看echarts官方文档里面seires中的effect好像没有callback


text:
[Enhancement] Field may be 'final'[EventMeshRebalanceService]

Search before asking

[X] I had searched in the issues and found no similar issues.

Enhancement Request

located at:
eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/core/protocol/tcp/client/rebalance/EventMeshRebalanceService.java line 35,37,39
Describe the solution you'd like
add final.
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

@Alonexc I can work on this.

@kuldeepsidhu88 Ok! Welcome to eventmesh community.

repo: apache/eventmesh


text:
[FLINK-2567]

Allow quoted strings in CSV fields to contain quotation character inside of the field, as long as its escaped Ex: 'Hi my name is 'Flink' '. If a quotation character is found, the previous character is checked to see if it is the escape character. If yes, the string continues to be searched for the closing quotation. Test classes were updated to include strings with escaped quotation characters.

This PR should also update documentation of the CSV parser (see "Data Sources" section in programming_guide.md").
Thanks, Fabian

Agree with Fabian. Other then the inline comments and the documentation, this looks good.

Looks good, merging this...

Thanks for your contribution @tammymendt. You pull request has been merged but the auto-close didn't work :) Could you close the pull request?

Is this merged? I cannot find this commit in ASF Git repository.

Pending merge... it is in my travis branch, tests ran over night...

Oh sorry, that was a reference from Stephan's repository.

@tammymendt Sorry for the confusion. Could you please re-open the pull request? I thought it had been merged but it was only merged into Stephan's repository...

Now it's merged. This time for real :)

authors: StephanEwen chiwanpark fhueske mxm tammymendt

repo: apache/flink


text:
[FLINK-16866] Make job submission non-blocking

What is the purpose of the change
This is changing the semantics of the job submission: Instead of completing the Dispatcher.submitJob() future after all the initialization happened (which can potentially involve calling external systems etc.), the .submitJob() call now returns as soon as the job has been accepted by the Dispatcher.
The benefit of this change is that the users will see the root cause of a submission timeout, instead of an akka.ask.timeout.
Brief change log

Introduce a DispatcherJob abstraction that manages the job in a new INITIALIZING state
Change web frontend to cope with initializing jobs
change clients to submit & poll

Verifying this change
This PR introduces various new tests for verification.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes 🔥
The S3 file system connector: no

Documentation
This change is transparent to the user and doesn't need a documentation update.

Thanks a lot for your extensive and detailed review. It was very helpful for me!
I have addressed most of you your comments. I extended the DispatcherJobTest.
I'm in the process of testing if the refactoring to the client has introduced any new test failures, and I might clean up my new tests in DispatcherJob (I would thus recommend you to review them last).
I might push a small change to the client code later today to better distinguish between initialization and runtime failures, so that we can solve the unstable FunctionITCase.testInvalidUseOfTableFunction() test.

Thanks a lot for this extensive review. I believe I have addressed all comments. I'm looking forward to more feedback.

Thanks a lot for your feedback! I addressed all comments and rebased the code to the latest master to get rid of the conflicts.

When closing the corresponding ticket, please add a release note stating the difference in submission semantics. E.g. when submitting a job one cannot immediately trigger job operation (triggering of savepoint, etc.). This is maybe one follow up issue we might wanna think through and adjust if it should become a usability problem.

authors: rmetzger tillrohrmann


text:
[FLINK-25032] Allow to create execution vertices and execution edges lazily

What is the purpose of the change
Allow to create execution vertices and execution edges lazily, for adaptive batch scheduler.
Brief change log
f976e3688916cd9fda355a90ab87ecb87ae80fd9 Allow ExecutionJobVertex to be initialized lazily.
f5d68b86934752cb7f30784cbb2bedf3ce434c3f Allow parallelism of ExecutionJobVertex to be decided lazily.
b86b30a505124aa2404eb1e0d6178eaf600d6368 Enable to create a dynamic graph.
Verifying this change
Add some unit tests in ExecutionJobVertexTest, DefaultExecutionGraphConstructionTest
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure


text:
[FLINK-6494] Migrate ResourceManager/Yarn/Mesos configuration options

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

[ ] General

The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
The pull request addresses only one issue
Each commit in the PR has a meaningful commit message (including the JIRA id)

[ ] Documentation

Documentation has been added for new functionality
Old documentation affected by the pull request has been updated
JavaDoc for public methods has been added

[ ] Tests & Build

Functionality added by the pull request is covered by tests
mvn clean verify has been executed successfully locally or a Travis build has passed

@zentol In addtion to ResourceManager, I have migrated the configuration options of yarn and mesos to their class files, would you please make a review while you are free, thanks :)

Could you split the commits into one for yarn/mesos/RM each, and remove the merge commit?

@zentol Thank you for your suggestion. I have split the commits into one for yarn/mesos/RM each :)

@zentol Thank you for your suggestion, I have fixed the problems you metioned. Thanks

@zentol I updated this PR and fix some code conflicts, could you please have a look when you're free, thanks

I'll try to get this in this week.

merging.

authors: zentol zjureel