text:
Prevent error on GroupAnswers on VR creation

Description
On VR creation it has been noticed that management server receives these answers:
{ Ans: , MgmtId: 7206518937063, via: 11, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.3.212, file: vm_password.json.2b0e06a0-821f-41f1-bf8f-b9805f490a6c","null - success: Error: inet prefix is expected rather than \"table\".\nError: inet address is expected rather than \"table\".\n"],"result":true,"wait":0}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.3.212, file: vm_metadata.json.01da0832-2dd7-451b-a526-1de81d737a80","null - success: Error: inet prefix is expected rather than \"table\".\nError: inet address is expected rather than \"table\".\n"],"result":true,"wait":0}}] }

This has been originated by empty parameters on VR commands: (cloud.log)
2018-11-05 16:50:59,260  CsHelper.py execute:188 Executing: ip route show default via  table Table_eth0 proto static
Error: inet prefix is expected rather than "table"

2018-11-05 16:50:59,269  CsHelper.py execute:188 Executing: ip route add default via  table Table_eth0 proto static
Error: inet address is expected rather than "table"

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?
Deploy VM on Isolated or Shared network - VR created

@blueorangutan package

@PaulAngus a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2417

@blueorangutan test matrix

@PaulAngus a Trillian-Jenkins matrix job (centos6 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

Trillian test result (tid-3156)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 23175 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3018-t3156-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 66 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_scale_vm
Error
13.28
test_scale_vm.py

Trillian test result (tid-3157)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 24455 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3018-t3157-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 65 look OK, 2 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_10_destroy_cpvm
Failure
48.26
test_ssvm.py

test_05_rvpc_multi_tiers
Failure
319.96
test_vpc_redundant.py

test_05_rvpc_multi_tiers
Error
342.71
test_vpc_redundant.py

smoketests generally look ok, but i'm manually checking the VPC failures.

Trillian test result (tid-3168)
Environment: vmware-65 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45361 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3018-t3168-vmware-65.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_root_resize.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 66 look OK, 1 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_00_deploy_vm_root_resize
Failure
502.54
test_deploy_vm_root_resize.py

source: gharchive/pull-request

authors: PaulAngus blueorangutan nvazquez

repo: apache/cloudstack

license_type: permissive

license_source: github-api


text:
Limit selecting of secondary storage with 90% utilization

Description

Upload should gracefully fail when the secondary storage selected has 90% utilization, even if its the only available storage.

Fixes: #3287

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

Create zone with single secondary pool and fill this to above 90%
Try to upload template / ISO / volume and observe this should fail

Simply returning null could possibly result in newer bugs? ( On a quick inspection it seems that the method consumers would not always check for null values ).
I wonder if more testing scenarios than mentioned on the PR should be covered. Thoughts?

Simply returning null could possibly result in newer bugs. ( On a quick inspection it seems that the method consumers would not always check for null values ).
I wonder if more testing scenarios than mentioned on the PR should be covered. Thoughts?

I debated with the thought of editing the method consumers to handle null returns intrinsically, but I noticed in most cases (at least to the best of my knowledge) null returns are eventually handled by level 2 consumers. I'd be curious to try out other related test cases

I can suggest doing some more tests with storage to max capacity -

Creating snapshots (VM and Storage)
Launching VMs
Migrating Volumes

The only place I saw where the null check was not carried was in SecondaryStorageManagerImpl-
    public boolean finalizeVirtualMachineProfile(VirtualMachineProfile profile, DeployDestination dest, ReservationContext context) {
        SecondaryStorageVmVO vm = _secStorageVmDao.findById(profile.getId());
        Map<String, String> details = _vmDetailsDao.listDetailsKeyPairs(vm.getId());
        vm.setDetails(details);

        DataStore secStore = _dataStoreMgr.getImageStore(dest.getDataCenter().getId());
        assert (secStore != null);```

I hope it helps.

@kioie some news here?

@svenvogel let me work on it tonight

Thank you for your PR. But it's incomplete and given we're closer to the freeze, I'll take over and submit issues around secondary storage selection affecting:
https://github.com/apache/cloudstack/issues/3041
https://github.com/apache/cloudstack/issues/3287
https://github.com/apache/cloudstack/issues/3478

No problem

authors: anuragaw kioie rhtyd svenvogel


text:
vm-import: fix unmanaged instance listing

Description
When the host and last host ID is not set for the VM, it may appear in the list of unmanaged instances.
This change fixes the behaviour by filtering unmanaged instances list for the host for the following three criteria:

host is set as host_id for the VM
host is set as the last_host_id for the VM
pod of the host is set as the pod_id for the VM and both host_id and last_host_id is NULL

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

Clusters:
(localcloud) SBCM5> > list clusters filter=id,name,podid,podname
{
  "cluster": [
    {
      "id": "3d79b022-0fd9-4f51-805d-ef71bef562c5",
      "name": "p1-c1",
      "podid": "c9e2eccc-9c94-4a9e-a971-fa15a0bf59c2",
      "podname": "Pod1"
    },
    {
      "id": "a5b2a7b8-f897-4d5f-889f-a9f1321e7c2b",
      "name": "10.0.35.234/Trillian/p1-c2",
      "podid": "c9e2eccc-9c94-4a9e-a971-fa15a0bf59c2",
      "podname": "Pod1"
    }
  ],
  "count": 2
}

Zone has two VMs. One managed VM - t1. Other unmanaged VM - test123. In the below tests, without PR changes ACS wrongly lists both VMs as unmanaged VMs. After changes ACS correctly lists only test123 as the unmanaged VM.
VMs list in the environment after inter-cluster migration of VM - t1:
(localcloud) SBCM5> > list virtualmachines filter=id,name
{
  "count": 1,
  "virtualmachine": [
    {
      "id": "7ce5dab4-ddad-4f25-85f1-8e3992ccb0a0",
      "name": "t1"
    }
  ]
}
(localcloud) SBCM5> > list volumes virtualmachineid=7ce5dab4-ddad-4f25-85f1-8e3992ccb0a0
{
  "count": 1,
  "volume": [
    {
      "account": "admin",
      "chaininfo": "{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[4f37137963f1317e9709563c8f57983e] i-2-5-VM/i-2-5-VM.vmdk\"]}",
      "clusterid": "3d79b022-0fd9-4f51-805d-ef71bef562c5",
      "clustername": "p1-c1",
      "created": "2021-09-06T10:17:07+0000",
      "destroyed": false,
      "deviceid": 0,
      "diskioread": 27,
      "diskiowrite": 9,
      "diskkbsread": 1155,
      "diskkbswrite": 74,
      "displayvolume": true,
      "domain": "ROOT",
      "domainid": "09046ad8-0bda-11ec-a29c-1e0094000118",
      "hypervisor": "VMware",
      "id": "c1340b3e-37dc-4959-af84-e1b5e7395efe",
      "isextractable": true,
      "name": "ROOT-5",
      "path": "i-2-5-VM",
      "podid": "c9e2eccc-9c94-4a9e-a971-fa15a0bf59c2",
      "podname": "Pod1",
      "provisioningtype": "thin",
      "quiescevm": false,
      "serviceofferingdisplaytext": "Small Instance",
      "serviceofferingid": "0820dfc7-d843-4120-9df6-ee4e216083f5",
      "serviceofferingname": "Small Instance",
      "size": 2147483648,
      "state": "Ready",
      "storage": "ps1",
      "storageid": "4f371379-63f1-317e-9709-563c8f57983e",
      "storagetype": "shared",
      "supportsstoragesnapshot": false,
      "tags": [],
      "templatedisplaytext": "CentOS 5.3(64-bit) no GUI (vSphere)",
      "templateid": "0907e0f0-0bda-11ec-a29c-1e0094000118",
      "templatename": "CentOS 5.3(64-bit) no GUI (vSphere)",
      "type": "ROOT",
      "virtualmachineid": "7ce5dab4-ddad-4f25-85f1-8e3992ccb0a0",
      "vmdisplayname": "t1",
      "vmname": "t1",
      "vmstate": "Stopped",
      "zoneid": "1991b455-cebf-4507-88c4-8c8a467971c3",
      "zonename": "pr4774-t1933-vmware-67u3"
    }
  ]
}

Unmanaged instances list before fix:
(localcloud) SBCM5> > list unmanagedinstances clusterid=3d79b022-0fd9-4f51-805d-ef71bef562c5 filter=name,hostid,hostname,clusterid
{
  "count": 2,
  "unmanagedinstance": [
    {
      "clusterid": "3d79b022-0fd9-4f51-805d-ef71bef562c5",
      "hostid": "4b7798ef-745c-4595-9715-be976dfbe963",
      "hostname": "10.0.34.154",
      "name": "test123"
    },
    {
      "clusterid": "3d79b022-0fd9-4f51-805d-ef71bef562c5",
      "hostid": "4b7798ef-745c-4595-9715-be976dfbe963",
      "hostname": "10.0.34.154",
      "name": "i-2-5-VM"
    }
  ]
}

Unmanaged instances list after fix:
(localcloud) SBCM5> > list unmanagedinstances clusterid=3d79b022-0fd9-4f51-805d-ef71bef562c5 filter=name,hostid,hostname,clusterid
{
  "count": 1,
  "unmanagedinstance": [
    {
      "clusterid": "3d79b022-0fd9-4f51-805d-ef71bef562c5",
      "hostid": "4b7798ef-745c-4595-9715-be976dfbe963",
      "hostname": "10.0.34.154",
      "name": "test123"
    }
  ]
}

@blueorangutan package

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan test centos7 vmware-67u3

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

Trillian test result (tid-2304)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 57663 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5400-t2304-vmware-67u3.zip
Smoke tests completed. 88 look OK, 5 have errors
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_redundant_vpc_site2site_vpn
Failure
515.86
test_vpc_vpn.py

test_deploy_vm_start_failure
Error
116.96
test_deploy_vm.py

test_deploy_vm_volume_creation_failure
Error
97.92
test_deploy_vm.py

test_vm_ha
Error
91.45
test_vm_ha.py

test_01_unmanage_vm_cycle
Error
45.18
test_vm_life_cycle.py

test_vm_sync
Error
230.95
test_vm_sync.py

@shwstppr Can you double check your test output? I see 't1' as a vm name in the initial state, and then 'test123' and 'i-2-5-VM' in the final report. I'm thinking 't1' and 'test123' are meant to be the same, but please confirm (is this output from two different test runs, or am I missing the point?)

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@shwstppr can you fix the conflicts

@DaanHoogland Zone had two VMs. One managed VM - t1. Other unmanaged VM - test123. In the tests, without PR changes ACS wrongly lists both VMs as unmanaged VMs. After changes ACS correctly lists only test123 as the unmanaged VM.
@blueorangutan package

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2364

@blueorangutan test centos7 vmware-67u3

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

Trillian test result (tid-3049)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38426 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5400-t3049-vmware-67u3.zip
Smoke tests completed. 91 look OK, 1 have errors
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_unmanage_vm_cycle
Error
41.11
test_vm_life_cycle.py

Moving to draft while I retest and fix the issue

@blueorangutan package

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2393

@DaanHoogland can you please give it a try now

@blueorangutan test centos7 vmware-67u3 keepEnv

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

@blueorangutan package

@blueorangutan package

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2404

@blueorangutan test centos7 vmware-67u3 keepEnv

Trillian Build Failed (tid-3076)

@blueorangutan test centos7 vmware-67u3 keepEnv

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

@blueorangutan test centos7 vmware-67u3 keepEnv

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

Trillian test result (tid-3101)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32853 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5400-t3101-vmware-67u3.zip
Smoke tests completed. 83 look OK, 9 have errors
Only failed tests results shown below:

Test
Result
Time (s)
Test File

ContextSuite context=TestAddVmToSubDomain>:setup
Error
3.90
test_accounts.py

test_DeleteDomain
Error
8.57
test_accounts.py

test_forceDeleteDomain
Failure
7.62
test_accounts.py

test_01_user_remove_VM_running
Error
2.08
test_accounts.py

test_09_project_suspend
Error
2.39
test_projects.py

test_10_project_activation
Error
2.28
test_projects.py

test_DeployVmAffinityGroup
Error
1.20
test_affinity_groups.py

test_DeployVmAntiAffinityGroup
Error
1.15
test_affinity_groups.py

test_nic_secondaryip_add_remove
Error
7.02
test_multipleips_per_nic.py

test_DeployVmAntiAffinityGroup_in_project
Error
1.31
test_affinity_groups_projects.py

ContextSuite context=TestAnnotations>:setup
Error
0.00
test_annotations.py

test_01_import_storage_policies
Error
4.56
test_storage_policy.py

ContextSuite context=TestVMWareStoragePolicies>:teardown
Error
4.59
test_storage_policy.py

test_query_async_job_result
Error
4.74
test_async_job.py

test_nested_virtualization_vmware
Error
4.54
test_nested_virtualization.py

good work @shwstppr , I now see both lists populated. I'll restart a set of smoke tests.
@blueorangutan test centos7 vmware-67u3

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

Trillian test result (tid-3119)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47882 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5400-t3119-vmware-67u3.zip
Smoke tests completed. 91 look OK, 1 have errors
Only failed tests results shown below:

Test
Result
Time (s)
Test File

test_01_invalid_upgrade_kubernetes_cluster
Failure
3612.37
test_kubernetes_clusters.py

test_02_upgrade_kubernetes_cluster
Failure
3604.26
test_kubernetes_clusters.py

test_03_deploy_and_scale_kubernetes_cluster
Failure
0.04
test_kubernetes_clusters.py

test_04_autoscale_kubernetes_cluster
Failure
0.03
test_kubernetes_clusters.py

test_05_basic_lifecycle_kubernetes_cluster
Failure
0.04
test_kubernetes_clusters.py

test_06_delete_kubernetes_cluster
Failure
0.04
test_kubernetes_clusters.py

test_07_deploy_kubernetes_ha_cluster
Failure
0.05
test_kubernetes_clusters.py

test_08_upgrade_kubernetes_ha_cluster
Failure
0.04
test_kubernetes_clusters.py

test_09_delete_kubernetes_ha_cluster
Failure
0.04
test_kubernetes_clusters.py

ContextSuite context=TestKubernetesCluster>:teardown
Error
40.91
test_kubernetes_clusters.py

Trillian test result (tid-3119) Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7 Total time taken: 47882 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5400-t3119-vmware-67u3.zip Smoke tests completed. 91 look OK, 1 have errors Only failed tests results shown below:
Test	Result	Time (s)	Test File
test_01_invalid_upgrade_kubernetes_cluster	Failure	3612.37	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	Failure	3604.26	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	Failure	0.04	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	Failure	0.03	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	Failure	0.04	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	Failure	0.04	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	Failure	0.05	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	Failure	0.04	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	Failure	0.04	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	Error	40.91	test_kubernetes_clusters.py

these failures ^^^ are unrelated to the changes in this PR.

authors: DaanHoogland blueorangutan rhtyd shwstppr sureshanaparti


text:
eunit couchdb_views_tests errors with 500: {'EXIT',noproc}

Current & Expected Behaviour
The couchdb_views_tests eunit test suite should always pass. Occasionally, it fails with a 500 error.
Environment: Jenkins, master, Ubuntu 16.04 with Erlang 18.3
Makefile output:
module 'couchdb_views_tests'
  View indexes cleanup
    couchdb_views_tests:214: should_have_two_indexes_alive_before_deletion...ok
    couchdb_views_tests:219: should_cleanup_index_file_after_ddoc_deletion...ok
    couchdb_views_tests:225: should_cleanup_all_index_files...ok
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
    [done in 0.242 s]
  View group db leaks
    couchdb_views_tests:228: couchdb_1138...[0.237 s] ok
    couchdb_views_tests:266: couchdb_1309...[0.037 s] ok
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
    [done in 0.339 s]
  View group shutdown
    couchdb_views_tests:315: couchdb_1283...[0.158 s] ok
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
    [done in 0.161 s]
  Upgrade and bugs related tests
    couchdb_views_tests:159: should_not_remember_docs_in_index_after_backup_restore...*failed*
in function couchdb_views_tests:'-query_view/4-fun-0-'/2 (test/couchdb_views_tests.erl, line 527)
in call from couchdb_views_tests:query_view/4 (test/couchdb_views_tests.erl, line 527)
in call from couchdb_views_tests:'-should_not_remember_docs_in_index_after_backup_restore/1-fun-8-'/1 (test/couchdb_views_tests.erl, line 173)
**error:{assertEqual,[{module,couchdb_views_tests},
              {line,527},
              {expression,"Code"},
              {expected,200},
              {value,500}]}
  output:<<"">>

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
    [done in 0.071 s]

couch.log output:
[info] 2017-06-08T18:34:25.819709Z nonode@nohost <0.24082.0> -------- Opening index for db: eunit-test-db-1496946865766965 idx: _design/foo sig: "5398c9ad12cad56d2228cc89a2770ddf"
[info] 2017-06-08T18:34:25.820150Z nonode@nohost <0.24085.0> -------- Starting index update for db: eunit-test-db-1496946865766965 idx: _design/foo
[info] 2017-06-08T18:34:25.825651Z nonode@nohost <0.24085.0> -------- Index update finished for db: eunit-test-db-1496946865766965 idx: _design/foo
[notice] 2017-06-08T18:34:25.826205Z nonode@nohost <0.24032.0> -------- 127.0.0.1 - - GET /eunit-test-db-1496946865766965/_design/foo/_view/bar 200
[info] 2017-06-08T18:34:25.826998Z nonode@nohost <0.24018.0> -------- db eunit-test-db-1496946865766965 died with reason shutdown
[info] 2017-06-08T18:34:25.827066Z nonode@nohost <0.24082.0> -------- Index shutdown by monitor notice for db: eunit-test-db-1496946865766965 idx: _design/foo
[error] 2017-06-08T18:34:25.831129Z nonode@nohost <0.24033.0> -------- Uncaught error in HTTP request: {error,{badmatch,{'EXIT',noproc}}}
[info] 2017-06-08T18:34:25.831593Z nonode@nohost <0.24082.0> -------- Closing index for db: eunit-test-db-1496946865766965 idx: _design/foo sig: "5398c9ad12cad56d2228cc89a2770ddf" because normal
[info] 2017-06-08T18:34:25.831969Z nonode@nohost <0.24033.0> -------- Stacktrace: [{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,lookup,3,[{file,"src/couch_btree.erl"},{line,284}]},{couch_btree,lookup,2,[{file,"src/couch_btree.erl"},{line,274}]},{couch_db,get_full_doc_info,2,[{file,"src/couch_db.erl"},{line,286}]},{couch_db,open_doc_int,3,[{file,"src/couch_db.erl"},{line,1366}]},{couch_db,open_doc,3,[{file,"src/couch_db.erl"},{line,189}]}]
[notice] 2017-06-08T18:34:25.832259Z nonode@nohost <0.24033.0> -------- 127.0.0.1 - - GET /eunit-test-db-1496946865766965/_design/foo/_view/bar 500
[error] 2017-06-08T18:34:25.832512Z nonode@nohost <0.24033.0> -------- httpd 500 error response:
 {"error":"badmatch","reason":"{'EXIT',noproc}"}
[info] 2017-06-08T18:34:25.842468Z nonode@nohost <0.7.0> -------- Application couch exited with reason: stopped

Possible Solution
Is this another case where the monitor shuts down too early?

Have we seen more of these after this commit: https://github.com/apache/couchdb/commit/05d2e1e6a7db3e2526530cdbb446034557b94e57 ?

@nickva this issue opened 23 hours ago, the commit done 4 days ago, so I guess the answer is yes :)
On a different topic - @wohali this issue seems to be duplicate of apache/couchdb#565, should we close one of them?

Sorry, my mistake. This is a duplicate of #565 so I'll close this one.

source: gharchive/issue

authors: eiri nickva wohali

repo: apache/couchdb


text:
Implement comparisons on nested data types such that distinct/except would work

Which issue does this PR close?
Closes #10749

Rationale for this change
This relies on newer functionality in arrow 52 and allows DataFrame.except() to properly work on schemas with structs and lists. I'm not sure if this is the appropriate way to handle this change per se, but I included the regression case from the issue as a test in order to demonstrate the correction of the issue

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

I am quite indifferent to the solution here as long as #10749 is resolved :smile:
Happy to have this closed out in favor of a better implementation!

I am quite indifferent to the solution here as long as #10749 is resolved 😄
Happy to have this closed out in favor of a better implementation!

This PR is great and I think a step forward (the code no longer errors!)
I'll make a follow on PR to try and simplify the implementation.

Thanks again @rtyler  and @jayzhan211

Filed https://github.com/apache/datafusion/pull/11149 with a proposed simpler implementation

authors: alamb rtyler

repo: apache/datafusion


text:
Reorganization proposal

Hi there,
We are consider using datasketches in our C++ environment. The current organization, where there a bunch of folders, each with an include/ directory, is a bit awkward for integrating into a foreign build system. Projects like pybind11 put all of the headers in a single include/ directory (perhaps with subdirectories underneath). Then there is just a single path to add as an include path.
Would you be open to a reorganization that moved around files to look a bit like this:
include/datasketches
  count/
    count_min.hpp
    count_min_impl.hpp
  common/
    MurmurHash3.h
    ...

Thanks for your consideration.

Interesting. I believe we install everything into a single datasketches directory, so we'd assumed the library would typically be installed. You're probable not doing that?
Certainly not trying to tell you how to develop your project, just trying to understand what people do in practice. We don't get much feedback about that.
I don't think I have an inherent objection to this. Will try to look into it in the near term.

Happy to send a PR for this as well.
Yeah, given that the project is (currently) header-only, there's not a huge benefit for us to install it instead of vendoring the files (which is an approach we've taken with other libraries that we plan to integrate with tightly). Currently, I've just flattened all the files into a single directory (basically what the build/install does).

If I try to add any new matrix sketches I think I'd try using Eigen (we have Frequent Directions in java for approximate SVD, for instance, but there may be other approaches that make that less interesting to port this many years later) which is also header-only. That's a combination of header-only considerations as well as playing nicely with our python wrapper (nanobind, which we adopted recently instead of pybind11).
@AlexanderSaydakov noted on Slack that this would be a major version change, though, and we did just do one of those. We do try not to do major version bumps toooooo frequently. So that'd probably be the main consideration.

@AlexanderSaydakov Several months later, I don't know if we want to revisit this yet?

This seems reasonable. I can see how fragmented "include" directory structure can complicate someone's life. However, if this is not urgent, I would just keep it in mind for the next major version.

@tjstum I know it's been a while but we haven't forgotten this. We discovered a change we merged recently does break the API slightly in some edge cases so we'll likely pick this up for the next release.

authors: AlexanderSaydakov jmalkin tjstum

repo: apache/datasketches-cpp


text:
[Bug] [java_task] Java type task cannot read variables

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
When I want to use variables in the java node, the configuration does not take effect. Checking the running log did not replace ${var} with a specific value.
The shell1 node passes the set_val variable value, and shell2 and java1 use the set_var variable; during actual operation, shell2 can read the set_var variable correctly, but java1 cannot.

What you expected to happen
I want java1's mainArgs to be replaced with the real value of set_var
How to reproduce
It can be reproduced as described above
Anything else
I found by looking at the source code that shellTask set properties when constructing IShellInterceptorBuilder, but JavaTask did not. Finally, when producing runShell, the propertyMap is empty.
ShellTask：

JavaTask：

shellBody：

Version
3.2.x
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

JavaTask  line 131 add .properties(ParameterUtils.convert(taskRequest.getPrepareParamsMap()))

JavaTask line 131 add .properties(ParameterUtils.convert(taskRequest.getPrepareParamsMap()))

Can i get the inner 、global、project params by java code ?

authors: 1032851561 YetAlpha jxdhuochai

repo: apache/dolphinscheduler


text:
[Feature][doc] Add how to use UDF in SQL task

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Documentation addition, should including

the code UDF definition
how dolphinscheduler work with UDF, and what command will be execute it
how to use in SQL script

Use case
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Please assign it to me : )

This task has already been done. UDF Manage

Thanks and i will close it.

close by #9658

authors: QuakeWang zhongjiajie


text:
[Improvement-11380][scp-host.sh] Set StrictHostKeyChecking=no option to ssh

Purpose of the pull request
This PR close: #11380
Even if the password-free account is configured, when the user uses the scp-host.sh script to log in to the machine for the first time, an error (Host key verification failed.) may occur.
So set the  -o StrictHostKeyChecking=no option to ssh in scp-hosts.sh.
Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

Hi @rickchengx , to target specific issue from PR, we recommend use keyword fix: #issue_id or close: #issue_id or closes: #issue_id in your PR describe(not title, just desc). It would not only connect issue to PR but also close issue automatically when PR is be closed.

Hi @rickchengx , to target specific issue from PR, we recommend use keyword fix: #issue_id or close: #issue_id or closes: #issue_id in your PR describe(not title, just desc). It would not only connect issue to PR but also close issue automatically when PR is be closed.

Sure. Thanks for your advice, I've added the keyword close: #issue_id.

authors: SbloodyS rickchengx


text:
[Fix-11434][Doc] Update the registry related configuration in values.yaml

Purpose of the pull request
close: #11434
Update the registry related configuration in values.yaml.
Brief change log

Parameter
Description
Default

zookeeper.service.port
The port of zookeeper
2181

externalRegistry.registryPluginDir
If exists external registry and set zookeeper.enable to false, specify the external registry plugin directory
lib/plugin/registry

externalRegistry.registryPluginName
If exists external registry and set zookeeper.enable to false, specify the external registry plugin name
zookeeper

externalRegistry.registryServers
If exists external registry and set zookeeper.enable to false, specify the external registry servers
127.0.0.1:2181

Verify this pull request

This pull request is code cleanup without any test coverage.

Hi @rickchengx please check the fail CI result.

Hi @rickchengx please check the fail CI result.

@rickchengx You may rebase your code on the latest dev branch and see whether these three unused images still there. If so, just remove them and CI will pass.

Hi @rickchengx please check the fail CI result.

Hi @rickchengx please check the fail CI result.

@rickchengx You may rebase your code on the latest dev branch and see whether these three unused images still there. If so, just remove them and CI will pass.

Hi, @caishunfeng @EricGao888 , thanks for your suggestions! I've rebased my PR and CI is passed.

Thanks

In this patch, we only want to remove two dangling configurations, but have to change other content to pass the spotless test. it will increase contributor and review time to handle the nosy, I think we should better bump all docs according to the spotless' rule, WDTY @caishunfeng @EricGao888

In this patch, we only want to remove two dangling configurations, but have to change other content to pass the spotless test. it will increase contributor and review time to handle the nosy, I think we should better bump all docs according to the spotless' rule, WDTY @caishunfeng @EricGao888

+1

In this patch, we only want to remove two dangling configurations, but have to change other content to pass the spotless test. it will increase contributor and review time to handle the nosy, I think we should better bump all docs according to the spotless' rule, WDTY @caishunfeng @EricGao888

+1, I will handle it and format all the docs once for all. Thanks for the suggestions.

authors: EricGao888 caishunfeng rickchengx zhongjiajie


text:
[Fix-13438] fix breadcrumb navigation and directory visit bugs in reso…

Purpose of the pull request

This PR closes: https://github.com/apache/dolphinscheduler/issues/13438

Brief change log

Add new api /resource/base-dir for querying base resource dir which add constraint that breadcrumb navigation can only visit resources or files under resource paths.

Refactor some breadcrumb navigation related codes in front-end.

Verify this pull request

Add some UT for back-end parts.
Manually test front-end parts.

FINALLY, Happy Lunar New Year 🐇 you all!

Codecov Report

Merging #13439 (953cf5f) into dev (3a6e103) will increase coverage by 0.01%.
The diff coverage is 37.93%.

@@             Coverage Diff              @@
##                dev   #13439      +/-   ##
============================================
+ Coverage     39.56%   39.58%   +0.01%
- Complexity     4327     4335       +8
============================================
  Files          1084     1086       +2
  Lines         40916    40957      +41
  Branches       4697     4685      -12
============================================
+ Hits          16187    16211      +24
- Misses        22927    22936       +9
- Partials       1802     1810       +8

Impacted Files
Coverage Δ

...nscheduler/api/controller/ResourcesController.java
53.70% <0.00%> (-1.02%)
:arrow_down:

...duler/plugin/storage/hdfs/HdfsStorageOperator.java
12.37% <0.00%> (ø)

...heduler/plugin/storage/oss/OssStorageOperator.java
32.53% <0.00%> (-0.27%)
:arrow_down:

...heduler/api/service/impl/ResourcesServiceImpl.java
38.52% <42.30%> (+0.13%)
:arrow_up:

...e/dolphinscheduler/dao/upgrade/WorkerGroupDao.java
41.17% <0.00%> (-14.38%)
:arrow_down:

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...erver/master/processor/queue/TaskEventService.java
75.00% <0.00%> (-5.36%)
:arrow_down:

...dolphinscheduler/remote/future/ResponseFuture.java
81.96% <0.00%> (-1.64%)
:arrow_down:

...hinscheduler/plugin/alert/script/ScriptSender.java
70.58% <0.00%> (-1.64%)
:arrow_down:

...hinscheduler/dao/upgrade/ProcessDefinitionDao.java
7.79% <0.00%> (-1.07%)
:arrow_down:

... and 34 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

PTAL @Amy0104 @songjianet, this PR mainly fixes front-end bug, thx!

@SbloodyS It seems that front-end codes have been modified in https://github.com/apache/dolphinscheduler/pull/11203 😓,  also solve breadcrumb navigation bug, however, we can keep the other change in this pr

Add new api /resource/base-dir for querying base resource dir which add constraint that breadcrumb navigation can only visit resources or files under resource paths.

I will close this one and create another pr later today.

authors: Radeity codecov-commenter


text:
[Improvement-5452][Task] ds flink task support submit a PyFlink job via the CLI

Purpose of the pull request
this close #5452
ds flink task support submit a PyFlink job via the CLI
$ ./bin/flink run --python examples/python/table/batch/word_count.py

Codecov Report

Merging #5453 (14ae297) into dev (d93a9dc) will increase coverage by 3.24%.
The diff coverage is 44.80%.

@@             Coverage Diff              @@
##                dev    #5453      +/-   ##
============================================
+ Coverage     45.21%   48.46%   +3.24%
- Complexity     3395     3855     +460
============================================
  Files           538      598      +60
  Lines         23053    24298    +1245
  Branches       2687     2784      +97
============================================
+ Hits          10423    11775    +1352
+ Misses        11698    11446     -252
- Partials        932     1077     +145

Impacted Files
Coverage Δ
Complexity Δ

...inscheduler/plugin/alert/email/EmailConstants.java
87.50% <ø> (ø)
1.00 <0.00> (ø)

...eduler/plugin/alert/script/ScriptAlertChannel.java
16.66% <0.00%> (ø)
1.00 <0.00> (ø)

...org/apache/dolphinscheduler/alert/AlertServer.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...che/dolphinscheduler/api/ApiApplicationServer.java
33.33% <0.00%> (ø)
1.00 <0.00> (ø)

...cheduler/api/controller/AccessTokenController.java
6.66% <ø> (-1.67%)
1.00 <0.00> (-1.00)

...scheduler/api/controller/AlertGroupController.java
8.00% <ø> (+2.73%)
2.00 <0.00> (ø)

.../api/controller/AlertPluginInstanceController.java
7.69% <ø> (+1.97%)
2.00 <0.00> (ø)

...heduler/api/controller/DataAnalysisController.java
9.09% <ø> (ø)
1.00 <0.00> (-1.00)

...scheduler/api/controller/DataSourceController.java
4.16% <0.00%> (-0.49%)
1.00 <0.00> (-1.00)

...inscheduler/api/controller/ExecutorController.java
8.33% <ø> (-2.78%)
1.00 <0.00> (-1.00)

... and 298 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5556e47...14ae297. Read the comment docs.

@zhuangchong Hi, how is it getting?

@zhuangchong Hi, how is it getting?

done.


text:
feat(.gitignore): added *.DS_Store

Proposed changes
Issue Number: #57
Problem Summary:
.gitignore added *.DS_Store
Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (No Need)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@JNSimba Hi Simba, please review this pr~

repo: apache/doris-flink-connector


text:
enhancement add profile to show column predicates

Proposed changes
add profile to show column predicates

Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.59 seconds
load time: 569 seconds
storage size: 17154644823 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221101112101_clickbench_pr_37280.html

authors: BiteTheDDDDt hello-stephen

repo: apache/doris


text:
fix fix some bugs in Subquery to window rule

Proposed changes
Issue Number: close #xxx
Problem summary

fix window function generation method, replace inner slot with equivalent outer slot
forbid below scenes
a. inner has a mapping project
b. inner has an unexpected filter
c. outer has a mapping project
d. outer has an unexpected filter
e. outer has additional table
f. outer has same table
g. outer and inner with different join condition
h. outer and inner has same table with different join condition

Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.33 seconds
stream load tsv:          448 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json:         24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc:          72 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230330023415_clickbench_pr_122590.html

run buildall

authors: hello-stephen morrySnow


text:
Bug fix failed on test_dup_tab_decimalv3 due to wrong precision

Proposed changes
fix failed on test_dup_tab_decimalv3 due to wrong precision
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.59 seconds
stream load tsv:          508 seconds loaded 74807831229 Bytes, about 140 MB/s
stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17165544555 Bytes

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 53.23 seconds
stream load tsv:          510 seconds loaded 74807831229 Bytes, about 139 MB/s
stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17169704157 Bytes

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 54.93 seconds
stream load tsv:          547 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17163996177 Bytes


text:
[improvement](hash join) Return result early if probe side has no data

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.08 seconds
stream load tsv:          542 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select:          98.7 seconds inserted 10000000 Rows, about 101K ops/s
storage size: 17162312998 Bytes

authors: hello-stephen jacktengg


text:
fix should not put bound expr into unbound group by list (#25938)

cherry-pick from master
PR: #25938
commit id: 2229d82acd895ab968f707b851f47c67d9a820ae
we put bound expr into unbound group by list by mistake. This will lead to bind twice on some exprssion.
Since binding is not idempotent, below exception will be thrown for sql
select k5 / k5 as nu, sum(k1) from test group by nu order by nu nulls first

Caused by: org.apache.doris.nereids.exceptions.AnalysisException: Input slot(s) not in child's output: k5#5 in plan: LogicalProject[176] ( distinct=false, projects=[(cast(k5#5 as DECIMALV3(16, 10)) / k5#5) AS `nu`#14, sum(k1)#15], excepts=[] ), child output is: [nu#16, sum(k1)#15]
plan tree:
LogicalProject[176] ( distinct=false, projects=[(cast(k5#5 as DECIMALV3(16, 10)) / k5#5) AS `nu`#14, sum(k1)#15], excepts=[] )
+--LogicalAggregate[168] ( groupByExpr=[nu#16], outputExpr=[nu#16, sum(k1#1) AS `sum(k1)`#15], hasRepeat=false )
   +--LogicalProject[156] ( distinct=false, projects=[k1#1, (cast(k5#5 as DECIMALV3(16, 10)) / k5#5) AS `nu`#16], excepts=[] )
      +--LogicalOlapScan ( qualified=default_cluster:regression_test_nereids_syntax_p0.test, indexName=test, selectedIndexId=503229, preAgg=OFF, Aggregate function sum(k1) contains key column k1. )
    at org.apache.doris.nereids.rules.analysis.CheckAfterRewrite.checkAllSlotReferenceFromChildren(CheckAfterRewrite.java:108) ~[classes/:?]

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
Explain add mv rewrite info

Proposed changes
in query rewrite by mv, we want know the mv rewrite process info
such as materializedView used by rewrite, which materializedView rewrite success, and
chosen which materializedView by cost finally.
we can run sql as following to see the mv rewrite process summary info
explain <query _sql>
materializedView rewrite info is under the MATERIALIZATIONS,
example as following:
we can see the mv2_3 materializedView is rewritten successfuly and chosen finally.
and mv2_4 and mv1_3 is failed,

========== MATERIALIZATION'S ==========

========== AVAILABLE MATERIALIZATION'S ==========
[
mv2_3
mv2_4
mv1_3
]
========== REWRITTEN SUCCESS MATERIALIZATION'S ==========
[
mv2_3
]
========== CHOSEN MATERIALIZATION'S ==========
[
mv2_3
]

if want to see detail info,we can also run sql as following to see the mv rewrite process info
explain memo plan as <query _sql>
materializedView rewrite info is under the MATERIALIZATIONS,
example as following:
we can see the mv2_3 materializedView is rewritten successfuly and chosen finally.
and mv2_4 and mv1_3 is failed with fali reason.

========== MATERIALIZATIONS ==========
materializationContexts:

MaterializationContext[mv1_3] ( rewriteSuccess=false, failReason=[
ObjectId : ObjectId#257.
Summary : Match mode is invalid.
Reason : matchMode is VIEW_PARTIAL.

ObjectId : ObjectId#260.
Summary : Match mode is invalid.
Reason : matchMode is VIEW_PARTIAL.

ObjectId : ObjectId#251.
Summary : Match mode is invalid.
Reason : matchMode is VIEW_PARTIAL.

ObjectId : ObjectId#254.
Summary : Match mode is invalid.
Reason : matchMode is VIEW_PARTIAL.

] )

MaterializationContext[mv2_4] ( rewriteSuccess=false, failReason=[
ObjectId : ObjectId#257.
Summary : The graph logic between query and view is not consistent.
Reason : graph logical is not equal, query join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
query filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]],
view join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
view filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]]
inferred edge with conds {}
with error edge <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]=[(o_orderdate#20 = 2023-12-01)].

ObjectId : ObjectId#260.
Summary : The graph logic between query and view is not consistent.
Reason : graph logical is not equal, query join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
query filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]],
view join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
view filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]]
inferred edge with conds {}
with error edge <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]=[(o_orderdate#20 = 2023-12-01)].

ObjectId : ObjectId#251.
Summary : The graph logic between query and view is not consistent.
Reason : graph logical is not equal, query join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
query filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]],
view join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
view filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]]
inferred edge with conds {}
with error edge <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]=[(o_orderdate#20 = 2023-12-01)].

ObjectId : ObjectId#254.
Summary : The graph logic between query and view is not consistent.
Reason : graph logical is not equal, query join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
query filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]],
view join edges is [<{0} --LEFT_OUTER_JOIN-- {1}>],
view filter edges is [<{0} --FILTER-- {}>, <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]]
inferred edge with conds {}
with error edge <{1} --FILTER-- {}>[[] , [<{0} --LEFT_OUTER_JOIN-- {1}>]]=[(o_orderdate#20 = 2023-12-01)].

] )

MaterializationContext[mv2_3] ( rewriteSuccess=true, failReason=[

] )

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall


text:
[fix](jdbc catalog) Fix query errors without jdbc pool default value on only BE upgrade

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
[fix](auto bucket) Fix hit not support alter estimate_partition_size

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

@deardeng I think it should be allowed to modified.

authors: dataroaring deardeng xiaokang


text:
refactor Clear unnecessary status in pipeline fragment context

…
Proposed changes
Issue Number: close #xxx

run buildall


text:
branch-2.1 Support read sync materialized view in async materialized view

Proposed changes
pick #37396

run buildall

run p0

run p0

run buildall

run p0

run feut

run beut

run buildall


text:
fix Fix cloud auto start and add a regression case

Fix the cluster being suspended and select not waking up the cluster. The reason is that all be nodes in the cluster are inactive, the cluster is skipped, and the cluster that needs to be woken up cannot be found, and the wake-up logic will not be reached. And delete the redundant function getAuthorizedCloudCluster
Add check after resume cluster, there must be at least one alive be in the cluster.
add auto start regression case

run buildall

run buildall


text:
DRILL-7337: Add vararg UDFs support

For details please see DRILL-7337.

This is a really useful feature!

Also could you please show how varargs functions will be described in sys.functions table. Do we need new flag to filter them out?
Example:
apache drill (sys)> select * from sys.functions limit 2;
+------+---------------------------------+------------+----------+----------+
| name |            signature            | returnType |  source  | internal |
+------+---------------------------------+------------+----------+----------+
| !=   | BIGINT-REQUIRED,BIGINT-REQUIRED | BIT        | built-in | false    |
| !=   | BIT-REQUIRED,BIT-REQUIRED       | BIT        | built-in | false    |
+------+---------------------------------+------------+----------+----------+

+1, please squash the commits.

authors: arina-ielchiieva cgivre vvysotskyi

repo: apache/drill


text:
Direct buffer memory error if a filter used twice.

We were facing below exception on some queries sent from superset. I did some investigation and realized it occurs if same key used in filter more than once. For example
SELECT TIME_FLOOR(CAST("__time" AS TIMESTAMP), 'P1D') AS "__timestamp",
       "id" AS "id",
       sum("count") AS "SUM(count)"
FROM "druid"."example_table"
WHERE "__time" >= '2022-05-31 12:47:38.000000'
  AND "__time" <= '2022-06-07 12:47:38.000000'
  AND "key" IN ('value1')
  AND "key" IN ('value2')
GROUP BY "id",
         TIME_FLOOR(CAST("__time" AS TIMESTAMP), 'P1D')
ORDER BY "SUM(count)" DESC
LIMIT 10000

Above query throws exception. If I remove one of key filter, it solves problem. If I add value1 to second filter like "key" in ('value1, 'value2') without removing first one, it also solves problem. Increasing direct memory with java opts also didn't solve issue. I believe this is a bug
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2.waitForFutureCompletion(GroupByMergingQueryRunnerV2.java:383) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2.access$800(GroupByMergingQueryRunnerV2.java:83) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:274) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:164) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:62) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.TopNSequence$1.make(TopNSequence.java:54) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:66) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.sql.http.SqlResource.doPost(SqlResource.java:118) ~[druid-sql-0.22.1.jar:0.22.1]
        at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_275]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_275]
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.3.jar:1.19.3]
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.3.jar:1.19.3]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.3.jar:1.19.3]
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.3.jar:1.19.3]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) ~[guice-servlet-4.1.0.jar:?]
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82) ~[druid-server-0.22.1.jar:0.22.1]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-0.22.1.jar:0.22.1]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75) ~[druid-server-0.22.1.jar:0.22.1]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:208) ~[?:?]
        at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-0.22.1.jar:0.22.1]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-0.22.1.jar:0.22.1]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) [jetty-util-9.4.40.v20210413.jar:9.4.40.v20210413]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Direct buffer memory
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272) ~[guava-16.0.1.jar:?]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96) ~[guava-16.0.1.jar:?]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2.waitForFutureCompletion(GroupByMergingQueryRunnerV2.java:358) ~[druid-processing-0.22.1.jar:0.22.1]
        ... 103 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:695) ~[?:1.8.0_275]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_275]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_275]
        at org.apache.druid.offheap.OffheapBufferGenerator.get(OffheapBufferGenerator.java:52) ~[druid-server-0.22.1.jar:0.22.1]
        at org.apache.druid.offheap.OffheapBufferGenerator.get(OffheapBufferGenerator.java:28) ~[druid-server-0.22.1.jar:0.22.1]
        at org.apache.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:118) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.collections.StupidPool.take(StupidPool.java:109) ~[druid-core-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByQueryEngineV2.process(GroupByQueryEngineV2.java:131) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.strategy.GroupByStrategyV2.process(GroupByStrategyV2.java:696) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.GroupByQueryRunnerFactory$GroupByQueryRunner.run(GroupByQueryRunnerFactory.java:105) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:244) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:232) ~[druid-processing-0.22.1.jar:0.22.1]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_275]
        at org.apache.druid.java.util.common.concurrent.DirectExecutorService.execute(DirectExecutorService.java:81) ~[druid-core-0.22.1.jar:0.22.1]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:1.8.0_275]
        at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58) ~[guava-16.0.1.jar:?]
        at org.apache.druid.query.DirectQueryProcessingPool.submitRunnerTask(DirectQueryProcessingPool.java:42) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:230) ~[druid-processing-0.22.1.jar:0.22.1]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1.apply(GroupByMergingQueryRunnerV2.java:222) ~[druid-processing-0.22.1.jar:0.22.1]
        at com.google.common.collect.Iterators$8.transform(Iterators.java:794) ~[guava-16.0.1.jar:?]
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-16.0.1.jar:?]
        at com.google.common.collect.Iterators.addAll(Iterators.java:357) ~[guava-16.0.1.jar:?]
        at com.google.common.collect.Lists.newArrayList(Lists.java:147) ~[guava-16.0.1.jar:?]
        at com.google.common.collect.Lists.newArrayList(Lists.java:129) ~[guava-16.0.1.jar:?]
        at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1.make(GroupByMergingQueryRunnerV2.java:218) ~[druid-processing-0.22.1.jar:0.22.1]
        ... 101 more

Broker configuration
druid.server.http.numThreads=60
druid.broker.http.numConnections=50
druid.broker.http.maxQueuedBytes=10MiB
druid.processing.buffer.sizeBytes=500MiB
druid.processing.numMergeBuffers=5
druid.processing.numThreads=1
druid.processing.tmpDir=var/druid/processing
druid.broker.cache.useCache=false
druid.broker.cache.populateCache=false

Historical configuration
druid.server.http.numThreads=60
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=4
druid.processing.numThreads=16
druid.processing.tmpDir=var/druid/processing
druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":3000000000000}]
druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true

Affected Version
0.22.1 deployed via docker image. with 1 broker 3 historical node.

maybe related to https://github.com/apache/druid/issues/12620

Issue seems resolved on 0.23.0

authors: iercan renatocron

repo: apache/druid