text:
[BEAM-13830] Properly shut down Debezium expansion service in IT script.

Quick fix to this issue.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @riteshghorse

LGTM. I missed this while adding a flag.

No worries, I missed it too when I added a flag earlier.

authors: riteshghorse youngoli


text:
Force version of sklearn in examples

Force sklearn version to the same version the models were trained on. This ensures that the examples will pass correctly.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Python 3.9 PostCommit

Please fix lint & yapf and run an integration test suite that runs the example.

Please fix lint & yapf and run an integration test suite that runs the example.

Whoops, missed the linting failures - thanks.
Python3.9 postcommit does run the example, so that's already done. I'll run it again though.

Run Python 3.9 PostCommit

source: gharchive/pull-request

authors: damccorm tvalentyn

repo: apache/beam

license_type: permissive

license_source: github-api


text:
add parallel number to gradle.properties

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Java_Kafka_IO_Direct PreCommit

Run Java_Kafka_IO_Direct PreCommit

Run Java_Kafka_IO_Direct PreCommit

run python_runners precommit

run python_coverage precommit

run java precommit

run java_gcp_io_direct precommit

java precommit failure is a flake

GCP IO test failure is unrelated. We might have a quota issue on Spanner tests?

That's a known issue with spanner permissions #26208

All 4 failures appear to be unrelated flakes

authors: johnjcasey jrmccluskey


text:
Add Cloud Object Translators for Coders

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

These are all the translators with known types.
Implementations and tests for standard coders that aren't known types in Dataflow are still in progress.

R: @lukecwik

Rebased, PTAL

Changes Unknown when pulling a71458dc509ca9c9e11cb7e9fc984ddc6b4febda on tgroh:cloud_object_translator_impls into ** on apache:master**.

Coverage increased (+0.3%) to 70.411% when pulling 612ca50a436a5fd9338f6fc09b19db4e2097a603 on tgroh:cloud_object_translator_impls into 3fff52d43cbdc3b34f7d3e2ed7ac62f5ff06ccb6 on apache:master.

authors: coveralls tgroh


text:
Beam quest Blog Addon

Rescale image and repoint path
Update blog post to include free.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @aaltay
R: @rszper


text:
Invoke teardown when DoFn throws in portable runners

Fix (partly) #31381
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

java.lang.RuntimeException: Dataflow job 2024-09-20_12_32_18-8696550461908579531 terminated in state RUNNING but did not return a failure reason.
	at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:149)

however the job does succeeded. The worker log shows
2024-09-20 15:34:47.080 EDT
Entering instruction processing loop
2024-09-20 15:35:02.066 EDT
Memory is used/total/max = 392/890/4951 MB, GC last/max = 3.47/3.47 % (configured threshold: 50.00%), #pushbacks=0, gc thrashing=false
2024-09-20 15:40:02.073 EDT
Memory is used/total/max = 392/890/4951 MB, GC last/max = 0.00/3.47 % (configured threshold: 50.00%), #pushbacks=0, gc thrashing=false
2024-09-20 15:45:02.078 EDT
Memory is used/total/max = 88/930/4951 MB, GC last/max = 1.06/3.47 % (configured threshold: 50.00%), #pushbacks=0, gc thrashing=false
2024-09-20 15:47:46.638 EDT
Filepattern gs://apache-beam-samples/shakespeare/sonnets.txt matched 1 files with total size 95662

took extended period of time to match the file source.

Java PVR Spark Batch failed twice, each time different test. This is due to #30512. None of them are related to ParDo lifecycle.

Rerun Dataflow Example and test passed. Looks like intermittent issue (cause the SDK harness start to work 12 minutes late)

PTAL @scwhittle @robertwb thanks!

R: @scwhittle

rebased onto latest HEAD to resolve merge conflict (on CHANGES.md)

Looks like the spark runner failure might be real from a test that is now not being filtered.

PostCommit Java PVR Spark Batch two attempt different random test failing due to #30512, not related to this PR, merging for now

authors: Abacn scwhittle


text:
[BEAM-5539] Beam Dependency Update Request: google-cloud-pubsub

for testing

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

run python postcommit

run portable_python precommit

run python postcommit

@udim Looks like the pubsub interface hasn't changed since 0.35.4. Is there anything I should check further?

No, test coverage in pre and postcommits should be enough.

@ccy for merge please

Thanks, this LGTM.

authors: charlesccychen ihji udim


text:
[BEAM-6794] Partially roll back BQ storage integration tests.

This change removes the BigQueryIOStorageReadIT integration test from
the googleCloudPlatformFnApiWorkerIntegrationTest Gradle task so that
the test will not run with the fnapi framework for the time being.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @chamikaramj @boyuanzz @Ardagan

Run Java PortabilityApi PostCommit

@chamikaramj Could you please help us merge this PR?

Run Python PreCommit

LGTM. Thanks.

authors: boyuanzz chamikaramj kmjung


text:
[BEAM-6982] Fix Cassandra-io flakiness and clean leftovers

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Faillures seem unrelated: re-running build

Run Java PreCommit


text:
BIGTOP-4018. Upgrade Alluxio to 2.9.3.

Description of PR
This PR upgrades Alluxio version to 2.9.3.
How was this patch tested?
Ran the following commands on Rocky 8 (x86_64).
$ ./gradlew allclean alluxio-pkg repo -Dbuildwithdeps=true
$ cd provisioner/docker
$ ./docker-hadoop.sh -d -dcp -C config_rockylinux-8.yaml -L -k hdfs,alluxio -s alluxio -c 1

...

> Task :bigtop-tests:smoke-tests:alluxio:test
Finished generating test XML results (0.013 secs) into: /bigtop-home/bigtop-tests/smoke-tests/alluxio/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.027 secs) into: /bigtop-home/bigtop-tests/smoke-tests/alluxio/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:alluxio:test (Thread[Daemon worker,5,main]) completed. Took 21.952 secs.

BUILD SUCCESSFUL in 1m 24s
31 actionable tasks: 5 executed, 26 up-to-date

For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
[x] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

Thanks for working on it, @sekikn
I'd like to verify your series of PRs on the Debian system(Ubuntu-22.04 Arm64).

The PR was verified in Arm64 Ubuntu-22.04.
But it failed to run the smoke tests in Arm64 Fedora-36:
Need to upgrade Fedora:
Error: Could not update: Execution of '/usr/bin/dnf -d 0 -e 1 -y install alluxio' returned 1: Error:
 Problem: conflicting requests
  - nothing provides libc.so.6(GLIBC_2.2.5)(64bit) needed by alluxio-2.9.3-1.fc36.aarch64
  - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by alluxio-2.9.3-1.fc36.aarch64
  - nothing provides libc.so.6(GLIBC_2.2.5)(64bit) needed by alluxio-2.8.0-2.fc36.aarch64
  - nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by alluxio-2.8.0-2.fc36.aarch64

The issue is related to GLIBC confliction, not caused by this PR.
Our 3.2.1 release jenkins also reported this issue:
https://ci.bigtop.apache.org/view/Test/job/Bigtop-3.2.1-smoke-tests/COMPONENTS=hdfs.alluxio@alluxio,OS=fedora-36-aarch64-deploy/lastBuild/consoleText
I'd like to upgrade Fedora to version 39 to fix this issue in the other thread.
+1 for this PR.
Thanks.

authors: guyuqi sekikn

repo: apache/bigtop

license_source: bigquery


text:
Improve catalog locations info and editability

takes advantage of improvements in https://github.com/apache/brooklyn-server/pull/70 esp in rest api to make the locations shown in catalog more informative and useful, disabling delete for those which can't be deleted, and even allowing editing for those following the new style of adding to the catalog
(should work nicely in conjunction with #17 !)
note this requires https://github.com/apache/brooklyn-server/pull/70

The code looks good but

I cannot edit any locations I manually added.
properties don't show up for locations I manually added.

It that normal @ahgittin ?

Oh wait, do I need to run the server with https://github.com/apache/brooklyn-server/pull/70?

yes, that's what i meant by "requires" :)

Sorry, I'm still not use to the new repo structure and forget things in the way

authors: ahgittin tbouron

repo: apache/brooklyn-ui


text:
AaaP follow-up: Handling of legacy remotes & CAS/Artifact services

See original issue on GitLab
In GitLab by [Gitlab user @tpollard] on May 15, 2019, 14:50
Background
With https://gitlab.com/BuildStream/buildstream/merge_requests/1292 landing, the way buildstream caches & defines an artifact has changed, moving from a reference based service. As such existing artifact remotes need to be updated to support this change and then repopulated with proto based artifacts. (Not supporting the migration of artifacts was discussed in the proposal https://mail.gnome.org/archives/buildstream-list/2019-January/msg00013.html)
When trying to create the user story of what happens when somebody using bst that has AaaP hits a remote that hasn't been updated to support it no direct conclusion was made (https://gitlab.com/BuildStream/buildstream/merge_requests/1292#note_169300280). As such at the current time of writing a user running in interactive mode will hit an error on each attempted artifact pull/push, with the quickest remedy being to disable the 'old' remote url until it has been updated. In the linked discussion it shows some of the challenges that have arisen when trying to improve the user experience here whilst still keeping the casremote class 'Artifact' free.
Task description

[ ] Decide what the user story should be if a user is unable to push/pull to a remote due to it not being compatible. What would the default behaviour be, should this be user configurable?
[ ] How does buildbox-casd help with separation of remote communication and CAS/Artifact detangling out of buildstream. Does something stopgap need to be put inplace in the meantime?

In GitLab by [Gitlab user @tpollard] on May 15, 2019, 14:50
marked this issue as related to #909

In GitLab by [Gitlab user @tpollard] on May 15, 2019, 14:52
changed the description

In GitLab by [Gitlab user @raoul].hidalgocharman on May 29, 2019, 16:37
mentioned in commit 738e6cc8986553c868c3b9f5a6c54061e55aeb80

In GitLab by [Gitlab user @raoul].hidalgocharman on May 29, 2019, 16:48
mentioned in commit f8036e413c776c9a69de9c148d723eb09716a072

In GitLab by [Gitlab user @raoul].hidalgocharman on May 30, 2019, 12:36
assigned to [Gitlab user @raoul].hidalgocharman

In GitLab by [Gitlab user @raoul].hidalgocharman on May 30, 2019, 12:39
mentioned in merge request !1362

In GitLab by [Gitlab user @raoul].hidalgocharman on May 30, 2019, 12:53
mentioned in commit 3e716841cc6da996f11a8fc4af5425275d8d81ed

In GitLab by [Gitlab user @raoul].hidalgocharman on May 30, 2019, 13:44
mentioned in commit f69b06bcaac2dc25560eacd77d014010de96dee0

In GitLab by [Gitlab user @jjardon] on May 30, 2019, 15:27
marked this issue as related to #1037

In GitLab by [Gitlab user @raoul].hidalgocharman on May 30, 2019, 17:24
mentioned in commit 559a39e5994184c9f6554dc78e2a91bb7c6a5aab

In GitLab by [Gitlab user @raoul].hidalgocharman on May 31, 2019, 09:43
mentioned in commit 74ae5dbf89bb3299d36034f2fd4b8a75259cdd23

In GitLab by [Gitlab user @raoul].hidalgocharman on May 31, 2019, 10:09
mentioned in commit d88c4aa38a2cb119c36c0648baa722b35369f786

In GitLab by [Gitlab user @raoul].hidalgocharman on May 31, 2019, 11:03
mentioned in commit 5ce42a724fcc317cfceb4de6deb43b1bbf0c29e3

In GitLab by [Gitlab user @raoul].hidalgocharman on May 31, 2019, 11:03
mentioned in commit 5bf5ef19b742aab1c2a381b41d5313f070e91932

In GitLab by [Gitlab user @raoul].hidalgocharman on May 31, 2019, 11:22
mentioned in merge request !1366

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 3, 2019, 09:55
mentioned in commit 916e069c5a6c585465d0040c73933b13dab229b1

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 3, 2019, 09:55
mentioned in commit 4b88e9adbcb8f34eeece537d3e4864c6198067e3

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 5, 2019, 10:02
mentioned in commit 9ca3dcfae25cd9acc366253d060afcd57ef41bcb

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 5, 2019, 10:02
mentioned in commit b80a71040cc3d52b3b9b82fee45d814d35632d72

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 5, 2019, 15:41
mentioned in commit 216e17384575e67ee1c70209d97f666672eb31bd

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 5, 2019, 15:41
mentioned in commit eda3d28249a2a21bf09bcc7340d324888ba6584d

In GitLab by [Gitlab user @marge-bot123] on Jun 5, 2019, 16:14
closed via merge request !1366

In GitLab by [Gitlab user @marge-bot123] on Jun 5, 2019, 16:14
mentioned in commit 246571606853aef6da856d778f783b14339a2cea

source: gharchive/issue

repo: apache/buildstream


text:
Creating Ingress and services with Kubernetes

Hi @nicolaferraro ,
I'm running Kamel on Docker for Mac + Kubernetes.
I've deployed a sample integration examples/routes-rest.js and was expecting it to create svc, ingress etc so that I can invoke my RESTful service.  (As you mentioned in : "So, for example, if you define a REST endpoint using the Camel REST DSL, the operator will make sure that your integration is exposed to the outside and it will create a Service and a Route on OpenShift to expose it, or a Ingress on vanilla Kubernetes." [1] )
However, I don't see that the Kamel operator create svc or ingress.
Do I have to manually create ingress, service etc for that? Here are the pod and svc information.
kubectl get pods
NAME                                READY     STATUS    RESTARTS   AGE
camel-k-operator-64c5686848-4zkzt   1/1       Running   0          2h
./kamel get
NAME		CONTEXT	STATUS
routes
routes-rest
kubectl get svc
NAME               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
camel-k-operator   ClusterIP   10.97.6.139           60000/TCP   2h
kubernetes         ClusterIP   10.96.0.1             443/TCP     3h
[1] https://www.nicolaferraro.me/2018/10/15/introducing-camel-k/

Hi @kasun04, Camel K needs a docker registry to push images to. That's why for local development it's better to use Minikube, because you can install a default registry through a Minikube addon.
Is a docker registry present in your dev-env?
Also, can you inspect the status of the platform objects?
kubectl get integrationplatform -o yaml
kubectl get integrationcontext -o yaml
kubectl get integration -o yaml

I suspect that the platform is not "Ready" because it cannot push the context images.

repo: apache/camel-k


text:
Release 1.12.0

Related:
https://github.com/apache/camel-k-runtime/issues/951
https://github.com/apache/camel-k-runtime/issues/950
https://github.com/apache/camel-kamelets/issues/1234
https://github.com/quarkusio/quarkus-platform/pull/753
Deps:

Camel 3.20.1
Camel-Quarkus 2.16.0
Camel-Kamelets 3.20.1 (already released)
Quarkus platform 2.16.0.Final

Also let's wait for #3977 to be merged

To be fixed, #4011

#3977 done and #4011 fixed

Some minor fix that should be merged today. Additionally @essobedo is working on the possibility to enable native on more DSL #4021 hopefully we can get it working for next week. I've started working on #3797 and I should have this complete by the end of the day.

#3904 Done.

#3903 Done. Just needs to update a bit the trait docs.
I guess we are only missing #4021

Yes, I think @essobedo PR is already in good shape, likely only few last details missing.

We also need #4058

I am also working to complete #4055 and have the CRD dependency published beside the rest of artifacts.

We're good to go. No outstanding issues.

Sounds good, I'll start probably next week the release. On Monday. Thanks.

Starting release process.

Released. Still needs to add operators. But done.

authors: oscerd squakez


text:
Release 2.3.1

As $subject

Release tasks in progress


text:
fix(e2e): Update latest released version to fix upgrade tests

Release Note
NONE

The failures are not related to this PR. Let's merge it to make the upgrade tests green again.


text:
[CARBONDATA-1485] Timestamp no dictionary bug

when partition column is timestamp type and dictionary exclude, data file is not operated correctly in add/split partition function. This pr is to fix this bug

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/284/

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/159/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/915/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/285/

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/160/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/916/

retest this please

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/171/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/296/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/927/

please add test cases

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1190/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/559/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1424/

retest this please

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/737/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1966/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3071/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4315/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5203/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7140/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5916/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/67/

Build Failed  with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8816/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/749/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/571/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2019/

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2196/

Build Failed  with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10271/

Build Failed  with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/92/

Build Failed  with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/91/

Build Failed  with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/89/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3253/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5435/

repo: apache/carbondata


text:
[HOTFIX] Refactor code to optimize partition pruning

Why is this PR needed?
If number of partitions is more on a segment, cost of checking if (indexPath  equalsIgnoreCase partitionsToPrune) in BlockletIndexFactory.getTableBlockUniqueIdentifierWrappers() is high.
What changes were proposed in this PR?

Store location of partition filter in a SET and check if it contains indexPath.
Remove unused variables

Does this PR introduce any user interface change?

No

Is any new testcase added?

No

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1022/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2734/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1023/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2735/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2737/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1025/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2763/

retest this please

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2766/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1053/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1055/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2768/

Build Failed  with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1056/

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2769/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2771/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1058/

retest this please

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1091/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2803/

@Indhumathi27 As we are checking the contains for the index Path and partition localtion, its better to say at place, instead of Set<Path> partitions , we can say Set<Path> partitionLocaltions. This is more better i think.

@Indhumathi27 , this is an improvement, can you please add a jira and change the PR heading?

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2820/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1107/

retest this please

Build Failed  with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2842/

retest this please

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1130/

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1136/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2848/

LGTM

LGTM


text:
CASSGO-35 standardize spelling of datacenter

Keeps the repo internally consistent. The public interface is kept to prevent breakage.

@martin-sucha Would yo be able to kick the CI here?

repo: apache/cassandra-gocql-driver


text:
sasi: AND optimizations

See CASSANDRA-12915
when op is AND:

do not use indexes that return way more results that the others
do not use more indexes if one of them returns no results
try to use something smarter to get the primary index (which is
important because it limits the searches done on other indexes)

Cosmetic changes such as changing logger.debug() to logger.trace() will be done later, I'd like a first opinion on the concept.

Is there a JIRA ticket associated to your patch?

I guess it is CASSANDRA-12915. Could you mark it as patch available?

Done, I'm not yet very familiar with the workflow.

Thanks. :-)
People are usually tracking the JIRA tickets and not the pull requests. So it is best to keep it in sync with your developments.

authors: blerer iksaif

repo: apache/cassandra


text:
[CELEBORN-1801] Remove out-of-dated flink 1.14 and 1.15

What changes were proposed in this pull request?
Remove out-of-dated flink 1.14 and 1.15.
For more information, please see the discussion thread: https://lists.apache.org/thread/njho00zmkjx5qspcrbrkogy8s4zzmwv9
Why are the changes needed?
Reduce maintenance burden.
Does this PR introduce any user-facing change?
Yes.
How was this patch tested?
Changes can be covered by existing tests.

Thanks for driving this, merged to main for 0.6

repo: apache/celeborn


text:
Why do we package final artifacts with developer profile?

This technically is not an issue, but I was just got curios why we package our final artifacts (both rpm and deb) with developer maven profile. in:

debian/rules#L16
packaging/centos63/cloud.spec#L209-223
packaging/centos7/cloud.spec#L192

It doesn't make sense since these are going to be installed in prod or alike, and most probably won't be any deploydb and db.properties.override.
I believe we should remove these.

@khos2ow there is no need to enable the developer profile, and enabling the profile causes no difference to the built rpm, deb files.

With the developer profile enabled, maven modules for test and tools are added to the build tree which is why it is needed as the rpm/deb create the marvin and integration-test pkgs. See https://github.com/apache/cloudstack/blob/master/pom.xml#L1187
Since the question has been answered, I'll close this. If you need to send any change, please directly submit a PR, thanks.

repo: apache/cloudstack


text:
SystemVMs running in Xen HVM mode are not configured

ISSUE TYPE

Bug Report

COMPONENT NAME

systemVM image 4.11 for XenServer

CLOUDSTACK VERSION

4.11.1

CONFIGURATION

Cloudstack 4.11.1
XenServer 7.1.1
SystemVM 4.11.1 Fri Jun 22 07:52:17 UTC 2018 running in HVM mode

OS / ENVIRONMENT

N/A
SUMMARY

Newly created system VMs are not configured.
Related to #2561
virt-what cpuid helper detects Xen HVM guests with viridian extension enabled as hyperV guests.
root@v-2158-VM:~# /usr/lib/x86_64-linux-gnu/virt-what-cpuid-helper
Microsoft Hv

So, virt-what returns the following on Xen HVM SystemVMs:
root@v-2158-VM:~# virt-what
hyperv
xen
xen-domU

In this case, the /opt/cloud/bin/setup/cloud-early-config script assumes that we're running on a paravirtualized guest and fails to get boot args.
EXPECTED RESULTS

hypervisor: xen-hvm
systemvm type: should be detected.
Sun Jul 22 03:18:28 UTC 2018 Executing cloud-early-config
Sun Jul 22 03:18:28 UTC 2018 Detected that we are running inside xen-hvm
Sun Jul 22 03:18:28 UTC 2018 Scripts checksum detected: oldmd5=76c6bbddfc1d55d1b212ad8ca07190d4 newmd5=76c6bbddfc1d55d1b212ad8ca07190d4
Sun Jul 22 03:18:28 UTC 2018 Configuring systemvm type=consoleproxy
Sun Jul 22 03:18:29 UTC 2018 Setting up console proxy system vm
...

ACTUAL RESULTS

hypervisor: xen-domU
systemvm type : empty

root@systemvm:~# cat /var/log/cloud.log
Fri Jul 20 12:55:19 UTC 2018 Executing cloud-early-config
Fri Jul 20 12:55:19 UTC 2018 Detected that we are running inside xen-domU
Fri Jul 20 12:55:19 UTC 2018 Scripts checksum detected: oldmd5=76c6bbddfc1d55d1b212ad8ca07190d4 newmd5=76c6bbddfc1d55d1b212ad8ca07190d4
Fri Jul 20 12:55:19 UTC 2018 Configuring systemvm type=
Fri Jul 20 12:55:19 UTC 2018 Finished setting up systemvm

@dpassante the only difference between how xen-domU and xen are used by the systemvm is wrt the way of getting an initial cmdline. Otherwise, there is no difference. Are you able to see systemvms come up or are they failing?

ping @dpassante based on the comment above I'll remove this from 4.11.2.0 milestone, if you can reply and share any suggestive fix we can help with reviewing and testing of a fix.

@rhtyd the systemvms are booting but not configured as the PV-ARGs are not present in /proc/cmdline.
The easy way to fix is to do not rely on virt-what for Xen guests and move lines 44-52 to the beginning of the hypervisior() function.
I'll also share the fix that is currently running on my environments.

authors: dpassante rhtyd


text:
Can't set the "allow.user.create.projects" to false - backend error

Current master:

ping @Spaceman1984 if you have time/will - please take a look. thx

Fixed in PR https://github.com/apache/cloudstack/pull/4320

authors: andrijapanicsb rhtyd


text:
CLOUDSTACK-9008 - Pass hypervisor snapshot reserve field in when crea…

…ting compute and disk offerings
https://issues.apache.org/jira/browse/CLOUDSTACK-9008

Hi @mike-tutkowski thanks for the details in the jira ticket and the pr. From a quick look it looks good but when i'm at my laptop again I will run some generic tests. The managed storage itself I cannot test but I trust you for that. Is there any test results you can share?

What I did was run a Marvin test I have that creates a VM whose root disk is placed on managed storage. The test then creates a hypervisor snapshot of the VM.
Before my Marvin changes, the test was failing (and the root disk was being corrupted). Now that Marvin has been enhanced, the hypervisor snapshot reserve value is being passed to CloudStack from Marvin successfully and the overall test passes.
The hypervisor snapshot reserve field is ignored by non-managed storage, so this change should not impact that type of storage.

LGTM, based on a set of tests that I run on this branch (which I rebased myself first):
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
component/test_vpc_redundant.py \
component/test_routers_iptables_default_policy.py \
component/test_routers_network_ops.py \
component/test_vpc_router_nics.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py \
smoke/test_network.py

Result:
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Stop existing router, add a PF rule and check we can access the VM ... === TestName: test_isolate_network_FW_PF_default_routes | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_RVR_Network_FW_PF_SSH_default_routes | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Check that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 29 tests in 13039.556s

OK

And:
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_privategw_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_vpc_vpn.py \
smoke/test_service_offerings.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

Result:
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 44 tests in 9256.766s

OK

These test do not cover your change, all they do is show you didn't break them. Someone else needs to review the code.

LGTM based on code inspection (Why the move of the customizediops bits? harmless but senseless)

I liked that ordering better...seems more logical: 1) customized IOPS
(true/false), 2) Min IOPS, and 3) Max IOPS (2 and 3 are ignored by the
backend if 1 is true).
On Monday, November 2, 2015, Daan Hoogland notifications@github.com wrote:

LGTM based on code inspection (Why the move of the customizediops bits?
harmless but senseless)
—
Reply to this email directly or view it on GitHub
https://github.com/apache/cloudstack/pull/1016#issuecomment-153002626.

--
Mike Tutkowski
Senior CloudStack Developer, SolidFire Inc.
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the cloud
http://solidfire.com/solution/overview/?video=play™

so in English: for readability. makes sense ;)

Thanks! I went ahead and closed https://issues.apache.org/jira/browse/CLOUDSTACK-9008.

authors: DaanHoogland mike-tutkowski remibergsma


text:
CLOUDSTACK-9460: For long running transactions, if the connection is

timed out by the mysql server then refresh it

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✖centos7 ✖debian. JID-5

Packaging result: ✔centos6 ✔centos7 ✖debian. JID-10

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-15

LGTM from the code change.

ACS CI BVT Run
Sumarry:
Build Number 104
Hypervisor xenserver
NetworkType Advanced
Passed=101
Failed=2
Skipped=4
Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0
Failed tests:

test_ssvm.py

test_03_ssvm_internals Failed

test_routers_network_ops.py

test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_3d_gpu_support
test_deploy_vgpu_enabled_vm
Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_snapshots.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

@blueorangutan help

@borisstoyanov I understand these words: "help", "hello", "thanks", "package", "test"
Test run usage: test (mgmt server, one of: centos6, centos7, ubuntu) (hypervisor, one of: kvm-centos6, kvm-centos7, kvm-ubuntu, xenserver-65sp1, xenserver-62sp1, vmware-60u2, vmware-55u3, vmware-51u1, vmware-50u1)
Authorized contributors for kicking Trillian Jenkins test jobs are: ['rhtyd', 'jburwell', 'murali-reddy', 'abhinandanprateek', 'PaulAngus', 'borisstoyanov', 'karuturi']

@blueorangutan test centos6 kvm-centos6

@borisstoyanov a Trillian-Jenkins test job (centos6 mgmt + kvm-centos6) has been kicked to run smoke tests

Trillian test result (tid-75)
Environment: kvm-centos6 (x2), Advanced Networking
Total time taken: 28242 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1674-t75-kvm-centos6.zip
Test completed. 40 look ok, 13 have errors

Test
Result
Time (s)

test_10_attachAndDetach_iso
Failure
60.147

test_04_rvpc_privategw_static_routes
Failure
173.295

test_03_vpc_privategw_restart_vpc_cleanup
Failure
92.665

test_02_vpc_privategw_static_routes
Failure
92.624

test_01_vpc_privategw_acl
Failure
52.252

test_router_dhcphosts
Error
809.220

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Error
741.847

test_network_rules_acquired_public_ip_2_nat_rule
Error
738.909

test_network_rules_acquired_public_ip_1_static_nat_rule
Error
739.236

test_isolate_network_password_server
Error
816.460

test_extendPhysicalNetworkVlan
Error
0.038

test_dedicatePublicIpRange
Error
0.369

test_04_extract_template
Error
5.115

test_02_routervm_iptables_policies
Error
780.923

test_01_vpc_remote_access_vpn
Error
0.105

test_01_snapshot_root_disk
Error
0.064

test_01_single_VPC_iptables_policies
Error
899.734

ContextSuite context=TestRouterDHCPHosts>:teardown
Error
870.550

ContextSuite context=TestListIdsParams>:setup
Error
0.000

ContextSuite context=TestDedicateGuestVlanRange>:setup
Error
0.000

test_router_dns_guestipquery
Success
106.818

test_router_dns_externalipquery
Success
0.077

test_rolepermission_lifecycle_update
Success
7.140

test_rolepermission_lifecycle_list
Success
7.074

test_rolepermission_lifecycle_delete
Success
6.926

test_rolepermission_lifecycle_create
Success
6.808

test_rolepermission_lifecycle_concurrent_updates
Success
7.005

test_role_lifecycle_update_role_inuse
Success
6.900

test_role_lifecycle_update
Success
12.019

test_role_lifecycle_list
Success
6.859

test_role_lifecycle_delete
Success
11.948

test_role_lifecycle_create
Success
6.938

test_role_inuse_deletion
Success
6.894

test_role_account_acls_multiple_mgmt_servers
Success
9.643

test_role_account_acls
Success
9.549

test_releaseIP
Success
249.394

test_reboot_router
Success
425.015

test_public_ip_user_account
Success
10.402

test_public_ip_admin_account
Success
40.355

test_oobm_zchange_password
Success
20.661

test_oobm_multiple_mgmt_server_ownership
Success
24.636

test_oobm_issue_power_status
Success
10.571

test_oobm_issue_power_soft
Success
15.618

test_oobm_issue_power_reset
Success
15.601

test_oobm_issue_power_on
Success
15.598

test_oobm_issue_power_off
Success
15.596

test_oobm_issue_power_cycle
Success
15.620

test_oobm_enabledisable_across_clusterzones
Success
58.820

test_oobm_enable_feature_valid
Success
5.224

test_oobm_enable_feature_invalid
Success
0.142

test_oobm_disable_feature_valid
Success
5.256

test_oobm_disable_feature_invalid
Success
0.163

test_oobm_configure_invalid_driver
Success
0.103

test_oobm_configure_default_driver
Success
0.117

test_oobm_background_powerstate_sync
Success
13.777

test_nic_secondaryip_add_remove
Success
193.817

test_network_acl
Success
66.313

test_deployvm_userdispersing
Success
20.746

test_deployvm_userdata_post
Success
10.449

test_deployvm_userdata
Success
86.200

test_deployvm_userconcentrated
Success
20.696

test_deployvm_firstfit
Success
55.883

test_deploy_vm_multiple
Success
253.992

test_deploy_vm_from_iso
Success
428.981

test_deploy_vm
Success
0.034

test_delete_account
Success
254.393

test_default_role_deletion
Success
6.976

test_create_pvlan_network
Success
5.276

test_createRegion
Success
0.051

test_createPortablePublicIPRange
Success
15.425

test_createPortablePublicIPAcquire
Success
15.710

test_assign_and_removal_lb
Success
134.141

test_advZoneVirtualRouter
Success
0.023

test_UpdateStorageOverProvisioningFactor
Success
0.172

test_UpdateConfigParamWithScope
Success
0.174

test_DeployVmAntiAffinityGroup_in_project
Success
101.869

test_DeployVmAntiAffinityGroup
Success
66.138

test_CreateTemplateWithDuplicateName
Success
40.544

test_10_destroy_cpvm
Success
192.093

test_09_reboot_router
Success
35.393

test_09_expunge_vm
Success
125.206

test_09_destroy_ssvm
Success
164.034

test_09_delete_detached_volume
Success
15.628

test_08_start_router
Success
35.356

test_08_resize_volume
Success
15.480

test_08_reboot_cpvm
Success
101.770

test_08_migrate_vm
Success
46.156

test_08_list_system_templates
Success
0.033

test_07_stop_router
Success
10.204

test_07_restore_vm
Success
0.137

test_07_resize_fail
Success
20.566

test_07_reboot_ssvm
Success
134.180

test_07_list_public_templates
Success
0.035

test_07_list_default_iso
Success
0.064

test_06_stop_cpvm
Success
106.898

test_06_router_advanced
Success
0.075

test_06_download_detached_volume
Success
15.372

test_06_destroy_vm
Success
126.302

test_05_template_permissions
Success
0.065

test_05_stop_ssvm
Success
164.554

test_05_rvpc_multi_tiers
Success
493.912

test_05_router_basic
Success
0.045

test_05_iso_permissions
Success
0.066

test_05_detach_volume
Success
100.249

test_04_rvpc_network_garbage_collector_nics
Success
842.023

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
249.137

test_04_restart_network_wo_cleanup
Success
5.876

test_04_extract_Iso
Success
5.154

test_04_delete_attached_volume
Success
10.239

test_04_create_fat_type_disk_offering
Success
0.085

test_04_cpvm_internals
Success
1.471

test_04_change_offering_small
Success
239.226

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
193.527

test_03_ssvm_internals
Success
3.755

test_03_restart_network_cleanup
Success
60.620

test_03_reboot_vm
Success
125.968

test_03_download_attached_volume
Success
15.365

test_03_delete_template
Success
5.121

test_03_delete_service_offering
Success
0.049

test_03_delete_iso
Success
95.220

test_03_delete_disk_offering
Success
0.050

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Success
506.023

test_03_RVR_Network_check_router_state
Success
271.622

test_02_sys_template_ready
Success
0.147

test_02_start_vm
Success
10.248

test_02_router_internal_adv
Success
1.345

test_02_redundant_VPC_default_routes
Success
536.188

test_02_port_fwd_on_non_src_nat
Success
55.798

test_02_list_cpvm_vm
Success
0.146

test_02_isolate_network_FW_PF_default_routes_egress_false
Success
251.745

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
527.119

test_02_edit_template
Success
90.286

test_02_edit_service_offering
Success
0.067

test_02_edit_iso
Success
0.095

test_02_edit_disk_offering
Success
0.061

test_02_deploy_vm_root_resize
Success
6.951

test_02_create_sparse_type_disk_offering
Success
0.093

test_02_create_lb_rule_non_nat
Success
187.782

test_02_attach_volume
Success
76.806

test_02_VPC_default_routes
Success
334.120

test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false
Success
326.669

test_01_vpc_site2site_vpn
Success
175.907

test_01_updatevolumedetail
Success
5.148

test_01_sys_vm_start
Success
0.184

test_01_stop_vm
Success
40.373

test_01_router_internal_basic
Success
0.714

test_01_reset_vm_on_reboot
Success
131.025

test_01_redundant_vpc_site2site_vpn
Success
271.972

test_01_primary_storage_nfs
Success
35.942

test_01_port_fwd_on_src_nat
Success
141.897

test_01_nic
Success
511.421

test_01_list_sec_storage_vm
Success
0.152

test_01_isolate_network_FW_PF_default_routes_egress_true
Success
236.871

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
435.796

test_01_deploy_vm_root_resize
Success
6.964

test_01_create_volume
Success
861.859

test_01_create_template
Success
45.615

test_01_create_service_offering
Success
0.139

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1090.599

test_01_create_lb_rule_src_nat
Success
240.602

test_01_create_iso
Success
66.957

test_01_create_disk_offering
Success
0.152

test_01_VPC_nics_after_destroy
Success
613.333

test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true
Success
320.237

test_00_deploy_vm_root_resize
Success
198.606

login_test_saml_user
Success
24.766

test_vm_nic_adapter_vmxnet3
Skipped
0.001

test_static_role_account_acls
Skipped
0.025

test_deploy_vgpu_enabled_vm
Skipped
0.041

test_3d_gpu_support
Skipped
0.041

test_06_copy_template
Skipped
0.000

test_06_copy_iso
Skipped
0.000

test_03_delete_vm_snapshots
Skipped
0.000

test_02_revert_vm_snapshots
Skipped
0.000

test_01_test_vm_volume_snapshot
Skipped
0.001

test_01_scale_vm
Skipped
0.001

test_01_primary_storage_iscsi
Skipped
0.040

test_01_create_vm_snapshots
Skipped
0.000

@abhinandanprateek ping, how severely do we need this for upcoming release?
@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-81

@rhtyd this issue impacts any user with long-running secondary storage jobs.  I think it is a valuable fix for all users as the only recourse to fix these errors is to manually cleanup the database.
@abhinandanprateek I would like to get this PR merged in time for 4.8.2.0.  Could you please change the base branch of this PR to 4.8 and rebase it to pull in the most recent Marvin enhancements/fixes?

LGTM. I'll re-kick tests.
@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-116

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-235)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 25871 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1674-t235-kvm-centos7.zip
Test completed. 42 look ok, 1 have error(s)

Test
Result
Time (s)
Test File

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Failure
395.88
test_vpc_redundant.py

test_01_vpc_site2site_vpn
Success
182.17
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
67.48
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
284.25
test_vpc_vpn.py

test_02_VPC_default_routes
Success
280.45
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
589.10
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
524.64
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1382.68
test_vpc_redundant.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Success
593.09
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Success
754.12
test_vpc_redundant.py

test_09_delete_detached_volume
Success
16.24
test_volumes.py

test_08_resize_volume
Success
15.59
test_volumes.py

test_07_resize_fail
Success
20.71
test_volumes.py

test_06_download_detached_volume
Success
15.35
test_volumes.py

test_05_detach_volume
Success
100.37
test_volumes.py

test_04_delete_attached_volume
Success
10.37
test_volumes.py

test_03_download_attached_volume
Success
15.54
test_volumes.py

test_02_attach_volume
Success
73.80
test_volumes.py

test_01_create_volume
Success
743.60
test_volumes.py

test_deploy_vm_multiple
Success
304.35
test_vm_life_cycle.py

test_deploy_vm
Success
0.03
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.02
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
26.76
test_vm_life_cycle.py

test_09_expunge_vm
Success
125.29
test_vm_life_cycle.py

test_08_migrate_vm
Success
41.43
test_vm_life_cycle.py

test_07_restore_vm
Success
0.28
test_vm_life_cycle.py

test_06_destroy_vm
Success
126.24
test_vm_life_cycle.py

test_03_reboot_vm
Success
126.00
test_vm_life_cycle.py

test_02_start_vm
Success
10.29
test_vm_life_cycle.py

test_01_stop_vm
Success
40.44
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
126.33
test_templates.py

test_08_list_system_templates
Success
0.05
test_templates.py

test_07_list_public_templates
Success
0.06
test_templates.py

test_05_template_permissions
Success
0.07
test_templates.py

test_04_extract_template
Success
5.19
test_templates.py

test_03_delete_template
Success
5.11
test_templates.py

test_02_edit_template
Success
90.12
test_templates.py

test_01_create_template
Success
45.71
test_templates.py

test_10_destroy_cpvm
Success
191.84
test_ssvm.py

test_09_destroy_ssvm
Success
163.73
test_ssvm.py

test_08_reboot_cpvm
Success
131.71
test_ssvm.py

test_07_reboot_ssvm
Success
134.12
test_ssvm.py

test_06_stop_cpvm
Success
162.07
test_ssvm.py

test_05_stop_ssvm
Success
139.03
test_ssvm.py

test_04_cpvm_internals
Success
1.22
test_ssvm.py

test_03_ssvm_internals
Success
3.62
test_ssvm.py

test_02_list_cpvm_vm
Success
0.24
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.19
test_ssvm.py

test_01_snapshot_root_disk
Success
11.38
test_snapshots.py

test_04_change_offering_small
Success
209.89
test_service_offerings.py

test_03_delete_service_offering
Success
0.04
test_service_offerings.py

test_02_edit_service_offering
Success
0.08
test_service_offerings.py

test_01_create_service_offering
Success
0.10
test_service_offerings.py

test_02_sys_template_ready
Success
0.14
test_secondary_storage.py

test_01_sys_vm_start
Success
0.21
test_secondary_storage.py

test_09_reboot_router
Success
45.54
test_routers.py

test_08_start_router
Success
30.38
test_routers.py

test_07_stop_router
Success
10.19
test_routers.py

test_06_router_advanced
Success
0.05
test_routers.py

test_05_router_basic
Success
0.04
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.70
test_routers.py

test_03_restart_network_cleanup
Success
60.59
test_routers.py

test_02_router_internal_adv
Success
1.08
test_routers.py

test_01_router_internal_basic
Success
0.62
test_routers.py

test_router_dhcphosts
Success
277.94
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
0.09
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
131.27
test_reset_vm_on_reboot.py

test_createRegion
Success
0.04
test_regions.py

test_create_pvlan_network
Success
5.21
test_pvlan.py

test_dedicatePublicIpRange
Success
0.42
test_public_ip_range.py

test_04_rvpc_privategw_static_routes
Success
548.95
test_privategw_acl.py

test_03_vpc_privategw_restart_vpc_cleanup
Success
615.89
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Success
439.63
test_privategw_acl.py

test_01_vpc_privategw_acl
Success
93.97
test_privategw_acl.py

test_01_primary_storage_nfs
Success
36.00
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.20
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
15.74
test_portable_publicip.py

test_isolate_network_password_server
Success
91.40
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.18
test_over_provisioning.py

test_extendPhysicalNetworkVlan
Success
15.35
test_non_contigiousvlan.py

test_01_nic
Success
779.07
test_nic.py

test_releaseIP
Success
286.51
test_network.py

test_reboot_router
Success
427.27
test_network.py

test_public_ip_user_account
Success
10.50
test_network.py

test_public_ip_admin_account
Success
40.40
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
67.01
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.93
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
124.44
test_network.py

test_delete_account
Success
306.67
test_network.py

test_02_port_fwd_on_non_src_nat
Success
56.02
test_network.py

test_01_port_fwd_on_src_nat
Success
112.17
test_network.py

test_nic_secondaryip_add_remove
Success
224.64
test_multipleips_per_nic.py

login_test_saml_user
Success
25.72
test_login.py

test_assign_and_removal_lb
Success
134.36
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
187.42
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
188.24
test_loadbalance.py

test_07_list_default_iso
Success
0.12
test_iso.py

test_05_iso_permissions
Success
0.11
test_iso.py

test_04_extract_Iso
Success
5.21
test_iso.py

test_03_delete_iso
Success
95.29
test_iso.py

test_02_edit_iso
Success
0.19
test_iso.py

test_01_create_iso
Success
22.65
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
260.52
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
212.78
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
580.48
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
497.12
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.53
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.14
test_global_settings.py

test_04_create_fat_type_disk_offering
Success
0.27
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.08
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.18
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.20
test_disk_offerings.py

test_01_create_disk_offering
Success
0.22
test_disk_offerings.py

test_deployvm_userdispersing
Success
21.22
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
51.58
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
56.41
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
10.46
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
55.95
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
8.89
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
7.21
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
235.19
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
266.15
test_deploy_vm_iso.py

test_DeployVmAntiAffinityGroup
Success
91.78
test_affinity_groups.py

test_03_delete_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_02_revert_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_01_test_vm_volume_snapshot
Skipped
0.00
test_vm_snapshots.py

test_01_create_vm_snapshots
Skipped
0.00
test_vm_snapshots.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_01_scale_vm
Skipped
0.00
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.04
test_primary_storage.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
0.01
test_deploy_vgpu_enabled_vm.py

Tests LGTM.

Code LGTM

authors: abhinandanprateek blueorangutan borisstoyanov cloudmonger jburwell murali-reddy nathanejohnson rhtyd


text:
Cleanup awsapi leftovers

Just a couple of things that were still in the code :)

I swear I did remove them, probably some merge issue. LGTM, merging now.

If if comes back we will just kill it again :)

authors: bhaisaab rsafonseca


text:
ClassUtils.getBaseClasses(desiredBase, packageName)

Hello,
I have been using a method in my company's (avadasoftware.com) code base that I created for getting a list of classes that implement an interface. These classes reside in a particular package that I query.  I use this method in my code base in the following manner:
List<FooInterface> fooClasses = ClassUtils.getBaseClasses(FooInterface.class, "com.avada.foo")
I would like to put this in Apache's ClassUtils to move this generic code out of our code base and into what I believe is a more appropriate place.  I'm striving to replace util/generic code in our code base with 3rd party libs as I believe it will be better tested/supported/enhanced/etc... in the OSS world.
I have written several tests that are passing to test exceptional behavior and also normal path behavior.

Just checked the Travis CI build and it failed because of an import of java.lang.IllegalArgumentException at the top which is not necessary...

Coverage increased (+0.008%) to 95.26% when pulling 4233117365c74a502a576c029eebf5c18bbe1f8b on zmacomber:master into 69e843890c09861a168c6fe77d63fc72f0c73195 on apache:master.

Should we overload getBaseClasses with the following:

ClassUtils.getBaseClasses(final Class<T> desiredBase, final String packageName, ClassLoader classLoader)
ClassUtils.getBaseClasses(final Class<T> desiredBase, final Package package)
?

Should we overload getBaseClasses with the following:

ClassUtils.getBaseClasses(final Class<T> desiredBase, final String packageName, ClassLoader classLoader)
ClassUtils.getBaseClasses(final Class<T> desiredBase, final Package package)
?

I have implemented this

Have not heard anything on this PR for 2 months.  Should I just close this PR?  This doesn't seem like the way open source should work with such a long time for feedback.  I'm fine with feedback like, "This is not going to fit in our codebase".

authors: aaabramov coveralls zmacomber

repo: apache/commons-lang


text:
Enabling power support for apache/cxf

Description: To Enable ppc64le support for apache/cxf.
Test Result:
ubuntu@powervm17-1:~$ mvn -version
Apache Maven 3.9.6 (bc0240f3c744dd6b6ec2920b3cd08dcc295161ae)
Maven home: /opt/apache-maven-3.9.6
Java version: 17.0.10, vendor: Eclipse Adoptium, runtime: /usr/lib/jvm/jdk-17.0.10+7
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.15.0-122-generic", arch: "ppc64le", family: "unix"
ubuntu@powervm17-1:~$ java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)

CXF_build_success_logs.txt

Is this another case of the certificate needing to be updated?
The test case is to ensure the wss40 cert is valid.

Duplicate of https://github.com/apache/cxf/pull/1787 ?

I pulled fixes-4.0.x branch, and ran a build on my POWER9 host with Adoptium 17 - the build was successful with all tests passing. Checking main branch now.

Main branch also completed a full build with tests successfully.

Closing this PR as the issue is already resolved in this Jira ticket

authors: anushkajuli jgoodyear reta

repo: apache/cxf


text:
doc: Add backward compatibility check before release

Brief Summary of The Change

doc: Add backward compatibility check before release
Pull Request checklist
I confirm that the following checklist has been completed.

[x] Add/Change test cases for the changes.
[x] Add/Change the related documentation, should also change docs/source/config.rst when you change file default_config.yaml.
[x] (Optional) Add your change to UPDATING.md when it is an incompatible change.

Codecov Report

Merging #86 (189aa48) into main (2ef725e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #86   +/-   ##
=======================================
  Coverage   90.75%   90.75%
=======================================
  Files          63       63
  Lines        2314     2314
=======================================
  Hits         2100     2100
  Misses        214      214

Flag
Coverage Δ

unittests
90.75% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter zhongjiajie

repo: apache/dolphinscheduler-sdk-python


text:
[Bug] [DataSource] can not load datasource of  HANA

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

What you expected to happen
att
How to reproduce
att
Anything else
No response
Version
dev
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

close by https://github.com/apache/dolphinscheduler/pull/15260

authors: fuchanghai zhongjiajie

repo: apache/dolphinscheduler


text:
[Feature][Task]Task plugin spi

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Codecov Report

:exclamation: No coverage uploaded for pull request base (task-plugin@4715de1). Click here to learn what that means.
The diff coverage is n/a.

@@              Coverage Diff               @@
##             task-plugin    #6116   +/-   ##
==============================================
  Coverage               ?   42.01%
  Complexity             ?     3528
==============================================
  Files                  ?      660
  Lines                  ?    25888
  Branches               ?     2850
==============================================
  Hits                   ?    10877
  Misses                 ?    13990
  Partials               ?     1021

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4715de1...30e1551. Read the comment docs.

authors: codecov-commenter zhuangchong


text:
[Fix-7236][common] Fix bug of the host cannot be recognized when the endpoint uses the domain name

Purpose of the pull request
fix: #7236
Brief change log

Upgrade hadoop version to 2.8.5
Added fs.s3a.path.style.access configuration, default false
Contains the jar packages and related licenses that hadoop 2.8.5 depends on

Verify this pull request
There are no test cases for now. Our company's internal dolphin scheduler version also upgraded hadoop to 2.8.5, which solved this problem and ran normally after going online.

@caishunfeng @BaoLiang
Help take a look at the submission. The hadoop official has revised this problem (https://issues.apache.org/jira/browse/HADOOP-12963), to solve this problem, you need to upgrade the hadoop version to 2.8.x. Considering the usage, the hadoop 2.8.5 version was chosen
帮忙看一下提交。 hadoop官方已修改此问题(),解决这个问题需要把hadoop版本升级到2.8.x。考虑到使用量，选择了hadoop 2.8.5版本

Purpose of the pull request
fix bug：#7236
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Please describe the brief change log and any test for this pr.

brief change
I added brief change. There are no test cases for now, but the relevant code modifications have passed the internal test of our company and have been launched.

I had approve to run the CI test.

There is a certain conflict between this pr and #8367. @caishunfeng

There is a certain conflict between this pr and #8367. @caishunfeng

@SbloodyS Thanks your feedback, but the #8367 is about COMPLEMENT_DATA bugfix, and this pr is about s3a, it seems no conflict.

SonarCloud Quality Gate failed.
  29 Bugs   0 Vulnerabilities   9 Security Hotspots   462 Code Smells
 1.2% Coverage  0.0% Duplication

@liqiquan Please check the sonar and CI check.

There is a certain conflict between this pr and #8367. @caishunfeng

@SbloodyS Thanks your feedback, but the #8367 is about COMPLEMENT_DATA bugfix, and this pr is about s3a, it seems no conflict.

Sorry for my mistake. It's #8637.

Sorry for my mistake. It's #8637.

Thanks, they all about s3a. I am not familiar about s3a, so PTAL @nobolity @SbloodyS

SonarCloud Quality Gate failed.
  29 Bugs   0 Vulnerabilities   9 Security Hotspots   462 Code Smells
 1.2% Coverage  0.0% Duplication

@liqiquan Please check the sonar and CI check.

SonarCloud Quality Gate failed.
  29 Bugs   0 Vulnerabilities   9 Security Hotspots   462 Code Smells
 1.2% Coverage  0.0% Duplication

@liqiquan Please check the sonar and CI check.
The error reported by sonar is basically the problem of the js code of the dolphinscheduler-ui module, but my submission does not involve this module

@caishunfeng
Ask a question, the code I submitted does not involve the dolphinscheduler-ui module, why is there a large number of errors in the js code of the dolphinscheduler-ui module in the sonar report?

@caishunfeng Ask a question, the code I submitted does not involve the dolphinscheduler-ui module, why is there a large number of errors in the js code of the dolphinscheduler-ui module in the sonar report?

It seems the sonar will check all the project files, not only the change files. You can skip the UI errors if you don't update the UI module.
But you have to check the CI build fail. see https://dolphinscheduler.apache.org/en-us/community/development/DS-License.html.
PTAL again @nobolity @lenboo

@caishunfeng
Can you help to run the ci test again?

@caishunfeng Can you help to run the ci test again?

Of course, I had approved it.

Codecov Report

Merging #8547 (4cd7232) into dev (9c162c8) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##                dev    #8547      +/-   ##
============================================
- Coverage     45.29%   45.27%   -0.02%
- Complexity     4073     4081       +8
============================================
  Files           700      702       +2
  Lines         27050    27187     +137
  Branches       2894     2908      +14
============================================
+ Hits          12251    12309      +58
- Misses        13645    13716      +71
- Partials       1154     1162       +8

Impacted Files
Coverage Δ

.../org/apache/dolphinscheduler/common/Constants.java
78.94% <ø> (ø)

...che/dolphinscheduler/common/utils/HadoopUtils.java
28.28% <0.00%> (-0.15%)
:arrow_down:

...org/apache/dolphinscheduler/alert/AlertSender.java
74.03% <0.00%> (-14.43%)
:arrow_down:

.../remote/command/alert/AlertSendRequestCommand.java
92.00% <0.00%> (-8.00%)
:arrow_down:

...hinscheduler/common/utils/TaskParametersUtils.java
50.00% <0.00%> (-2.18%)
:arrow_down:

...g/apache/dolphinscheduler/dao/utils/DagHelper.java
45.55% <0.00%> (-1.83%)
:arrow_down:

...e/dolphinscheduler/dao/entity/ProcessInstance.java
66.00% <0.00%> (-1.35%)
:arrow_down:

...nscheduler/api/controller/ResourcesController.java
57.89% <0.00%> (-1.04%)
:arrow_down:

...eduler/api/service/impl/AlertGroupServiceImpl.java
90.80% <0.00%> (-0.87%)
:arrow_down:

...heduler/api/service/impl/ResourcesServiceImpl.java
53.66% <0.00%> (-0.68%)
:arrow_down:

... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c162c8...4cd7232. Read the comment docs.

@caishunfeng
Please help to run the ci test again, thx

authors: SbloodyS caishunfeng codecov-commenter liqiquan


text:
[Fix] [UI Next] Alert instance data form issues

close #9172

Codecov Report

Merging #9258 (cf0a879) into dev (a56e1d2) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                dev    #9258   +/-   ##
=========================================
  Coverage     40.09%   40.09%
  Complexity     4374     4374
=========================================
  Files           820      820
  Lines         32743    32743
  Branches       3640     3640
=========================================
  Hits          13129    13129
  Misses        18387    18387
  Partials       1227     1227

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c1578a...cf0a879. Read the comment docs.

authors: codecov-commenter devosend


text:
[BugFix](file cache) don't clean clone dir when doing _gc_unused_file_caches

Proposed changes
Issue Number: close #xxx
Previous implementation would clean tablet_path/clone/xxx.dat file even though we didn't set file_cache_max_size_per_disk. The log is as follows:

Problem summary
Describe your changes.
Make the _gc_unused_file_caches skip clone dir.
BTW, i changed one function call which throws unhandled exception due to the clone dir is deleted to use to one noexcept overload.
Checklist(Required)

Does it affect the original behavior:

[x] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[x] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.98 seconds
load time: 443 seconds
storage size: 17178881682 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221111094425_clickbench_pr_44315.html

authors: ByteYue hello-stephen

repo: apache/doris


text:
typo change-function-version

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
add auto analyze test case

Proposed changes
There is the basic auto analyze functional case.

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Depend on other pr:
https://github.com/apache/doris/pull/24026
https://github.com/apache/doris/pull/24098


text:
fix if nullptr do not merge

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
(enhance)(Cooldown)Add bvar to monitor number of the unused rowsets

Proposed changes
Issue Number: close #xxx
In order to monitor whether the cooled rowset has been thoroughly reclaimed, a new boolean variable (bvar) has been introduced to track the count of all unused rowsets. This bvar may not be entirely accurate, as situations like push handlers can add unused rowsets too. However, it provides a rough indication of the reclamation status of cooled rowsets

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
doc modify release verify doc

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
Refactor Remove some unused code in pipeline query engine

Proposed changes

change some api unify
remove some unused code

run buildall

run buildall


text:
[Enhancement] (nereids)implement adminCleanTrashCommand in nereids

What problem does this PR solve?
Issue Number: close #42846
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall

run buildall

run buildall


text:
DRILL-7628: Fix Mongo tests broken after the fix for DRILL-7547

DRILL-7628: Fix Mongo tests broken after the fix for DRILL-7547
Description
Fixed test suite name in pom file, since all mongo tests were skipped during tests run because of this issue. Made changes to create mongo user for tests using a conf file. Added missed TestMongoStoragePluginUsesCredentialsStore to the test suite.
Documentation
NA
Testing
Ran all tests.

@vvysotskyi I think maybe we'll get rid of this test suite logic?
We can make all Mongo tests extend MongoTestBase.class, annotate MongoTestBase.class with @Category({MongoStorageTest.class}) so we can skip or include Mongo tests and that's it, so suite logic will be required.

+1, LGTM

authors: arina-ielchiieva vvysotskyi

repo: apache/drill


text:
[Proposal] Design for a Configurable Index Type

Motivation
Alternative approaches to indexing events in-memory might have benefits in some workloads, or be irrelevant in other workloads.
But the current Druid design allows only one (hard-coded) approach for that end; i.e., incremental-index, specifically OnheapIncrementalIndex.
Other implementations such as the OffheapIncrementalIndex might also be beneficial in some workloads (albeit deprecated).
Furthermore, future approaches such as suggested in PR #10001 (ISSUE #9967), can achieve much better resource utilization than the existing approaches (source).
But for current and future approaches to be implemented, tested, and embedded into Druid, Druid must allow users to select different indexing approaches (as core or as extensions).
The purpose of this issue is to present and discuss a design for a configurable index type.
Afterwhich, we plan to implement the decided (and approved) approach and submit it as a PR.
Proposed changes
We propose the following modifications to Druid design:

Add AppendableIndexBuilder and AppendableIndexSpec interfaces
Add an ingestion knob for tuningConfig
Instantiate an AppendableIndexBuilder using the given spec

These incur many minor changes as depicted in #10335 (WIP).
Below we describe the notable changes.
Add AppendableIndexBuilder and AppendableIndexSpec interfaces
AppendableIndexBuilder will be an abstract class. It will be identical to the existing IncrementalIndex.Builder class, except for the build() method that will be abstract.
Each implementation of IncrementalIndex will have its own Builder.
AppendableIndexSpec will be an extension point (@UnstableApi for now). It will have two methods:
@UnstableApi
public interface AppendableIndexSpec
{
  // Returns a builder of the appendable index.
  AppendableIndexBuilder builder();

  // Returns the default max bytes in memory for this index.
  long getDefaultMaxBytesInMemory();
}

Each appendable index will have its own implementation of AppendableIndexSpec, which will be registered as JsonSubType via a dedicated AppendableIndexModule.
The getDefaultMaxBytesInMemory() is required because different implementations might have different restrictions (e.g., on-heap is limited by the heap size limit and the off-heap is limited by the direct size limit).
Add an ingestion knob for tuningConfig
First, we add a method to the AppenderatorConfig interface: AppendableIndexSpec getAppendableIndexSpec().
Then, we add to each of its implementations a constructor parameter:
@JsonProperty("appendableIndexSpec") @Nullable AppendableIndexSpec appendableIndexSpec

which will be returned by the getAppendableIndexSpec() method.
This will affect all the implementations of AppenderatorConfig: RealtimeTuningConfig, RealtimeAppenderatorTuningConfig, IndexTask.IndexTuningConfig, ParallelIndexTuningConfig, HadoopTuningConfig, SeekableStreamIndexTaskTuningConfig, KafkaIndexTaskTuningConfig, KafkaSupervisorTuningConfig, KinesisIndexTaskTuningConfig, KinesisSupervisorTuningConfig.
The above will introduce a new configuration knob.
For example, the on-heap (default) will be configured as such:
"tuningConfig": {
  "appendableIndexSpec": {
    "type": "onheap"
  }
}

and the off-heap as such (the buffer size is set to 8MB and the cache-size is 1GB):
"tuningConfig": {
  "appendableIndexSpec": {
    "type": "offheap",
    "bufferSize": 8388608,
    "cacheSize": 1073741824
  }
}

Instantiate an AppendableIndexBuilder using the given spec
All places that instantiate IncrementalIndex.Builder, will be modified to instantiate it from
config.getSchema().getTuningConfig().getAppendableIndexSpec().builder()

That is: Sink, IndexGeneratorJob (hadoop).
Rationale
We considered a few alternative choices for this design.

Adding a new AppendableIndex interface
Avoiding using builder factory: “appendableIndexSpec” parameter will be a String rather than a builder factory

Adding a new AppendableIndex interface
AppendableIndex will include all the externally visible API of the current IncrementalIndex.
For example:
public interface AppendableIndex
{
  FactsHolder getFacts();
  boolean canAppendRow();
  String getOutOfRowsReason();
  AppendableIndexAddResult add(InputRow row, boolean skipMaxRowsInMemoryCheck) throws IndexSizeExceededException;
  // etc...
}

The AppendableIndexBuilder.build() will return an AppendableIndex instead of an IncrementalIndex, and all places that refer to the IncrementalIndex will refer to this interface instead.
This will allow developers to choose a completely different implementation but will introduce many small changes in the code.
This can be beneficial, for example, for #10001 which does not need incremental-index to function but it inherits from it to reduce the total diff size.
To minimize diff size, we prefer to avoid this new interface for now, but it should be considered for future work.
Avoiding using builder factory: “appendableIndexSpec” parameter will be a String rather than a builder factory
An alternative design could be adding another member/method to the existing IncrementalIndex.Builder, and adding a “generic” build() method:
// ...

public Builder setIncrementalIndexType(final String incrementalIndexType)
{
  this.incrementalIndexType = incrementalIndexType;
  return this;
}

// ...

public IncrementalIndex build()
{
  switch (incrementalIndexType) {
    case "onheap":
      return buildOnheap();
    case "offheap":
      return buildOffheap();
    case "novel-index":
      return buildNovel();
    default:
      throw new IllegalArgumentException("Unsupported incremental index: " + incrementalIndexType);
  }
}

Then, add the ingestion configuration as a string, rather than a factory class.
This alternative is simpler and is the one that is currently implemented in #10001.
But it does not allow extensions to implement their own appendable-index.
This is especially important with the current effort to support Java-11.
Currently, #10001 only supports Java-8, but it will support Java-11 it in the future.
Since extensions allowed lagging behind this effort (e.g., data sketches), we prefer the design that supports index extensions.
Operational impact
This change will not affect any existing clusters. It will work seamlessly when the appendableIndexSpec parameter is omitted.
Future work

Integration of #10001 with the new design
Add AppendableIndex interface
Similar configuration can be added to queries; e.g., to allow alternative indexing methods in group-by queries (such as in GroupByQueryHelper)

Sorry @liran-funaro, it took a while to get back to this proposal. The proposed changes sound reasonable to me. Having AppendableIndexBuilder in tuningConfig with appendableIndex-specific knobs looks especially useful. However, I'm assuming that those knobs will have default values and users usually don't have to worry about them. Is this right?
Regarding AppendableIndexSpec interface, do you think it's stable enough or do you perhaps need to make some changes in the near future? If it's the later, UnstableApi may be better than making it an ExtensionPoint already.

Similar configuration can be added to queries; e.g., to allow alternative indexing methods in group-by queries (such as in GroupByQueryHelper)

GroupByQueryHelper is used in only groupBy v1 which is deprecated in favor of groupBy v2. I'm not sure how much it's slower compared to groupBy v2 or how much using Oak can help with its performance. You may want to look into concurrent processing in groupBy v2 more closely such as ConcurrentGrouper.

@jihoonson Thank you for taking the time to review this proposal.

I'm assuming that those knobs will have default values and users usually don't have to worry about them. Is this right?

Yes. They will be @Nullable and will be set to the default value in that case. E.g., for OffheapIncrementalIndex.Spec:
    public Spec(
        final @JsonProperty("bufferSize") @Nullable Integer bufferSize,
        final @JsonProperty("cacheSize") @Nullable Integer cacheSize
    )
    {
      this.bufferSize = bufferSize != null && bufferSize > 0 ? bufferSize : DEFAULT_BUFFER_SIZE;
      this.cacheSize = cacheSize != null && cacheSize > this.bufferSize ? cacheSize : DEFAULT_CACHE_SIZE;
      // ...
    }

do you think it's stable enough or do you perhaps need to make some changes in the near future?

Using the @UnstableApi seems like a good idea at this point.

I'm not sure how much it's slower compared to groupBy v2 or how much using Oak can help with its performance. You may want to look into concurrent processing in groupBy v2 more closely such as ConcurrentGrouper.

I'll look into that. Improving query performance is on our agenda, and we considered comparing the groupBy v1 with Oak to the groupBy v2 mechanism. We will report our results in a different issue in the future.

@liran-funaro thanks for your answer.

I'll look into that. Improving query performance is on our agenda, and we considered comparing the groupBy v1 with Oak to the groupBy v2 mechanism. We will report our results in a different issue in the future.

Sounds good. I'm looking forward to seeing your report. :slightly_smiling_face:

@jihoonson Thanks! We hope we will get to evaluate queries in 2021, but for now, ingestion is our main focus.
I updated the proposal as per your suggestion. I also updated the linked WIP PR (#10335).
Do you have any suggestions on how to move this forward?
Do we need more reviewers for the design/PR?

@liran-funaro this proposal has been open for a while and we consider silence as acceptance :slightly_smiling_face:. I think you can go ahead and make the linked PR ready for review.

Great! The PR (#10335) is now ready for review.

Reviewed PR #10335
I suggest to split it into 2 PRs:

all refactoring replacing method/class names etc - this incur changing >100 files but has no logical/operational affect
the changes that actually add the extension point - this will only change few to dozen files and will make it easier for the reviewer to give feedback

Anyway, this change (as one PR or split into 2 PRs) should not affect the normal execution of ingestion/queries; it is only meant to enable future flexibility in supporting external (pluggable) incremental index implementations.
I would suggest to adopt it as is.
Would appreciate if others in the community can take a look and support the PR
@jihoonson @a2l007

@jihoonson @a2l007
As per @Eshcar suggestion, I moved some of the refactorings in this RR to an independent (much smaller) PR.
Please checkout out #10478.

authors: Eshcar jihoonson liran-funaro

repo: apache/druid