text:
Make side inputs a map, rather than embedding the name in the message.

The "local" name only make sense in context.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

RFC: @kennknowles
A downside of this change is that a single PCollection used multiple times as a side input would be listed as multiple inputs. Not sure if we want that.

In fact, I believe you prior change that removed PCollection references from SideInput makes this change mandatory:

Previously, the local names could be different and the validation condition was that the PCollection refs in side inputs were a subset of the PCollection refs in the generic graph inputs.
Now, that cannot be validated. This change makes it much more clear that a subset check on the keys is the new validation condition.

I think the new world after your two PRs is better, more constrained in a good way. I'm not bothered by a PCollection appearing multiple times as a generic graph node input.
Caveat: What I do care about is that it is easy for an SDK to implement positional access to side inputs but I think they could do this by embedding generated tags in the serialized UDF.
We will find out more when we start replacing existing bits with ParDo payloads. So LGTM, but I think we should pause the micro edits until we are using the API in Java, then Python.

Coverage increased (+0.01%) to 69.144% when pulling b2c0f93d0f4d63836a79956d75b6dd100e9138ba on robertwb:runner-protos into 16736a6b612ba38b19694ae2a23dea7f5fab1563 on apache:master.

I just thought of a big downside to this: likely the local name will be used in the side input (state) protocol (and in particular the cache). Not allowing multiple names would mean the SDK would have to keep track of which ones actually represent the same collection.
(As an aside, it seems edits are easier before we have code using them...)

I don't understand what you mean by "not allowing multiple names". If a collection is side input twice, it will appears under two different names in the generic graph inputs and also under the same different names in the side inputs map. In general, the two specs in the side input map will be different and cached separately by the SDK harness post-ViewFn.
If you are suggesting adding a pre-ViewFn cache, and a desire to share whenever it has the same (PCollection, materialization) key, that seems pretty easy to implement too.
Perhaps you can say more to make me more worried?

Yes, I was thinking a pre-ViewFn cache. It's a question of imposing the
implementation of deduplicating the cache on something other than the local
name on each SDK. That being said, it could be that this is an uncommon
(and easy to implement if need be) improvement, so still worth doing this
change (I'm leaning that way) but wanted to call this out.
On Tue, Feb 28, 2017 at 10:35 AM, Kenn Knowles notifications@github.com
wrote:

I don't understand what you mean by "not allowing multiple names". If a
collection is side input twice, it will appears under two different names
in the generic graph inputs and also under the same different names in the
side inputs map. In general, the two specs in the side input map will be
different and cached separately by the SDK harness post-ViewFn.
If you are suggesting adding a pre-ViewFn cache, and a desire to share
whenever it has the same (PCollection, materialization) key, that seems
pretty easy to implement too.
Perhaps you can say more to make me more worried?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/2124#issuecomment-283124270, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAdqgevZk66e32AdXhJCERCu90ORHcVOks5rhGjWgaJpZM4MN4KZ
.

OK, still not worried :-)
I envision the SDK focusing on post-ViewFn cache since then it will have constructed a language-appropriate data structure that could actually be potentially expensive - today it is dramatically the case. So assuming smart decisions there, a pre-ViewFn cache might not actually have much benefit except in just this rare case where the same PCollection is viewed two ways. I wouldn't characterize it as deduplication, since the two different accesses to the same PCollection could easily have different enough access patterns to violate caching assumptions.
Out of scope here is that the runner's worker will likely manage a pre-ViewFn protocol including caching that is tightly coupled with the runner's private implementation of the materialization and possibly much more complex.
And if this turns out to be a wrong view of the future, I really thing the adaptation code will be quite trivial, ~zero in the scheme of things. Seems worth it for the clarity, though I don't care strongly either way; just a smidge more than style.

authors: coveralls kennknowles robertwb


text:
Bump runner v2 beam-master to 20240109

Update the beam-master container of runner v2 to beam-master-20240109.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

LGTM. Thanks.

source: gharchive/pull-request

authors: chamikaramj shunping

repo: apache/beam

license_type: permissive

license_source: github-api


text:
Implement LengthPrefixCoder.to_type_hint

LengthPrefixCoder doesn't define to_type_hint even though it seems pretty well defined. I've implemented it by just inferring it from its value coder.
Motivating use case - we have some debug tools that infer the types of pcollections based on to_type_hint. The LPCoder is used in a lot of places.
Bonus/unrelated change - I've pulled out the GitHub issue referenced in the NotImplementedError raised by to_type_hint. Every time I run into this error, I check the github issue and get more confused. It doesn't seem useful to refer to it in the error message, but does still seem relevant, so I pulled it out into a comment.
Testing Done
I haven't actually run the new unit test I wrote since I'm struggling to get my python sdk dev environment up (I've posted on the dev@ mailing list). Hoping the GH Action can just do it for me :)

R: @robertwb

Bump @robertwb (lmk if I should just use the regular reviewer-assignment system going forward)

Not sure who to @ here to keep this moving. @damccorm ?

I'll merge later after checks pass

I'll merge later after checks pass

Thanks for all the reviews! I really appreciate it!

Uh oh, looks like a bad rebase

I've just created a duplicate PR without the rebase issues
https://github.com/apache/beam/pull/30857

SGTM

authors: damccorm hjtran


text:
Do Not Merge, [BEAM-4687] Test Jenkins JIRA Plugin

Please add a meaningful description for your change here

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Spark

Go

---
---
---
---
---

Java

Python

---

---
---
---

Run Dependency Check


text:
[BEAM-4007] Fix TODO style in typehints.py

This change fixes the TODO style for a few lines introduced by https://github.com/apache/beam/pull/5337.

R: @RobbeSneyders

LGTM, thanks.

authors: RobbeSneyders charlesccychen


text:
[BEAM-6050] Use correct type on @ProcessElement method for SplittableDoFns

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Run Spark ValidatesRunner

R: @chamikaramj

Run Spark ValidatesRunner

Roll forward fix for ValidatesRunner tests to unbreak failing post commits.


text:
[BEAM-7859] set SDK worker parallelism to 1 in word count test

This does not affect the Flink runner because of its default behavior:
https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobInvoker.java#L73-L75
However, I would like to make this explicit, because work must happen in a shared filesystem for this job to succeed.
R: @angoenka
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Python PreCommit

Run Java PreCommit

Run Java PreCommit

Ping @angoenka -- this should be a no-op


text:
Support backup request policy

What problem does this PR solve?
Issue Number: resolve #2668 , resolve #2695
Problem Summary:
What is changed and the side effects?
Changed:
Side effects:

Performance effects(性能影响):

Breaking backward compatibility(向后兼容性):

Check List:

Please make sure your changes are compilable(请确保你的更改可以通过编译).
When providing us with a new feature, it is best to add related tests(如果你向我们增加一个新的功能, 请添加相关测试).
Please follow Contributor Covenant Code of Conduct.(请遵循贡献者准则).

LGTM

LGTM

authors: chenBright wwbmmm yanglimingcn

repo: apache/brpc


text:
Remote Execution: Don't ignore stdout and stderr

See original issue on GitLab
In GitLab by [Gitlab user @juergbi] on Nov 29, 2018, 17:58
The remote execution client currently ignores stdout and stderr in the ActionResult.

In GitLab by [Gitlab user @juergbi] on Nov 30, 2018, 14:31
I've pushed an old patch to partially solve this to the juerg/remote-execution-stdout-stderr branch. It does not yet support retrieving stdout and stderr from CAS (in case the server did not embed them). However, it may be useful as a starting point.

In GitLab by [Gitlab user @juergbi] on Mar 2, 2019, 11:49
mentioned in commit 555e34580f14c95c3e784287e51a5786c8d90628

In GitLab by [Gitlab user @juergbi] on Mar 2, 2019, 11:54
mentioned in merge request !1192

In GitLab by [Gitlab user @juergbi] on Mar 6, 2019, 10:31
mentioned in commit f7461e17575f6cdb0dad1319b3b999e9d29dfcac

In GitLab by [Gitlab user @juergbi] on Mar 8, 2019, 14:55
mentioned in commit 8c6666391abf74a1c9aaf29b21ed1c2f63739e79

In GitLab by [Gitlab user @martinblanchard] on Mar 11, 2019, 14:38
I think BuildStream should also not ignore the ExecuteResponse.message. If configured to do so, the BuildGrid and buildbarn server implementations will embed a link to a web report for the build it just perform in that field. Would be nice to print if as a INFO log record or something on BuildStream side (at least on build errors).

source: gharchive/issue

repo: apache/buildstream


text:
Tests are broken on CentOS

See original issue on GitLab
In GitLab by [Gitlab user @BenjaminSchubert] on Dec 21, 2018, 14:16
Summary
Trying to add a new runner which is CentOS, we can see that multiple tests fail.
https://gitlab.com/BuildStream/buildstream/-/jobs/137741349 is the job failing when trying to add a CentOS runner
Steps to reproduce
Run tests on CentOS
What is the current bug behavior?
tests fail
What is the expected correct behavior?
they should succeed
This is done when:

[ ] the tests pass on CentOS
[ ] We run the tests on CentOS everytime to prevent regression

In GitLab by [Gitlab user @cs-shadow] on Dec 21, 2018, 14:33
It is also worth mentioning that these failures happen in the same way on Red Hat EL 7.5  as well, which is the upstream distro of CentOS.

In GitLab by [Gitlab user @tpollard] on Dec 21, 2018, 14:56
As mentioned in #buildstream it looks like most of the failures are coming from git, leading to an assumption that it could be down to the host git config/version and the recent changes made to the git plugin.

In GitLab by [Gitlab user @cs-shadow] on Jan 9, 2019, 21:21
Related MR that demonstrates this failure: https://gitlab.com/BuildStream/buildstream/merge_requests/1053.
Once we fix the issue, this MR would ideally start passing.

In GitLab by [Gitlab user @cs-shadow] on Jan 10, 2019, 10:20
The git version is 1.8.3.1
docker run --rm -it buildstream/testsuite-centos:7.6-917af42e-c3153dea git --version
git version 1.8.3.1

In GitLab by [Gitlab user @danielsilverstone-ct] on Jan 10, 2019, 10:45
Regardless of whether or not it ought to have an effect, that version of git is over five years old, so there'll be a LOT to trawl through if we were to try and determine changes in behaviour from the git side.  As such, we probably need to break down some of the failing tests more carefully.

In GitLab by [Gitlab user @tpollard] on Jan 11, 2019, 09:24
OOI were the tests working before the changes to the git plugin (how we interact with tags it seems) landed?

In GitLab by [Gitlab user @BenjaminSchubert] on Jan 11, 2019, 09:26
I hadn't have tested in ~1 month, but I had a time were everything was working yes

In GitLab by [Gitlab user @tpollard] on Jan 11, 2019, 16:44
For what it's worth, using the centos dockerfile I've been able to see the same tests failures. Sadly I've not had enough time to poke at the git tag / describe changes that have caused this in buildtstream as of yet

In GitLab by [Gitlab user @richardmaw-codethink] on Jan 14, 2019, 15:52
https://gitlab.com/BuildStream/buildstream/commit/719ade06cdcf5af43e95b0cc06b091b14bc9c1b3 will fix that problem, however there are other problems with the old version of git that make the tests continue to fail.
https://gitlab.com/BuildStream/buildstream/blob/719ade06cdcf5af43e95b0cc06b091b14bc9c1b3/tests/sources/git.py#L872 will throw an exception because the old version of git won't have --first-parent.
This could be handled by parameterising the test so it's not run if your version of git is too old. This isn't ideal, but not a problem if other platforms are proving the history is there.
https://gitlab.com/BuildStream/buildstream/blob/719ade06cdcf5af43e95b0cc06b091b14bc9c1b3/tests/sources/git.py#L876 however demonstrates that it is not, since the git source plugin will treat the absence of --first-parent as a non-fatal error, it won't notice that it should include tag1, so the test fails.
Ironically this has also proven that the current implementation is also including commit history that it does not need to, since the commit that the missing tag would point to is still present, so the number of commits remains the same.

Given the lack of --first-parent isn't treated as a fatal error, and instead produces a different cache key I'm assuming this behaviour is intentional, so I'm going to amend the test suite to check whether --first-parent is missing and deal with tag1 not being there.

In GitLab by [Gitlab user @BenjaminSchubert] on Jan 14, 2019, 16:02

Given the lack of --first-parent isn't treated as a fatal error, and instead produces a different cache key I'm assuming this behaviour is intentional, so I'm going to amend the test suite to check whether --first-parent is missing and deal with tag1 not being there.

I'm not sure we should have a different key there. I might be misunderstanding or this might be an oversight but it might be good to check the intent with the author of this part :)

In GitLab by [Gitlab user @richardmaw-codethink] on Jan 14, 2019, 16:38
mentioned in merge request !1066

In GitLab by [Gitlab user @richardmaw-codethink] on Jan 14, 2019, 16:39
[Gitlab user @valentindavid] we could do with your input, since it determines whether to accept the change, or require a newer version of git, and not support that version of CentOS.

In GitLab by [Gitlab user @valentindavid] on Jan 15, 2019, 10:24
The reason we ignore errors originally is because there might be no tag using first parents. So the silent error here was not intended for that purpose.
However, this code happens only during tracking. So reproduction is not an issue here. If you track with the wrong git version, you still build what is explicitly saved in the element or in project.refs. What we build is exactly what is in the element and project.refs.
So yes. We can just ignore those tests when git version detected to be too old.
Other solutions:

Fail track when git version is too old.
Warn when tracking and git version is too old.
Disable git describe feature when git version is too old.
Traverse manually the graph to find the tags.

Minimum version for --first-parent is 1.8.4 from August 23rd, 2013. (actually since rc0).
https://github.com/git/git/commit/e00dd1e9485c50f202cc97dfae19d510e108b565

In GitLab by [Gitlab user @valentindavid] on Jan 15, 2019, 12:17
It was not very obvious to understand. The behavior of older and newer git should be the same during build.
Coming back to this graph:
  F1 ← refs/tags/tag1
 /
F3  F2 ← refs/tags/tag2
 \  /
  M ← master, HEAD

F1 is included because git rev-list --boundary lists F1 as a boundary. For the moment we include all boundary commits.
This has nothing to do with the fact there is tag on it. This is in the output of git rev-list --boundary tag2..master.
[Gitlab user @richardmaw]-codethink is right, using --ancestry-path seems to fix issue. The good news is that it works with git rev-list.

In GitLab by [Gitlab user @valentindavid] on Jan 15, 2019, 12:25
mentioned in commit 69c11a321853c7d40f89f89322568277fad4da08

In GitLab by [Gitlab user @valentindavid] on Jan 15, 2019, 12:26
mentioned in merge request !1069

In GitLab by [Gitlab user @valentindavid] on Jan 15, 2019, 13:03
mentioned in commit f379b131bdc6c7cc47614ac92c490b9f14d107bf

In GitLab by [Gitlab user @valentindavid] on Jan 15, 2019, 13:04
[Gitlab user @richardmaw]-codethink I have fixed the issue with too many commits included in !1069. I have also fixed another bug I came across while testing. We do not track properly when the HEAD is tagged. Could you review it?

In GitLab by [Gitlab user @richardmaw-codethink] on Jan 15, 2019, 16:25
[Gitlab user @valentindavid] I have reviewed.
There's other issues causing tests to break on CentOS, so I'm going to work through them and list them as I go before putting together an MR.

In GitLab by [Gitlab user @valentindavid] on Jan 15, 2019, 16:31
mentioned in commit e4788ddee68a550ee2e06be1d6ee58205bf2e753

In GitLab by [Gitlab user @valentindavid] on Jan 16, 2019, 10:05
mentioned in commit 0eac40085e5c8b8807002759f597f8e19a9745b2

In GitLab by [Gitlab user @richardmaw-codethink] on Jan 18, 2019, 15:36
mentioned in merge request !1085

In GitLab by [Gitlab user @richardmaw-codethink] on Jan 18, 2019, 15:38
I submitted https://gitlab.com/BuildStream/buildstream/merge_requests/1085 to make the test suite pass, and included https://gitlab.com/BuildStream/buildstream/merge_requests/1053 to test whether it works generally or just worked in my own container.

In GitLab by [Gitlab user @valentindavid] on Jan 21, 2019, 11:47
closed via merge request !1085

In GitLab by [Gitlab user @valentindavid] on Jan 21, 2019, 11:47
mentioned in commit 33782865c019a993fde5f725b350262e7735c53f


text:
[DRAFT]fix #924 kamelets heve wrong type for properteries.

added validation of propery types to match json schema allowed values.

Converted to draft

depends on https://github.com/apache/camel-kamelets/pull/928

Needs to double check the google authentication.

Could this be rebased or start from scratch again?

authors: oscerd valdar

repo: apache/camel-kamelets


text:
Filtering the component / kamelet dialog should update counter

When filtering then the counter (123 and 308) should update with how many matches the filter.
For example typing xml then there is 9 components and 2 kamelets. So it would be good if the counters is updated so you quickly can see this.

Fixed https://github.com/apache/camel-karavan/pull/517

authors: davsclaus mgubaidullin

repo: apache/camel-karavan


text:
openstack: added neutron port, subnet and nova flavor tests #1943

[ ] An issue should be filed for the change unless this is a trivial change (fixing a typo or similar). One issue should ideally be fixed by not more than one commit and the other way round, each commit should fix just one issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful and properly spelled subject line and body. Copying the title of the associated issue is typically enough. Please include the issue number in the commit message prefixed by #.
[ ] The pull request description should explain what the pull request does, how, and why. If the info is available in the associated issue or some other external document, a link is enough.
[ ] Phrases like Fix # or Fixes # will auto-close the named issue upon merging the pull request. Using them is typically a good idea.
[ ] Please run mvn process-resources -Pformat (and amend the changes if necessary) before sending the pull request.
[ ] Contributor guide is your good friend: https://camel.apache.org/camel-quarkus/latest/contributor-guide.html

Great work, thanks @aldettinger !

authors: aldettinger ppalaga

repo: apache/camel-quarkus


text:
Randomize http test port

Fixes #553

ok to test

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/camel-quarkus-pr/493/

authors: asf-ci lburgazzoli


text:
[camel-main] fixes #6492 - removed DoubleRoutesPerClassTest

fixes https://github.com/apache/camel-quarkus/issues/6492
(Scenario is invalid since 4.9.0, the use case is deprecated in 3.15.,x - https://github.com/apache/camel-quarkus/commit/08923cb5c4e8f2ca80ae190b63b1da66b56fcebc)

@JiriOndrusek is this still relevant? If so, you can rebase on main.

I'll check the validity tomorrow

We deprecated Lifecycle.PER_CLASS in 3.16, therefore removal of the tests is valid from my POV. I rebased the change into main branch. (see the commit)

authors: JiriOndrusek jamesnetherton


text:
CAMEL-11345: Remove dependency on spring-core from gRPC component

This is PR on top of the camel-2.19.1

Is there something on my end, but just got this compile error on the 2.19.x branch
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project camel-grpc: Compilation failure: Compilation failure:
[ERROR] /Users/davsclaus/workspace/camel/components/camel-grpc/src/main/java/org/apache/camel/component/grpc/GrpcUtils.java:[62,56] error: cannot find symbol
[ERROR]   symbol:   method findMethod(Class,String,Class[])
[ERROR]   location: class ReflectionHelper
[ERROR] /Users/davsclaus/workspace/camel/components/camel-grpc/src/main/java/org/apache/camel/component/grpc/GrpcUtils.java:[78,40] error: cannot find symbol
[ERROR]   symbol:   method findMethod(Class<CAP#1>,String,Class[])
[ERROR]   location: class ReflectionHelper
[ERROR]   where CAP#1 is a fresh type-variable:
[ERROR]     CAP#1 extends Object from capture of ? extends Object
[ERROR] /Users/davsclaus/workspace/camel/components/camel-grpc/src/main/java/org/apache/camel/component/grpc/GrpcUtils.java:[102,40] error: cannot find symbol
[ERROR]   symbol:   method findMethod(Class<CAP#1>,String,Class[])
[ERROR]   location: class ReflectionHelper
[ERROR]   where CAP#1 is a fresh type-variable:
[ERROR]     CAP#1 extends Object from capture of ? extends Object
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

with this PR merged

I have not rebuild 2.19.x branch but it pulled down the ASF SNAPSHOT JARs.

Yes that was the problem, after rebuild locally it works.

Thanks for the PR it has been merged. Do you mind closing this?

Thanks, PR closed

authors: davsclaus dmvolod

repo: apache/camel


text:
Update camel-fhir Karaf feature

Thanks!

Go for it

@oscerd looks like I don't have the rights? Can't see the merge button

Can't you push directly on the git repo? Did you add your account in gitbox for Apache Camel project?

You ought to be able to work straight with the github repo, my origin is
[remote "origin"]
        url = https://github.com/apache/camel.git

And there is a green button here "squash and merge" you can do for PRs.
Also mind sometimes to backport to older branches which must be done manually via git commands.

@davsclaus @oscerd I don't see the squash and merge button here but I'll try pushing directly to master

I get the following when trying to push to master
remote: Permission to apache/camel.git denied to johnpoth. fatal: unable to access 'https://github.com/apache/camel.git/': The requested URL returned error: 403

For the moment I merged manually, but we need to find out what's the problem. Thanks for the PR

I agree, thanks @oscerd !

authors: davsclaus johnpoth oscerd


text:
CAMEL-13191: Fix Regex Pattern to hide passwords in URI

Backport from 3.x to 2.22.x

Thanks, merged

authors: drmaniac oscerd


text:
CAMEL-15056: camel-main: provide a way to bring a custom engine to control shutdown logic

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] If you're unsure, you can format the pull request title like [CAMEL-XXX] Fixes bug in camel-file component, where you replace CAMEL-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean install -Psourcecheck in your module with source check enabled to make sure basic checks pass and there are no checkstyle violations. A more thorough check will be performed on your pull request automatically.
Below are the contribution guidelines:
https://github.com/apache/camel/blob/master/CONTRIBUTING.md

@davsclaus here a small POC to provide a custom shut-down strategy, mind having a look ?

After merging this PR we may want to compile camel-example and camel-spring-boot in case something is affected.

After merging this PR we may want to compile camel-example and camel-spring-boot in case something is affected.

I'm on it

@davsclaus the spring boot side is also done https://github.com/apache/camel-spring-boot/pull/69

authors: davsclaus lburgazzoli


text:
[CAMEL-16907] camel-jt400 program call usability and doc improvements

Added usability improvements to the program call support as documented in the referenced JIRA issue.
Also, snuck in some doc improvements:

Stopped referring to it as "remote program call" as we have a lot of people running Camel on IBM i now
Removed the 'type' path parameter (it's not required since the type is inferred from the path)
added a more practical program call example

Converting to draft to add doc to more explicitly address concerns in this JIRA ticket


text:
[HOTFIX] Fix JVM crash in search mode

Problem :
In search mode handler there is no task id set so memory manager try to clear the memory with old task id which is set during loading. So it crashes JVM.
Solution :
Always use unique task id for each request so that memory manager can clear the leftover blocks . Set the taskid in SearchRequestHandler
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[ ] Any interfaces changed?

[ ] Any backward compatibility impacted?

[ ] Document update required?

[ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5427/

LGTM

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4261/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4573/

authors: CarbonDataQA jackylk ravipesala

repo: apache/carbondata


text:
[ISSUE-4313] Add @Override for override method of processing module

Why is this PR needed?
There are some code smells in processing module
For example
1.Didn't add @OverRide for override method
2.magic number
https://github.com/apache/carbondata/issues/4313
What changes were proposed in this PR?
 Add @Override for override method of processing module

Does this PR introduce any user interface change?

No

Is any new testcase added?

No, only Add @Override for override method of processing module

Build Failed  with Spark 2.3.4, Please check CI http://159.138.8.58:12602/job/ApacheCarbonPRBuilder2.3/6383/

Build Failed  with Spark 2.4.5, Please check CI http://159.138.8.58:12602/job/ApacheCarbon_PR_Builder_2.4.5/4676/

Build Failed  with Spark 3.1, Please check CI http://159.138.8.58:12602/job/ApacheCarbon_PR_Builder_3.1/768/

LGTM


text:
CLIMATE-749 - Changing temporal_resolution key in CLI

This is a follow up of the previous ticket, CLIMATE-732. CLI now uses correct keys when calling dataset_processor.temporal_rebin.

Merged build triggered. Test Failed.

Merged build started. Test Failed.

Merged build finished. Test Failed.

authors: OCWJenkins huikyole

repo: apache/climate

license_source: bigquery


text:
Cloudstack management v4.12 does not start after trying to upgrade v4.13

I cannot start version 4.12 cloudstack-management any more.
It gives error;
2019-10-08 01:00:15,775 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:null) (logid:) Retrieving keys from RootCAProvider
2019-10-08 01:00:15,803 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:null) (logid:) Registering extension [RootCAProvider] in [Ca Providers Registry]
2019-10-08 01:00:15,803 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:null) (logid:) Registered org.apache.cloudstack.ca.provider.RootCAProvider@20e4e27f
2019-10-08 01:00:15,803 INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Configuring CloudStack Components
2019-10-08 01:00:15,828 WARN [o.a.c.s.m.c.ResourceApplicationContext] (main:null) (logid:) Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean '
cloudStackLifeCycle'; nested exception is java.lang.NullPointerException
2019-10-08 01:00:15,829 WARN [o.e.j.w.WebAppContext] (main:null) (logid:) Failed startup of context o.e.j.w.WebAppContext@58fdd99{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is java.lang.NullPointerException
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:186)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:884)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:552)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContext(DefaultModuleDefinitionSet.java:145)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$2.with(DefaultModuleDefinitionSet.java:122)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.loadContexts(DefaultModuleDefinitionSet.java:117)
at org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:79)
at org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:58)
at org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.(CloudStackSpringContext.java:62)
at org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1515)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1477)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:273)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:115)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:133)
at org.eclipse.jetty.server.Server.start(Server.java:418)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:107)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:385)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.cloudstack.ServerDaemon.start(ServerDaemon.java:186)
at org.apache.cloudstack.ServerDaemon.main(ServerDaemon.java:103)
Caused by: java.lang.NullPointerException
at org.apache.cloudstack.utils.security.CertUtils.pemToPublicKey(CertUtils.java:145)
at org.apache.cloudstack.ca.provider.RootCAProvider.loadRootCAKeyPair(RootCAProvider.java:335)
at org.apache.cloudstack.ca.provider.RootCAProvider.setupCA(RootCAProvider.java:381)
... 55 more
2019-10-08 01:00:15,831 INFO [o.e.j.s.h.ContextHandler] (main:null) (logid:) Started o.e.j.s.h.MovedContextHandler@679b62af{/,null,AVAILABLE}
2019-10-08 01:00:15,850 INFO [o.e.j.s.AbstractConnector] (main:null) (logid:) Started ServerConnector@1a1d6a08{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2019-10-08 01:00:15,850 INFO [o.e.j.s.Server] (main:null) (logid:) Started @22604ms

Please consult @rhtyd he will surely help
Sorry if this disrupts your question but I really need assistance. I am unable to add the host to the management server through the UI. They are separate machines in the same network. Please help

did you restore to the DB backup from 4.12 (that you took before trying to upgrade to 4.13) ?
If so, stop mgmt, restore DB, and start 4.12 mgmt - easy as that.
What/why did upgrade fail to 4.13 - did you

have new records added in the guest_os  and/or guest_os_hypervisor tables?
did you register 4.11.3 systemVM template while on 4.12, before trying to upgrade to 4.13?

Hello,
yes I restored DB from 4,12 backup,
but 4.12 mgmt does not start any more, giving error I post
So I also cannot register 4.11.3 systemVM template
BR,
Tarmo
On Thu, Oct 10, 2019 at 1:11 PM Andrija Panic notifications@github.com
wrote:

did you restore to the DB backup from 4.12 (that you took before trying to
upgrade to 4.13) ?
If so, stop mgmt, restore DB, and start 4.12 mgmt - easy as that.
What/why did upgrade fail to 4.13 - did you

have new records added in the guest_os and/or guest_os_hypervisor
tables?
did you register 4.11.3 systemVM template while on 4.12, before
trying to upgrade to 4.13?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/cloudstack/issues/3628?email_source=notifications&email_token=ACGEEIEXVZDTZU3DNFHPVF3QN35VJA5CNFSM4I6Q3CCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA3WB2Y#issuecomment-540500203,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACGEEID56RMZMCL6TLR7ZEDQN35VJANCNFSM4I6Q3CCA
.

@tarmof not sure what exactly is your problem but it has to do with certificate initialisation:
java.lang.NullPointerException
at org.apache.cloudstack.utils.security.CertUtils.pemToPublicKey(CertUtils.java:145)
at org.apache.cloudstack.ca.provider.RootCAProvider.loadRootCAKeyPair(RootCAProvider.java:335)
at org.apache.cloudstack.ca.provider.RootCAProvider.setupCA(RootCAProvider.java:381)

There seems to be an issue with the root ca pub/priv key in the db (not being able to convert the value in the db by the mangement server as a PublicKey instance).

I've seen people just restore the old DB (mysql import) withOUT deleting the "upgraded" DB - so please make sure to:
-delete cloud* DBs

create emtpy cloud and cloud_usage DBs
import a proper mysql backup for both DBs
make sure to start 4.12 binaries

Proper rollback should be possible always.

@tarmof any update on this one, still an issue or should we close it?

I do not have solution, I will try in January
Happy New Year!
BR
Tarmo
On Tue, Dec 31, 2019 at 3:01 PM Andrija Panic notifications@github.com
wrote:

@tarmof https://github.com/tarmof any update on this one, still an
issue or should we close it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/cloudstack/issues/3628?email_source=notifications&email_token=ACGEEICXEKJXUMSDMEJS7BTQ3M7CZA5CNFSM4I6Q3CCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH4F5MY#issuecomment-569925299,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACGEEICBWXBZKBDD2DI7RJLQ3M7CZANCNFSM4I6Q3CCA
.

@tarmof can you re-open if this is still applicable, 4.12 is now a very old release; pl try upgrading to 4.14. Thanks.

authors: DaanHoogland andrijapanicsb gngusen rhtyd tarmof

repo: apache/cloudstack


text:
Disk offering doesn't change when resize operation is performed

ISSUE TYPE
Bug/Enhancement/Improvement request/
COMPONENT NAME
Component: UI/API/Usage
CLOUDSTACK VERSION
Cloudstack version 4.18
SUMMARY
Disk offering doesn't change when the resize operation is performed
https://cloudstack.apache.org/api/apidocs-4.18/apis/resizeVolume.html
Steps to reproduce the issue

Create a data disk with offering 5gb ( Small disk offering)

Click on the Resize

Navigate volume> select the disk and click on Resize option

or execute the resizeVolume api call
https://cloudstack.apache.org/api/apidocs-4.18/apis/resizeVolume.html

Provide the new disk size let's say 10gb

Cloudstack resizes the data disk to 10gb, successfully but doesn't change the disk offering to custom disk offering

Actual behavior
According to docs
http://docs.cloudstack.apache.org/en/4.18.1.0/adminguide/storage.html?highlight=disk resize#resizing-volumes
Additionally, using the resizeVolume API, a data volume can be moved from a static disk offering to a custom disk offering with the size specified. This functionality allows those who might be billing by certain volume sizes or disk offerings to stick to that model, while providing the flexibility to migrate to whatever custom size is necessary.
Even for the resizevolume api call is  disk offering id is not mandatory
https://cloudstack.apache.org/api/apidocs-4.18/apis/resizeVolume.html

Not changing the disk offering id could lead billing issue if the end user considers disk offering for billing
If we take a look at the Database entries the offering id remains the same even after the resize is done, ideally it should change to a custom disk offering
Usage_event table
The offering_id=3 is not  updated when a volume.resize is performed
mysql> use cloud;
mysql> select * from usage_event where resource_id=41 \G;
*************************** 1. row ***************************
           id: 146
         type: VOLUME.CREATE
   account_id: 2
      created: 2023-12-28 05:19:12
      zone_id: 1
  resource_id: 41
resource_name: ty
  offering_id: 3
  template_id: NULL
         size: 5368709120
resource_type: NULL
    processed: 0
 virtual_size: NULL
*************************** 2. row ***************************
           id: 147
         type: VOLUME.RESIZE
   account_id: 2
      created: 2023-12-28 05:19:29
      zone_id: 1
  resource_id: 41
resource_name: ty
  offering_id: 3
  template_id: NULL
         size: 8589934592
resource_type: NULL
    processed: 0
 virtual_size: NULL

mysql> use cloud_usage;
mysql>select * from cloud_usage where usage_type=6

              id: 235
         zone_id: 1
      account_id: 2
       domain_id: 1
     description: Volume Id: 41 usage time (DiskOffering: 3)
   usage_display: 0.499999 Hrs
      usage_type: 6
       raw_usage: 0.49999913573265076
  vm_instance_id: NULL
         vm_name: NULL
     offering_id: 3
     template_id: NULL
        usage_id: 41
            type: NULL
            size: 8589934592
      network_id: NULL
      start_date: 2023-12-28 05:45:14
        end_date: 2023-12-28 06:15:14
    virtual_size: NULL
       cpu_speed: NULL
       cpu_cores: NULL
          memory: NULL
quota_calculated: 0
       is_hidden: 0

mysql> select * from usage_volume where volume_id=41 \G;
*************************** 1. row ***************************
              id: 29
       volume_id: 41
         zone_id: 1
      account_id: 2
       domain_id: 1
disk_offering_id: 3
     template_id: NULL
            size: 5368709120
         created: 2023-12-28 05:19:12
         deleted: 2023-12-28 05:19:29
*************************** 2. row ***************************
              id: 30
       volume_id: 41
         zone_id: 1
      account_id: 2
       domain_id: 1
disk_offering_id: 3
     template_id: NULL
            size: 8589934592
         created: 2023-12-28 05:19:29
         deleted: NULL
2 rows in set (0.00 sec)

Expected behavior
Cloudstack should update the disk offering to custom disk offering in UI and the databases (Cloud and cloud_usag)
mysql> select * from disk_offering where id=6 \G;
*************************** 1. row ***************************
                         id: 6
                       name: Custom
                       uuid: 9cc36337-08a4-4b00-9360-22353fa3503f
               display_text: Custom Disk
                  disk_size: 0
                       tags: NULL
                recreatable: 0
          use_local_storage: 0
                unique_name: Cloud.Com-Custom
                 customized: 1
                    removed: NULL
                    created: 2023-12-26 11:01:13
                   sort_key: 0
           display_offering: 1
            customized_iops: NULL
                   min_iops: NULL
                   max_iops: NULL
            bytes_read_rate: NULL
        bytes_read_rate_max: NULL
 bytes_read_rate_max_length: NULL
           bytes_write_rate: NULL
       bytes_write_rate_max: NULL
bytes_write_rate_max_length: NULL
             iops_read_rate: NULL
         iops_read_rate_max: NULL
  iops_read_rate_max_length: NULL
            iops_write_rate: NULL
        iops_write_rate_max: NULL
 iops_write_rate_max_length: NULL
                      state: Active
              hv_ss_reserve: NULL
                 cache_mode: NULL
          provisioning_type: thin
       disk_size_strictness: 0
               compute_only: 0
                    encrypt: 0

There is no issue observed if change offeringfor volume is performed
https://cloudstack.apache.org/api/apidocs-4.18/apis/changeOfferingForVolume.html

as far as I know, size is just a property of disk offering.
disk offering has other properties, for instance, storage type, QoS, service offering, etc.
I think the disk offering id should not be changed to the id of customized disk offering.
@kiranchavala do you know if there is any issues caused by it ?
cc @kiranchavala @DaanHoogland @sureshanaparti
what's your opinion ?

as far as I know, size is just a property of disk offering. disk offering has other properties, for instance, storage type, QoS, service offering, etc. I think the disk offering id should not be changed to the id of customized disk offering. @kiranchavala do you know if there is any issues caused by it ?

If changing disk size is allowed, this also must have been a property of the diskoffering. If it is not allowed in the offering and it succeeds anyway, i think this is a bug. Disks can also be changed by changing the offering, but I am not sure what the effects of that are. I agree that offering should not implicitly change on resize, only explicitly, if the user asks so.

as far as I know, size is just a property of disk offering. disk offering has other properties, for instance, storage type, QoS, service offering, etc. I think the disk offering id should not be changed to the id of customized disk offering. @kiranchavala do you know if there is any issues caused by it ?

If changing disk size is allowed, this also must have been a property of the diskoffering. If it is not allowed in the offering and it succeeds anyway, i think this is a bug.

but it is not mentioned in @kiranchavala 's description

Disks can also be changed by changing the offering, but I am not sure what the effects of that are. I agree that offering should not implicitly change on resize, only explicitly, if the user asks so.

yes @DaanHoogland
I think this is not a bug.
since resizeVolume API supports diskofferingid parameter, it could be a UI improvement to specify disk offering when resize a volume.

the disk offering selection was present before.
it has been removed in #5008
@harikrishna-patnala do you remember why it is removed ? thanks

@weizhouapache Agree with you this is not a bug, probably we only need to update the document what Kiran is pointing out !!!
Regarding the removal of selection of diskoffering on resize form is to make only single point to change the diskoffering using changeOfferingForVolume. There is an action icon for changing the offering.

@weizhouapache Agree with you this is not a bug, probably we only need to update the document what Kiran is pointing out !!!
Regarding the removal of selection of diskoffering on resize form, it is to make only single point to change the diskoffering using changeOfferingForVolume. There is an action icon for changing the offering.

cool, thanks a lot @harikrishna-patnala
@kiranchavala @DaanHoogland
let's close this if there is no objections ?

closing
@kiranchavala
as Hari mentioned, please use changeOfferingForVolume to change the disk  offering of volume

authors: DaanHoogland harikrishna-patnala kiranchavala weizhouapache


text:
server: acquire IPv4 address when add secondary IP to nic if IP is not specified

Description
After commit fbf488497fb863c13fc0908281e3f4f86906df43, admin need to specify an ipv4 or ipv6 addresses when add IP to nic which breaks backward compatibity. If IP is not specified, a IPv4 address should be returned.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):
How Has This Been Tested?

When I add an IP to a nic and do not specify an IP,
Before this PR, it gives an error "Allocating ip to guest nic 111 failed"
After this PR, it succeeds.

@wido @GabrielBrascher @rhtyd
can you please review and test it ?

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-344

@blueorangutan test

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-503)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27830 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3635-t503-kvm-centos7.zip
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

authors: DaanHoogland blueorangutan rhtyd ustcweizhou


text:
configDrive: Fix failure to delete (unstarted) VM

Description
This PR fixes the issue observed when trying to expunge a VM created using config drive userdata service (with startVM=false) and the env has no secondary stores.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[X] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

On an env with no secondary store, set the global setting: vm.configdrive.primarypool.enabled to true
Deploy a VM with a network offering with configDrive userdata service, but do not start it
Attempt deleting(expunge) the VM - the VM deletes successfully without NPE

Performed the above test with  vm.configdrive.primarypool.enabled set to false and was also able to expunge the VM

@blueorangutan package

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with  KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2987

@blueorangutan test

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-3726)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35622 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6146-t3726-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_primary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py

@blueorangutan test

@weizhouapache a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-3732)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36691 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6146-t3732-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test
Result
Time (s)
Test File

authors: Pearl1594 blueorangutan nvazquez weizhouapache


text:
Fixed: Network Update from RVR offering to Standalone offering fails

Problem: Moving a RVR network offering to standalone makes the status of VR's as UNKNOWN and Redundant Router marked with YES.
Fix: The network's isRedundant was not getting updated.

@kansal Thanks! Can you add the Jira issue id please? Looks like CLOUDSTACK-8844, right?! If you add it, it will also put a comment on the Jira issue.

@remibergsma Yes its CLOUDSTACK-8844. It's already syncing the comments in the JIRA issue.

LGTM, @kansal can you git commit --amend the message to reflect the JIRA ID

LGTM

Did anyone validate the functionality or do you want me to do that?

@remibergsma I tested it on my local setup and worked fine. Maybe you can also check to make sure I haven't missed anything.

@kansal Ok, please rebase against current master so we can run the tests against this branch.

@remibergsma Rebased against the master. Please have a look. Thanks.

@kansal I see many issues on the tests, but I think they're not introduced by you. I rebased locally and run some tests and then it looks much better. Please rebase again with current master and ping me so Ican run the tests again! Thanks!

LGTM, based on a set of tests that I run on this branch (which I rebased myself first):
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
component/test_vpc_redundant.py \
component/test_routers_iptables_default_policy.py \
component/test_routers_network_ops.py \
component/test_vpc_router_nics.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py \
smoke/test_network.py

Result:
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Stop existing router, add a PF rule and check we can access the VM ... === TestName: test_isolate_network_FW_PF_default_routes | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_RVR_Network_FW_PF_SSH_default_routes | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 27 tests in 11894.249s

OK

And:
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_privategw_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_vpc_vpn.py \
smoke/test_service_offerings.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

Result:
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 44 tests in 8923.785s

OK

These test do not cover your change, all they do is show you didn't break them. Someone else needs to review the code.

@DaanHoogland Sorry It was something to do with my IDE's settings. Corrected and rebased.

LGTM I ran the compile on this one and did a code review.
@remi, no changes since your last check, you can merge at your will

authors: DaanHoogland bhaisaab jayapalu kansal remibergsma


text:
LANG-1060 NumberUtils.isNumber assumes number starting with Zero is octal

Add test case

Thanks! 👍

authors: PascalSchumacher piotrkosyoo

repo: apache/commons-lang


text:
Minor refactorings

remove unnecessary (un)boxing
replace for: add with bulk add Collections.addAll
replace iterator with for each where possible

Thanks! 👍
It would be nice if you could fix the check-style violations, see: https://travis-ci.org/apache/commons-text/jobs/297158450
Run mvn checkstyle:check  (or just mvn) to replicated it locally.

Sure. Will be done in an hour)

@PascalSchumacher done. Force pushed this PR.

Coverage decreased (-0.003%) to 98.295% when pulling 58f0f3e6a7f730eb6cb7bc77cabbbaf54ae48353 on Abrasha:fix/remove-unnecessary-(un)boxing into 6313d401d1631884b6a4ae31f2714b066b62f629 on apache:master.

authors: Abrasha PascalSchumacher coveralls

repo: apache/commons-text


text:
Fix UI API called on a background thread (#550, #530, #447)

Platforms affected
iOS
Motivation and Context

It fixes #530, fixes #550, fixes #447, fixes #509, fixes #515
Description

The plugin initializes a picker in the main thread now.
Previously made pull request by @ryaa: https://github.com/ryaa/cordova-plugin-camera/commit/a998ef45d902b514fc1dea53ebb24439034ac108
Testing

I've launched my app and tried to start camera session. It is ok now. No exceptions or low speed.
Checklist

[x] I've run the tests to see all new and existing tests pass
[x] Commit is prefixed with (platform) if this change only applies to one platform (e.g. (android))
[x] If this Pull Request resolves an issue, I linked to the issue in the text above (and used the correct keyword to close issues using keywords)

When is this update expected to be released?

Please merge these fixes as this camera delay opening is a real trouble making in may devices

When is version 4.2.0 going to be released to npm? Why the delay?

4.2.0 is broken and failed release vote.

authors: AndrWeisR erisu exxbrain gvsakhil stevenleija

repo: apache/cordova-plugin-camera


text:
Android X - Handle location permission when user selects- "Allow only when using the app"

Bug Report
Problem
What is expected to happen?
Prior to Android X, when getCurrentPosition() is called, And user clicks on Allow, we used to get the current location.
Now with Android X, we get new dialog with 3 options -
Allow
Allow only when using the app
Deny
When the second option is selected, and the location is off from the device settings, Nothing happens - neither error nor success.
What does actually happen?
We should get appropriate response from the method, so that we can handle the error gracefully.
Information

Command or Code

Environment, Platform, Device

Version information

Checklist

[ ] I searched for existing GitHub issues
[ ] I updated all Cordova tooling to most recent version
[ ] I included all the necessary information above

With Android11, we get a dialog with 3 options -
Only this time
While using the app
Deny
When user select [Only this time] or [While using the app], getCurrentPosition() callback error with timeout

Also seeing this issue on Android 11. When I select 'Only this time' or 'While using the app', getCurrentPosition is timing out.

Any fix for this?

Fixed by https://github.com/apache/cordova-plugin-geolocation/pull/250

repo: apache/cordova-plugin-geolocation


text:
Maybe we need a new admin UI?

I love CouchDB and I think CouchDB is great! However it's admin UI looks old in 2018 now. It works fine but the UI design is not so attractive any more especially compared with other DB systems.
Maybe the core team already have a plan to improve it? If that's ture, I'll be glad to do some contributions on weekend.
Thanks. :)

Great that you want to contribute. The CouchDB Fauxton dashboard can be found here https://github.com/apache/couchdb-fauxton
Take a look at the tickets and see which is something you would like to help with.

@garrensmith Thank you. 😃

authors: Jianru-Lin garrensmith

repo: apache/couchdb


text:
connect to couchdb {"error":"error","reason":"eacces"}

Hello All,
I install a V1.6.1 couchdb on centos7.3
there is a issue occurred as blow:
[root@localhost /]# curl 10.42.149.16:5984
{"couchdb":"Welcome","uuid":"5bf4831eb34af53276e6dbf516d87962","version":"1.6.1","vendor":{"version":"1.6.1","name":"The Apache Software Foundation"}}
[root@localhost ~]# curl localhost:5984
{"couchdb":"Welcome","uuid":"263a82f03e188a88855f95a2cb9770f0","version":"1.6.1","vendor":{"version":"1.6.1","name":"The Apache Software Foundation"}}
[root@localhost ~]# curl -X PUT http://localhost:5984/_config/admins/admin -d '"admin"'
{"error":"error","reason":"eacces"}
[root@localhost /]# curl -X PUT http://10.42.149.16:5984/_config/admins/admin -d '"admin"'
{"error":"error","reason":"eacces"}
local.ini
[httpd]
port = 5984
bind_address = 0.0.0.0
I tried google many times but failed
please help
thank you

eacces means your file permissions are wrong. Check wherever you installed couchdb, and ensure that the *.ini files are writeable by the CouchDB user.
You will want to check the same for your data directory too.

@wohali
thank you

authors: bruceforlearn wohali


text:
Depend on another extension to load schema files from jars.

It registers a content handler which provides virtual document support for jar URIs. Requires the Metals extension >=1.13.0, but the latest version is 1.21.0 anyway.
Fixes #265.

+1 If scala's metals license is ok.
Please verify their license allows us to depend on it.

Their license is Apache 2.0.

I found https://github.com/wmanth/vscode-jar-viewer, which also has the ability to view files in jars. Seems it works by creating a new TextDocumentContentProvider:
https://vshaxe.github.io/vscode-extern/vscode/TextDocumentContentProvider.html
Creating a custom one for our use cases feels like it might be a better approach and gives us more control over how the file is opened--maybe we can eventually provide more useful features since we know it's a DFDL schema.

Creating a custom one for our use cases feels like it might be a better approach and gives us more control over how the file is opened--maybe we can eventually provide more useful features since we know it's a DFDL schema.

Actually, looking at that extension more, maybe that's unique to opening just a Jar file, and not a file:// URI that points inside a jar. Regardless, it feels like something along those lines might be a better approach than depending on the metals extension.

I found https://github.com/wmanth/vscode-jar-viewer, which also has the ability to view files in jars.

I replaced Metals with the jar-viewer plugin and it loads the jar-contained schemas the same way:

It uses the MIT License.

Also, since this extension allows browsing of jar files, it could perhaps also be used to fix #76 by allowing navigation into the schema files. I'll test that out also.

@arosien I believe your issue is actually something that I fix in #414 , where the a parseArgs method is a nodejs 19 thing. If you look at the yarn test for  https://github.com/apache/daffodil-vscode/actions/runs/3907091146/jobs/6688638589 it actually fails because that reason. So I am thinking you will need that fix here

@arosien Would you be able to approve #414? This will fix the yarn test failing in the CI (but still passing for some reason). Then once that is merged if you rebase and fix the format issue I will approve this.

@arosien I believe your issue is actually something that I fix in #414 , where the a parseArgs method is a nodejs 19 thing. If you look at the yarn test it actually fails because that reason, not sure why it turns green though. So I am thinking you will need that fix here.
(the line you added to install that extension in the test was a good catch also just believe it will still fail for the above reason)

@Shanedell I reran the failing check and it passed, so it was some nondeterministic effect going on. I'll look at #414 anyway. Thanks.

authors: Shanedell arosien stevedlawrence

repo: apache/daffodil-vscode


text:
WIP: move parquet related functionality to datafusion-parquet crate

Which issue does this PR close?
related to #11182
Rationale for this change
Splitting FileFormat implementations out of datafusion-core will help ensure datafusion remains modular and extensible, as well as improve maintainability. ``
What changes are included in this PR?
Early work on moving parquet related code to a datafusion-parquet crate.
Are these changes tested?
No (not complete yet)
Are there any user-facing changes?
Yes

Nice -- thanks @devinjdangelo
I was thinking about potential ways to organize the crates. One potential in my mind was
datafusion-catalog (has TableProvider, CatalogProvider, etc, maybe Mem*Provider)
datafusion-catalog-listing (ListingTable)
datafusion-datasource-parquet
datafusion-datafsource-avro
datafusion-datasource-csv
datafusion-datasource-json
datafusion-datasource-arrow

Though perhaps that might be overkill (especially for formats like datafusion-datasource-csv...)
Maybe it would be better like
datafusion-catalog (has TableProvider, CatalogProvider, etc, maybe Mem*Provider)
datafusion-catalog-listing (ListingTable)
datafusion-datasource (built in formats like avro, csv, json, arrow, parquet)

authors: alamb devinjdangelo

repo: apache/datafusion


text:
[DSIP-][SDK] Java SDK

Search before asking

[X] I had searched in the DSIP and found no similar DSIP.

Motivation
I think we need to create a new module, perhaps called dolphinscheduler-spec, and move all the POJO objects from the task-api, registry-api, datasource-api, and dolphinscheduler-api modules into this new module. This way, we can easily create a Java SDK for external use.
Design Detail
dolphinsheduler-spec
+- dolphinsheduler-task-api
+- dolphinsheduler-registry-api
+- dolphinsheduler-datasource-api
+- dolphinsheduler-scheduler-api
Compatibility, Deprecation, and Migration Plan
No response
Test Plan
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

I think we need to create a new module, perhaps called dolphinscheduler-spec, and move all the POJO objects from the task-api, registry-api, datasource-api, and dolphinscheduler-api modules into this new module. This way, we can easily create a Java SDK for external use.

dolphinsheduler-spec
+- dolphinsheduler-task-api
+- dolphinsheduler-registry-api
+- dolphinsheduler-datasource-api
+- dolphinsheduler-scheduler-api
I don't see how this is related to JAVA SDK. Please describe clearly what you want firstly. Then the design scheme.

I also can't find out the relationship between jdk and pojo

@SbloodyS Sb @davidzollo da
For the SDK, we need to share base classes with SDK users, such as ProcessDefinition, DEPENDENT task, K8s task, and Schedule. If we can move these classes to a common API module, this module can then be reused by both the SDK module and the Master/Worker modules.
Of course, we can create some DTO classes, but we also need to plan for creating DTOs for future classes.

I think what you want to say is:
move project's POJO class to a splited module, then the system out of DS can import this module and use those POJO, am I right?
Maybe you can update the title then we can understand more easily.

authors: Dyqer SbloodyS davidzollo liunaijie

repo: apache/dolphinscheduler


text:
solve Condition node bug

Purpose of the pull request

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

solve condition node bug

I noticed that your ISSUE is reported to the NPE on the back end, and you seem to only modify the front end. The back end should also make relevant judgments.


text:
[Feature-5951][Improvement][UI] When creating an alarm instance, the alarm instance name input box prompts optimization

Purpose of the pull request
close #5951

Codecov Report

Merging #7770 (2c52593) into dev (fa906e7) will decrease coverage by 0.07%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev    #7770      +/-   ##
============================================
- Coverage     40.95%   40.88%   -0.08%
+ Complexity     3671     3666       -5
============================================
  Files           642      642
  Lines         26707    26707
  Branches       2997     2997
============================================
- Hits          10939    10919      -20
- Misses        14716    14742      +26
+ Partials       1052     1046       -6

Impacted Files
Coverage Δ

...he/dolphinscheduler/common/enums/SqoopJobType.java
0.00% <0.00%> (-88.89%)
:arrow_down:

...r/plugin/registry/zookeeper/ZookeeperRegistry.java
48.21% <0.00%> (-8.04%)
:arrow_down:

...org/apache/dolphinscheduler/remote/utils/Host.java
37.77% <0.00%> (-2.23%)
:arrow_down:

...inscheduler/common/task/sqoop/SqoopParameters.java
74.00% <0.00%> (-2.00%)
:arrow_down:

...e/dolphinscheduler/remote/NettyRemotingClient.java
52.11% <0.00%> (-0.71%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa906e7...2c52593. Read the comment docs.

authors: SbloodyS codecov-commenter


text:
以小时自动创建分区，分区创建的时间越来越迟，最后可能第一个按小时的分区不会创建导致数据丢失

1.以小时自动创建分区，第一个分区和第二个分区的创建时间并不是严格的相差1小时，而是比1小时多几秒
2.一天有24小时，则最后一个分区的创建时间就会延迟数十秒才创建
3.到第二天系统再自动创建分区时，第一个分区就明显比第一次创建延迟了很久才创建
放图报错，暂时无法放图
6号创建的分区比3号创建的分区完了30多秒

目前动态分区的后台调度周期是由 fe的 dynamic_partition_check_interval_seconds 参数来控制的，每sleep这么多秒就会进入下一次判断，而每次执行添加新partition，删除过期partition的耗时，就导致下次任务的调度会有顺延才会再执行
其实延伸一下，这个地方会存在一个调度不及时的问题。当前dynamic_partition_check_interval_seconds参数默值是600秒，10分钟才调度一次动态分区的检查和添加（如果需要），如果一个fe调度任务在每小时的09、19、29、39、49、59分去执行，这就存在一种情况：当到了需要添加新分区的整小时时，比如按hour的0分时，按天的0点0分时，新分区的添加是会延迟9分钟才执行的，这就有可能导致这段没有及时添加新分区的时间里，新数据的插入因为没有对应的分区，而会丢失的，但是这种插入是不会报错的。

好的，谢谢解答疑惑

repo: apache/doris


text:
[Feature] flinksql增量读取doris

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
近实时场景，不想维护hudi，业务数据同步到doris，flinksql读取doris历史及增量数据，写入Doris或mysql
Use case
No response
Related issues
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

What is your design proposal?

authors: stalary ynzzxc


text:
enhancement: avoid contruct groupExpr in graph-simplifier

Signed-off-by: xiejiann jianxie0@gmail.com
Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.34 seconds
load time: 465 seconds
storage size: 17171000255 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230206065031_clickbench_pr_90710.html

authors: XieJiann hello-stephen


text:
fixdescription of recover_with_skip_missing_version in fe-config

…ument
Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 37.78 seconds
stream load tsv:          447 seconds loaded 74807831229 Bytes, about 159 MB/s
stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet:          29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select:          63.2 seconds inserted 10000000 Rows, about 158K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230626103649_clickbench_pr_167872.html

rebase

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.65 seconds
stream load tsv:          503 seconds loaded 74807831229 Bytes, about 141 MB/s
stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select:          29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17161928150 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230710110514_clickbench_pr_175467.html

run buidlall

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.88 seconds
stream load tsv:          449 seconds loaded 74807831229 Bytes, about 158 MB/s
stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc:          57 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet:          29 seconds loaded 861443392 Bytes, about 28 MB/s
insert into select:          25.8 seconds inserted 10000000 Rows, about 387K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230710052531_clickbench_pr_175557.html

authors: hello-stephen yz-jayhua zy-kkk


text:
enhancement  Add dbName && tableName in CreateTableRecord (#26901)

Proposed changes
Add dbName && tableName in CreateTableRecord
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

#26901

authors: JackDrogon xiaokang


text:
fix support amazonaws.com.cn endpoint

Proposed changes
Issue Number: close #28335

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix colcoate node attr lost after merge fragment (#30818)

pick from master #30818
commit id 3e4a2514e1aed741472dd43c77f09a230d2987f2
Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
branch-2.1 reduce routine load job transaction info storage in memory

Proposed changes
#30903

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fix: clone the producer plan and put logicalAnchor generated by Or_Expansion above logicalSink

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall


text:
BugFix fix modify partition's storage policy with different resource issue succeed with execption

Proposed changes
Issue Number: close #34229
when we try to modify a partition's storage policy which have different resource with the previous one, we get error message "currently do not support change origin storage policy to another one with different resource", but the modification same also succeed!
expect:
the modification fail with "currently do not support change origin storage policy to another one with different resource".
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

@Johnnyssc pls add testcase

@Johnnyssc pls add testcase

@xiaokang  thx for reminding，the regression test case has already passed in local env and added in the latest commit

run buildall

run P0

run p0

run p0

run p0

run buildall

run p0

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

LGTM

authors: DarvenDuan Johnnyssc xiaokang


text:
fix Remove file if downloading faile is failed #41778

cherry pick from #41778

run buildall

run buildall


text:
DRILL-7745: Add storage plugin for IPFS

Add storage plugin for IPFS. See detailed introduction here.
TODOs:

[x] Port to Drill 1.18.0
[ ] Add more tests
[x] Support JSON reader

Authors:
This storage plugin was contributed by Bowen Ding, Yuedong Xu and Liang Wang. The authors are affiliated with Fudan University.

@dbw9580
Thanks for contributing this.  Do you want review comments now?

Yes, please.
Some major problems I'm working on are:

support for different data formats. Currently only JSON files are supported, and CSV support is removed due to changes introduced in v1.17 and v1.18. The original implementation was a copy-paste of the easy format plugin (org/apache/drill/exec/store/easy/text/TextFormatPlugin.java). I wonder if there is a better way to reuse the code there.
support for CREATE TABLE statements. This will require changes to the Drill framework responsible for SQL parsing, query plan generation, etc. I would appreciate opinions from you.

Added small refactoring ideas. Did not check implementation details

@dbw9580
We're looking to get a release of Drill out by the end of June.  If you can get these revisions done soon, I can help expedite the review so that we can get this (or at least an MVP of this) into Drill 1.18.

Yes I'll try my best. I was stuck with CSV and writer support, and haven't had much progress so far. Can we settle with basic JSON and reader support for now, and maybe add those later?

@dbw9580
Why don't you break out CSV and the writer support as separate PRs.  Then we can get this in, and work on those for the next release.
I've never done a writer, but I can assist with the CSV reader.   Take a look here:
https://github.com/apache/drill/blob/master/contrib/storage-http/src/main/java/org/apache/drill/exec/store/http/HttpCSVBatchReader.java
The HTTP storage plugin can read either JSON or CSV.  I attempted to use Drill's built in CSV reader but I would have had to do a lot of work on the CSV reader to get it to work... So... just used this simple version.

@dbw9580
I had another thought about the design.  I wonder if instead of "rolling your own" readers, if you could do what the file plugin does and make use of all the existing format plugins.  Basically, that way any new file format for which there is a format plugin could also be read by IPFS.  I don't know IPFS that well to know if that would even make sense, or not.
That also might simplify the code a lot.

@dbw9580
This is definitely making progress.  Will testing require an IPFS installation?

Yes, a running IPFS daemon is required.

@dbw9580
This is definitely making progress.  Will testing require an IPFS installation?

Yes, a running IPFS daemon is required.

Can you write some tests that do not require the IPFS daemon?  I realize that we'll need it for some unit tests, but is it possible to test various components w/o the daemon?

some tests that do not require the IPFS daemon

I guess for the storage plugin config, scanSpec, etc, they are static and independent on a running IPFS daemon. Also for the json reader,  I think I can supply test data from memory directly, w/o actually retrieving it from IPFS.

@cgivre I've added more tests. The tests are not passing, something about Error while applying rule DrillScanRule. However, I was able to successfully execute the test queries through Drill web interface. I don't know how to fix these tests?

@cgivre I've added more tests. The tests are not passing, something about Error while applying rule DrillScanRule. However, I was able to successfully execute the test queries through Drill web interface. I don't know how to fix these tests?
Edit: attach log file.
org.apache.drill.exec.store.ipfs.TestIPFSQueries.txt

It appears that the query is not getting through the planning phase.  My suggestion is to take a look at this tutorial about writing storage plugins:
https://github.com/paul-rogers/drill/wiki/Storage-Plugin, and
and specifically, follow the debugging procedures that Paul outlines.  My hunch here is that something is going wrong with the  schema resolution.

@cgivre it turned out what was blocking the tests was that the default number of providers in test config was too large, as a result IPFS could not find any other providers in time, thus the TimeoutExceptions. I wish the test logs had included full stack traces, which could have saved me hours looking into the Drill planner internals... 😓

Cleaning up the PR. I was thinking about the unit tests and it might be good to include unit tests using Mockito to mock up some of the various components. That way we can test at least some of this without the IPFS daemon. I can post an example if you'd like.

Would appreciate that.

Cleaning up the PR. I was thinking about the unit tests and it might be good to include unit tests using Mockito to mock up some of the various components. That way we can test at least some of this without the IPFS daemon. I can post an example if you'd like.

Would appreciate that.

Take a look here for an example:
https://github.com/apache/drill/blob/5900cdfaae20e216d4b87795bd2efc8199e648e6/contrib/storage-elastic/src/test/java/org/apache/drill/exec/store/elasticsearch/ElasticSearchGroupScanTest.java#L42-L96

@dbw9580
I'll take a look over the weekend.  Thanks for the contribution!

@dbw9580
Please verify that the project builds and passes all checkstyles.  TestIPFQueries fails the checkstyle due to unused imports.

TestIPFQueries fails the checkstyle due to unused imports.
@cgivre Hmm I don't see any unused imports in this file and my builds are passing.

@dbw9580
I redownloaded and it built for me.  Please disregard previous comments.

I'm still having issues actually getting the unit tests that require the IPFS daemon to actually execute.

@cgivre Actually I am having trouble making that test run, too. I keep getting errors like "connection rejected: /127.0.0.1:31011" or "Protocol family unavailable: /0:0:0:0:0:0:0:1:31011". I can test successfully manually through the web ui with drill-embedded, though.
Can you try testing through the web ui, too? The simple dataset should be easy to add to IPFS and test:
ipfs object patch set-data $(ipfs object new) <path-to-simple-dataset.json>

This will return the hash of the simple dataset, which is QmcbeavnEofA6NjG7vkpe1yLJo6En6ML4JnDooDn1BbKmR.
Then run a query through the web ui: select * from ipfs.`/ipfs/QmcbeavnEofA6NjG7vkpe1yLJo6En6ML4JnDooDn1BbKmR#json`  .
If the query takes too long to complete, try reducing the timeout values as well as the max-peers-per-leaf value in the plugin config.

If I leave an instance of Drill running and then run the unit test (TestIPFSQueries), then it passes. I think the unit test does not actually build and  run a full Drill server, which is why the connections are rejected.

@dbw9580
The ClusterTest class is supposed to start a Drill cluster so that you can execute queries.  You should not need to have a Drill cluster running for the unit tests to complete.
I think the reason this isn't doing what you're expecting is that in the initIPFS function in IPFSTestSuit you are creating a plugin with a null configuration and hence isn't initializing correctly.
I stepped through testSimple() with the debugger and the dataset object is null, hence the test fails.  My suspicion is that there is one small step being missed here.  Could you please take a look and step through this to make sure that Drill is being initialized correctly?
Thanks

@cgivre Does Drill support connections from IPv6 sockets? Is it enabled by default or do I have to toggle some configuration items? The "protocol family unavailable" error could be due to lack of support for IPv6.

@dbw9580 I believe Drill does support connections from IPv6 sockets.  There was a recent PR for this in fact: (https://github.com/apache/drill/pull/1857) but I'm not sure if that is directly relevant.
Were you able to get it working?

The connection rejected: /127.0.0.1:31011 failure was because sometimes Drill does not bind to the default ports (31010, 31011, 31012). It can bind to later ports like 31013, 31014, 31015, hence the connection was rejected.
I believe the reason Drill didn't bind to the default ports is that those ports was used by the process from the last test run and had not been recycled by the system. If I wait for a minute or two before starting another round of testing, it's likely the test will pass.
This is part of DRILL-7754, but I haven't come up with a plan to reliably store the ports info in IPFS.

@dbw9580 I believe Drill does support connections from IPv6 sockets. There was a recent PR for this in fact: (#1857) but I'm not sure if that is directly relevant.
Were you able to get it working?

I don't see Drill binding to any IPv6 address in ss -6ltnp. I blocked IPv6 addresses in 9494a30 and the tests are now passing (most of the time, due to https://github.com/apache/drill/pull/2084#issuecomment-674045895).

@dbw9580
The unit tests are passing now on my machine.

@cgivre
I tried to set the ports to their default values in c090a43, but it did not seem to do the trick. Why is that?

Found that if I ran tests with mvn clean test -DforkMode=never, then the port already in use errors were gone. Have no idea why.

Hi @dbw9580
Thanks for these updates.  I didn't have any issues running your unit tests before this.  However, I took a look at the Maven docs, and I'm wondering if you can specify the number of forks directly in the pom.xml file. 1

Hi @dbw9580
Thanks for these updates. I didn't have any issues running your unit tests before this. However, I took a look at the Maven docs, and I'm wondering if you can specify the number of forks directly in the pom.xml file. 1

Thanks!!

@dbw9580
Can you please rebase on current master as well?

@cgivre I think it's based on the current master already.

@dbw9580
This is looking pretty good. I'm going to do a final check this evening or tomorrow, but can you please:

Squash all commits and add message of DRILL-7745: Add storage plugin for IPFS as the commit message
Go through and do a final code hygiene check (make sure there are no extra spaces, commented out blocks etc). Drill does have a code formatter1 and just verify that the code complies with the coding standards for spacing and all that.  (I didn't see anything jump out at me, but it always helps to double check)

Thanks!

@cgivre sure, but I have to do these tomorrow (it's now midnight in my timezone). And maybe allow some time for the IPFS API repo to release an official version: https://github.com/ipfs-shipyard/java-ipfs-http-client/pull/172#issuecomment-674938957 ?

@dbw9580
What would happen in the following scenario.  Let's say you have user A who executes a query using IPFS and which spins up new Drillbits.  User B then decides to execute a query that does not use IPFS.  Is it possible that if these two queries are concurrent, could user B's query end up on the Drillbits for IPFS and then either not find data or cause some other problem?
Alternatively, what would happen if user B executes a query while user A's IPFS queries are running.  What would happen if user A's query completes before user B?  Would it tear down the Drillbits and cause a crash?
I'm asking because I really don't know here..

The test failure looks irrelevant.

@dbw9580
Hi there!  I hope all is well.  Are you still interested in completing this PR?

@dbw9580
Hi there! I hope all is well. Are you still interested in completing this PR?

Yes. I'm currently busy with other projects, and haven't had time to look further into this. I remember we were having some discussions about the way this plugin interacts with the Drill coordinator that needs a major design reconsideration. When I can spare more time on this, I will continue where I left off.

@dbw9580 I was looking at this again after someone tagged me in an issue, but I think there is a WAY easier way to get this done.  Take a look at this PR that I submitted to add the Dropbox file system to Drill: (https://github.com/apache/drill/pull/2337).  It re-uses a lot of Drill's internals so that you don't have to deal with all the format readers.

authors: cgivre dbw9580 sanel

repo: apache/drill