text:
[BEAM-8962] Add option to disable the metric container accumulator

The FlinkMetricContainer wraps the Beam metric container for reporting metrics,
but also stores them as Flink accumulators. With high parallelism jobs with over
a thousand tasks and many built-in Beam metrics for every Beam step, this can
accumulate to over 100MB of serialized data which is stored in the JobManager's
ExecutionGraph. This then fails to even sent over the wire, due to the
akka.framesize limit (10MB by default), and manifests in {{500 Internal Server
Error}}s in the web frontend.
This introduces an option to disable the reporting via accumulators. The
accumulator is mostly useful for batch workloads where you can retrieve the
final accumulator values at the end of the job. It adds a lot of memory and
network overhead.
In the future, we might disable it by default on streaming workloads.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

SUCCESS
--none--

I think it is worth to think about disabling the metric accumulator by default and only enable it via a --enable_metric_accumulator. The reason is that it provides very little value. The accumulator is used to aggregate the final metric values to write them to the configured Beam MetricSink. However, this is only done on job completion which makes this feature useless for streaming applications. Even for batch, you probably want to be able to see metrics during job execution which the accumulator does not provide. I'm inclined to remove the feature entirely.

authors: asf-ci mxm


text:
[BEAM-9601] unbreaking precommits

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

r: @udim

I've merged to clear up precommits.

source: gharchive/pull-request

repo: apache/beam

license_type: permissive

license_source: github-api


text:
[BEAM-11032] Use metric for Java BigQuery streaming insert API latenc…

…y logging
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @ajamato
Uses Metrics for logging latencies instead of parameters. Reverted BigQueryService API signature changes for histogram and counter.

@ajamato PTAL

@ajamato Friendly ping

@ajamato Friendly ping

@pabloem Would you take a look as well, as a committer. Or lmk if you have someone else in mind on the java side.

@pabloem Would you take a look as well, as a committer. Or lmk if you have someone else in mind on the java side.

Run Java PreCommit

Run Java PreCommit

Run Java_Examples_Dataflow PreCommit

Run Java_Examples_Dataflow PreCommit

Run Java PreCommit

Run Java PreCommit

Run Java PreCommit

Run Java PreCommit

authors: ajamato ihji


text:
[BEAM-12011] Eliminate WindowFn.getOutputTime method

See dev@ thread where consensus is that the existence of this method is a bug and it should be eliminated.
See next user@ thread where no one responded. Perhaps because it is complex and esoteric.
Important points to consider about this change, worth repeating in the PR:

the method is annotated "experimental" but it has been around since the very early days of Beam so we should ignore that
it was added as response to a real user problem: sliding windows + EARLIEST timestamp combiner delays downstream aggregations a lot
but when that problem occurred, timestamp combiner EARLIEST was the default; with the change to how lateness is defined we changed the default to END_OF_WINDOW

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

I think it ended up being ambiguous whether we should temporarily introduce an EARLIEST_SHIFTED timestamp combiner. I have not included it in this change, based on zero replies on user@ (admittedly a very low subscription list).

Run Java PreCommit

run flink validatesrunnner

run spark validatesrunner

run dataflow validatesrunner

run dataflow streaming validatesrunner

Run Flink ValidatesRunner

Flink failure has to do with lifecycle. Almost certainly unrelated. Rerunning.

run flink validatesrunner

Streaming VR suite appears to be perma-red actually, so this is not a useful signal.

The actual failure was unrelated:
12:42:57 > Task :runners:google-cloud-dataflow-java:validatesRunnerLegacyWorkerTestStreaming
14:26:37
14:26:37 org.apache.beam.sdk.transforms.ParDoTest$StateTests > testMapState FAILED
14:26:37     java.lang.RuntimeException at ParDoTest.java:2408
14:48:08
14:48:08 org.apache.beam.sdk.transforms.ParDoTest$StateTests > testSetState FAILED
14:48:08     java.lang.RuntimeException at ParDoTest.java:2352

This is fixed by #14302

A high level question regarding to

it was added as response to a real user problem: sliding windows + EARLIEST timestamp combiner delays downstream aggregations a lot
but when that problem occurred, timestamp combiner EARLIEST was the default; with the change to how lateness is defined we changed the default to END_OF_WINDOW

With this PR, what will happen to the real user problem mentioned above?

A high level question regarding to

it was added as response to a real user problem: sliding windows + EARLIEST timestamp combiner delays downstream aggregations a lot
but when that problem occurred, timestamp combiner EARLIEST was the default; with the change to how lateness is defined we changed the default to END_OF_WINDOW

With this PR, what will happen to the real user problem mentioned above?

Short answer: the real user problem will be back but not as bad.
Longer answer: EARLIEST used to be the default when getOutputTime was introduced to solve the problem. Now the default is END_OF_WINDOW so the problem is not as bad. To use sliding windows (or other overlapping windows) with EARLIEST, the user has some choices:

Just be OK with the delay of downstream GBK (already true for Python)
Set up a non-default trigger (this will free watermark holds and allow progress downstream)
Manually move element timestamps forward with a ParDo before GBK, achieving identical behavior

I think further discussion should continue on the dev@ thread instead of the PR though.

Run Java PreCommit

A high level question regarding to

it was added as response to a real user problem: sliding windows + EARLIEST timestamp combiner delays downstream aggregations a lot
but when that problem occurred, timestamp combiner EARLIEST was the default; with the change to how lateness is defined we changed the default to END_OF_WINDOW

With this PR, what will happen to the real user problem mentioned above?

Short answer: the real user problem will be back but not as bad.
Longer answer: EARLIEST used to be the default when getOutputTime was introduced to solve the problem. Now the default is END_OF_WINDOW so the problem is not as bad. To use sliding windows (or other overlapping windows) with EARLIEST, the user has some choices:

Just be OK with the delay of downstream GBK (already true for Python)
Set up a non-default trigger (this will free watermark holds and allow progress downstream)
Manually move element timestamps forward with a ParDo before GBK, achieving identical behavior

I think further discussion should continue on the dev@ thread instead of the PR though.

Thanks for the explanation. If it's something more like a common issue, we should document this in the release note or somewhere else.

run java precommit

Run Java PreCommit

Run Java PreCommit

Everything was green except for what looked like a flake in Java PreCommit unable to bind a local port. I just rebased with master.

Run Java_Examples_Dataflow PreCommit

authors: boyuanzz kennknowles


text:
Add sdk_harness_log_level_overrides option for python sdk

Fixes #18222
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[x] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

R: @tvalentyn

PTAL

authors: Abacn asf-ci


text:
Fix incorrect parsing of None value parameters.

When a parameter is set as None it gets parsed as the string "None". By not setting the flag in pipeline_options.py we allow it to be set to the default value, which correctly resolves to None.
R: @robertwb
R: @lukecwik

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ n/a ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ n/a ] Update CHANGES.md with noteworthy changes.
[ n/a ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

PythonDocs test is unrelated and fix is provided at https://github.com/apache/beam/pull/25000

R : @damccorm @jrmccluskey  for final review ( Valentyn on vacation)

Run Python PreCommit

authors: AnandInguva damccorm melaniedc


text:
Update default Go version to 1.21.4

Update Go version used to build container boot loaders.
Note, should be cherry picked to release branch in case of a next RC.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @damccorm

It looks like the Jenkins tasks are hung, but I'm not worried about things failing for this change. Merging.


text:
Change dead partition detection to only look at the current topic

instead of looking at all topics. This limits how much data we query for, and avoids access denied errors when a user queries for a topic they don't have permissions to

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@Naireen can you take a quick look at this?


text:
WIP - Bundle retry prototype

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Have you been in contact with @mariapython regarding #3679, which seems to do the same thing?

@charlesccychen we were working on making the test not use global variables, so this PR was just meant to be shared with @mariapython. The code to review is still #3679, I'll close this soon.


text:
[BEAM-4145] Populate the worker_id metadata in the Java SDK Harness

The ID is required for any control service with multiple clients, which is the general case.
Enable ManagedChannelFactory implementations to register interceptors on all of the
created channels.
Add an interceptor which attaches the ID to the metadata. Use this interceptor in the
Sdk Harness Control Client.
Populate the ID passed via the container contract in the boot go code.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

R: @angoenka @youngoli

run java precommit

This PR is replaced by #5680.

authors: jkff tgroh


text:
[BEAM-5760] Add support for multi-element bundles to portable Flink runner.

This makes the operator compatible with the bundle control of the underlying DoFnOperator. Specifically, bundle size can be controlled via the maxBundleSize/maxBundleMillis pipeline options.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

There is no reason to limit to single element bundles, since the Flink Java SDK runner already has the support for larger bundles built in. Essentially this was just the state of portable runner as of the PoC many months ago and we were busy making other things work since then.

@mxm PTAL
FYI no change to the operator javadoc since you are planning to do that as part of the state PR.

Run Flink ValidatesRunner

Java Flink PortableValidatesRunner Tests

Java Flink PortableValidatesRunner

Run Python Flink ValidatesRunner

Run Java Flink PortableValidatesRunner

The Python tests actually pass locally: ./gradlew :beam-sdks-python:flinkCompatibilityMatrixStreaming @angoenka we need to resolve flakiness

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

The PVR job fails due to resource issues Cannot serve slot request, no ResourceManager connected. Adding as pending request @angoenka

This seems to be different failure than the one we normally encounter (segfault) but this gives a sigh that we are definitely under resource pressure.
Also, do committers have access to jenkins workspace? We can check for more information their.

authors: angoenka mxm tweise


text:
[BEAM-6058] Adding option for flink configuration directory and setting config in exectution environment

This is needed as when submitting to remote execution environment, client config from the request is used.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Run Python Flink ValidatesRunner

Run Flink ValidatesRunner

I think this is required to load a custom config for portability. If you submit the "legacy" way, you would have a Flink installation with a custom config which would be automatically discovered.

I think this is required to load a custom config for portability. If you submit the "legacy" way, you would have a Flink installation with a custom config which would be automatically discovered.

Same applies in either case (which is why we need documentation). In portability case the job server replaces the legacy runner and it would discover that configuration (that's how we run the job server in Lyft).

I agree that we should document the reason for changes in JIRA.
How do you let the JobServer discover the config? Via the Java property?

The machine has Flink installed and flink-conf.yaml is same as on every cluster node. No extra properties are required for the job server (other than --flink-master-url or eq pipeline option).

Properties like taskmanager.runtime.large-record-handler are read from the submitted JobGraph and are not read by cluster when using RemoteEnvironment. Hence we need to read and set the properties in the job graph. Many of these properties are not even exposed via ExecutionConfig so we need to read these properties directly.
Reading from a config dir seems be the right way as we are already doing it for parallelism.

Also updated the jira with more details.

@tweise Reminder for review :)

Something that may make sense as part of pipeline options is to let the user provide key value pairs of configuration options and merge those into the configuration object.

Run Python Flink ValidatesRunner

Run Flink ValidatesRunner

I think this needs to be adapted to also use the config in streaming translation. Otherwise it looks fine!

Thanks @mxm .
Updated the PR to pass the parameter in case of streaming as well. I think I missed it earlier.

Is this change now equivalent to setting FLINK_CONF_DIR? If so, it would be good to document that.


text:
[BEAM-6170] Change Nexmark stuckness warnings to not fail pipeline

The monitoring in the NexmarkLauncher watches the pipeline results and source for events.  If no events are consumed from the source or published to the output for over
STUCK_WARNING_DELAY it prints an warning message:
"WARNING: streaming query appears to have been stuck for %d min.",
However it also adds this to the errors list, which causes the pipeline to terminate with an exception.  There is a separate STUCK_TERMINATE_DELAY which is used to terminate the pipeline.  It seems inconsistent to cause the pipeline to fail with the warning timeout.  It is also not entirely accurate to say the pipeline is stuck since it may be busy processing stages other than the input or output.
R: @echauchot

@kennknowles @lgajowy @scwhittle I'm a bit busy on the spark runner. I plan to take a look before the end of the week.

starting the review

retest this please

Thanks Etienne and Lukasz, I made that change as you suggested. I also cleaned up some comments and made logging a little more consistent. Let me know what you think

Run Java PreCommit

Run Java PreCommit

Failure is unrelated (beam-sdks-java-io-hadoop-format), merging

authors: echauchot scwhittle


text:
bthread可能突破数量限制

Describe the bug (描述bug)
现在每个worker的bthread数量限制是4096，但是bthread_start_urgent的产生不会校验worker上面的bthread的数量，这样一个worker上面的bthread数量就有可能突破4096。这块是否应该在bthread_start_urgent加个校验？这块加校验感觉会影响性能。
或者在EventDispatcher::Run()里面做一些校验？
To Reproduce (复现方法)
Expected behavior (期望行为)
Versions (各种版本)
OS:
Compiler:
brpc:
protobuf:
Additional context/screenshots (更多上下文/截图)

我理解urgent就是为了处理紧急任务，所以不应该受到个数限制。类似mysql线程池的处理：当前线程池中无法分配出来新的线程来处理连接了，但是有一个紧急的数据库管理任务需要执行，就开一个特殊连接。

但是现在brpc里面urgent这个意思和你说的还是有差别的，urgent现在用来解析网络数据包，backgroup用于处理数据包，如果处理数据包的任务比较慢，就会造成urgent起来的bthread很多。当然这样的场景是比较极端的，在连接特别多的时候会产生。

source: gharchive/issue

authors: hnwyllmm yanglimingcn

repo: apache/brpc


text:
Shouldn't use a cached failed build with a different strict key

Building in non-strict mode, I got one usual linker error: a library bumped the version, and a library that depends on it wasn't rebuilt and I was building a  third element that depends on both. I bst artifact delete both libraries, disable the remote cache, and launch the build again. After building some stuff, I am presented with the exact same error.
I had a big WTF moment, before remembering that bst has cached failures. Retrying the build indeed works.
I think we shouldn't present a different build (i.e. a build with a different strict key) as the same.

Building in non-strict mode, I got one usual linker error: a library bumped the version, and a library that depends on it wasn't rebuilt and I was building a third element that depends on both.

So we're on the same page here: The above type of error is expected in non-strict build correct ? Because the dependency of a library was not explicitly declared as a strict dependency, and in this particular case it perhaps should have been, given that the versioning information of the library is relevant in this case and causes errors down the line at build time ?

... I bst artifact delete both libraries, disable the remote cache, and launch the build again. After building some stuff, I am presented with the exact same error.
I had a big WTF moment, before remembering that bst has cached failures. Retrying the build indeed works.
I think we shouldn't present a different build (i.e. a build with a different strict key) as the same.

But I'm a bit lost at this point as to what is going on and what happened.
Perhaps the problem is that bst artifact delete will only delete a single artifact, for which it has resolved the key for in the session of running bst artifact delete (was --no-strict also specified at bst artifact delete time ?), and does not provide any option to delete all locally cached artifacts for a given element ?

Because the dependency of a library was not explicitly declared as a strict dependency, and in this particular case it perhaps should have been, given that the versioning information of the library is relevant in this case and causes errors down the line at build time ?

I'd say it's not: bumping soname (that's what I meant by version) is not something that happens often, and making most dependencies strict is almost equivalent to the strict mode while being more verbose. So I guess it's something we should live with as it doesn't happen often.

Perhaps the problem is that bst artifact delete will only delete a single artifact, for which it has resolved the key for in the session of running bst artifact delete (was --no-strict also specified at bst artifact delete time ?), and does not provide any option to delete all locally cached artifacts for a given element ?

Ok, I guess I wasn't explicit enough: I didn't delete the artifact, but I was expecting bst not to reuse it.
So I have three elements: app1.bst, library1.bst and library2.bst. app1 depends on library1, which depends on library2. library2 bumped its soname, and library1 was built against the old soname. I'm building app1, linking fails because library1 needs the old soname of library2 which cannot be found (*). Up to this point, everything is as expected.
I run bst artifact delete library1.bst library2.bst (deleting library1 only should be enough, but this is what I said above), then run the build again without the remote cache.
At this point I expect the build to succeed, building all three elements. However, what happened is that it built the two libraries and presented me with the exact same error in (*) above. It was a cached failure, and answering retry on the prompt built app1 correctly.
My point is that the second build shouldn't reuse the cached failure from the first one as they are different. So it should check the strict key rather the the weak key to determine whether to use the cached failure. I guess this would be even more problematic if it actually pulled the cached failure rather than having it already in the local cache.

Studying this again as it is linked from #1461

My point is that the second build shouldn't reuse the cached failure from the first one as they are different. So it should check the strict key rather the the weak key to determine whether to use the cached failure. I guess this would be even more problematic if it actually pulled the cached failure rather than having it already in the local cache.

Finally after dancing in circles around this in my head, I think I understand and agree with your proposed behavior. I believe that you have confused the (admittedly complex/confusing) terminology of cache keys.
Weak cache key
The weak key takes only the element's input into consideration, disregarding dependencies
Strict cache key
The strict cache key is ultimately what the cache key would be if the element and all of it's dependencies were built in strict mode
Strong cache key
Also referred to simply as the cache key, is calculated from the element's input, and all of the dependencies against which the element was actually built against
Conclusion
I think that what you are suggesting, is that:

If the build is in strict mode
If an artifact is found by it's weak key
If the found artifact does not have the actual strong key which we would have assigned it
If the found artifact is actually a failed build
Then we should ignore the found artifact and attempt to build it again under the new strong key

This behavior would essentially cause failed builds to be retried in non-strict mode as long as anything in the element or it's dependencies has changed since the last failed build.
This would be great indeed.
However, I think we can safely rule out the relevance of this proposed enhancement with issue #1461

authors: abderrahim gtristan

repo: apache/buildstream


text:
Dynamically set Element public data

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on Apr 28, 2017, 09:27
Public data
is exposed on an Element so that other Elements which depend on the given element can read it
It is only interesting to read public data from dependencies
when an Element assembles its artifact. Current use cases of
this include integration commands and artifact splitting rules.
In some cases, like when deploying debian packages from imported
debian supporting sources, it will be important to populate the
splitting rules (essentially file manifests) at the time when the
artifact is built, as we will only really know which files to put
into which package after having run the build.
This task will consist of:

ensuring that public data is recorded in an artifact's metadata
ensuring that the public data is read back into the BuildStream data model whenever the pipeline is loaded and the artifact is found to already be cached
adding some API to the Element allowing the Element to programmatically assign public data at Element->assemble() time.

In GitLab by [Gitlab user @juergbi] on Jul 4, 2017, 14:25
mentioned in merge request !45

In GitLab by [Gitlab user @juergbi] on Jul 4, 2017, 14:32
assigned to [Gitlab user @juergbi]

In GitLab by [Gitlab user @tristanvb] on Jul 5, 2017, 13:14
Fixed by Jürg's branch !

In GitLab by [Gitlab user @tristanvb] on Jul 5, 2017, 13:14
closed


text:
Configurable retries for network related tasks

See original issue on GitLab
In GitLab by [Gitlab user @tristanvb] on May 18, 2017, 07:40
Sometimes a build fails simply because of a network timeout or such, this can happen especially when on build has failed and we've suspended all tasks to debug it, often this can cause an ongoing Fetch task to fail when resumed.
We should:

Have a configuration for maximum retries for network related activities
Reschedule failed network related tasks until maximum retries is reached
Frontend should not suspend tasks when a failure occurs that is in fact being retried

This requires that the failure message for a retry-able task should be special, or, the Message object could have a retry field added to it, otherwise the frontend will suspend tasks and try to handle the failure.

In GitLab by [Gitlab user @juergbi] on Jul 27, 2017, 11:59
mentioned in commit bfd822b3232f69ed9df48a20fedf31be7e0986f1

In GitLab by [Gitlab user @tristanvb] on Jul 27, 2017, 12:06
assigned to [Gitlab user @juergbi]

In GitLab by [Gitlab user @tristanvb] on Jul 27, 2017, 12:08
closed via merge request !67


text:
CAMEL-20814: Revert due to build-info causes to not be able to do Camel releases

Description

Target

[ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ ] I checked that each commit in the pull request has a meaningful subject line and body.

[ ] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

okay @gzurowski I have cherry picked this PR to camel-4.7.x branch.

repo: apache/camel


text:
[CARBONDATA-1817] Reject creating datamap on streaming table

Since streaming segment does not support building index and pre-aggregate yet, so streaming table should not support create datamap operation

[X] Any interfaces changed?
No
[X] Any backward compatibility impacted?
No
[X] Document update required?
No
[X] Testing done
One testcase added
[X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1495/

retest this please

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1501/

@jackylk Changes is not only related to blocking the data map creation. Its better to raise changes done for refactoring in different pr

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1912/

@kumarvishal09 This PR is on top of #1573 , so it contains commit on top of that. I will rebase after #1573 is merged

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1566/

LGTM

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1965/

authors: CarbonDataQA QiangCai jackylk kumarvishal09 ravipesala

repo: apache/carbondata


text:
Cassandra 18948 5.0

Thanks for sending a pull request! Here are some tips if you're new here:

Ensure you have added or run the appropriate tests for your PR.
Be sure to keep the PR description updated to reflect all changes.
Write your PR title to summarize what this PR proposes.
If possible, provide a concise example to reproduce the issue for a faster review.
Read our contributor guidelines
If you're making a documentation change, see our guide to documentation contribution

Commit messages should follow the following format:
<One sentence description, usually Jira title or CHANGES.txt summary>

<Optional lengthier description (context on patch)>

patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####

Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>

The Cassandra Jira

Ci j11 j17

repo: apache/cassandra


text:
Fix incorrect nodetool suggestion when gossip mode is running

patch by Francisco Guerrero; reviewed by TBD for CASSANDRA-19905

Closed via https://github.com/apache/cassandra/commit/fe025c7f79e76d99e0db347518a7872fd4a114bc


text:
[BUG] With VPC - LoadBalancing tab should be hidden if the IP is already used for i.e. Port Forwarding or Static NAT

In VPC, there is no preserve mode with Public IPs - and for Port Forwarding, StaticNAT and LoadBalancing - if any of these services is enabled on a single (additional) Public IP - the other 2 services/tabs/buttons should be hidden - otherwise, user will attempt to configure and get an error like below (an expected error):

@andrijapanicsb I dont see any port forwarding or loadbalancer tab is static nat is enabled

If its not enabled then I see both tabs

that is correct only for StaticNAT @ravening - needs to be fixed also for PF or LB - I'm not sure if @davidjumani is already on it or not?

@andrijapanicsb Havn't started on it yet

@davidjumani can you pick this or advise @utchoang on what/how to fix this? Thanks

@utchoang I think fixing this would involve merging DetailsTab, FirewallRules, PortForwarding, LoadBalancing and VpnDetails into a single tabbed vue. Once this is done :

If the resource has port forwarding / load balancing rules, the enableStaticNat button should be hidden.
If it is an ip associated with a vpc network, allow adding firewall rules, and either port forwarding or load balancing rules. Hide the port forwarding tab if load balancing rules are present and vice versa

@andrijapanicsb can you check the behaviour against old UI?

@rhtyd @davidjumani Currently I don't know which field to rely on to hide the Load Balancer Rule or PortFowardRule tab.

Or is it based on getting data from the 2 API listPortForwardingRules and listLoadBalancerRules?

It is based on the listPortForwardingRules and listLoadBalancerRules apis, while the vpcid filed indicates whether it belongs to a vpc

@utchoang I've created/tested the cases that you can check and fix. Here are the cases: (@andrijapanicsb appreciate if you can be more specific while filing bugs, thanks)
:heavy_check_mark: Case 1 - in case of static NAT I've verified that we don't show any tabs: (old UI has same UX)

:heavy_multiplication_x: Case 2 - VPC has LB rule, it shouldn't show firewall or PF tabs:

:heavy_multiplication_x: Case 3 - VPC has PF rules, it shouldn't show firewall or LB tabs:

:heavy_check_mark: Case 4 - in case the VPC IP has no tier, PF or LB rules, we should show PF and LB tabs for VPC:

@utchoang the real issue are only case 2 and 3; it may need tweaking here https://github.com/apache/cloudstack-primate/blob/master/src/components/view/ResourceView.vue#L115 (we may need to somehow emit and ensure this code runs, after doing the specific list APIs calls (for listing pf and lb rules) and then hide appropriately)

@andrijapanicsb can you check the behaviour against old UI?

I did check, and went with @davidjumani in details through the old UI etc.
This is kinda common thing to check the old UI and it's a matter of 1.5minute testing.

authors: andrijapanicsb davidjumani ravening rhtyd utchoang

repo: apache/cloudstack-primate


text:
Compute offering "Custom constrained" not working

ISSUE TYPE

Bug Report

COMPONENT NAME
Core

CLOUDSTACK VERSION
4.17.0.1 installed on CentOS 7

CONFIGURATION
KVM Hypervisor with compute offering type custom unconstrained
OS / ENVIRONMENT
CentOS 7
SUMMARY
When applying a custom constrained compute offering on a virtual machine on a KVM host it fails to start with error: "internal error: process exited while connecting to monitor: qemu: invalid ram size".
Parameter "memory" in the compute offering is indicated in MB.
There are no issues when applying on the VM a fixed compute offering.
STEPS TO REPRODUCE
Create a VM with a custom constrained compute offering on a KVM host

EXPECTED RESULTS
The VM is created and starts as it happens on previous cloudstack versions.

ACTUAL RESULTS
VM creation fails with error "internal error: process exited while connecting to monitor: qemu: invalid ram size"

@cdfgallo I'm not able to reproduce in my lab. Can you please share logs of the complete deploy VM action?
What value of RAM was VM provided?

here attached you can find the log of the vm deploy.
debug.log
I trried as RAM value 1024 or 2048 with same result.

@cdfgallo can you try with a template or compute offering which dynamically scalable parameter set to false.
I was able to reproduce this with dynamically scalable enabled for VM with a centos7 host having following config:
# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 1.5.3

However same thing worked fine with a Ubuntu host with the following config,
# virsh version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0

@DaanHoogland @GutoVeronezi cc @rohityadavcloud @weizhouapache @nvazquez @harikrishna-patnala VM deployment with following qemu-libvirt config on CentOS7, for a dynamically scalable VM with custom offerings,
# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 1.5.3

Should this be documented or do we need to add a check in KVM hypervisor code for qemu version? Didn't find any existing documentation for it.

it is very operator friendly if we do an software check for version on each feature but this will not be very maintainable. I vote documentation.

I've created doc PR to address this, https://github.com/apache/cloudstack-documentation/pull/283
@cdfgallo can you please try deploying VM with template and offering that do not have dynamic scaling enabled

@cdfgallo can you please try deploying VM with template and offering that do not have dynamic scaling enabled

@shwstppr sorry but if I apply a fixed compute offering how can it scale dinamically?

@cdfgallo can you please try deploying VM with template and offering that do not have dynamic scaling enabled

@shwstppr sorry but if I apply a fixed compute offering how can it scale dinamically?

@cdfgallo , I think the message is that you can not if
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 1.5.3

but if for instance
# virsh version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0

you can.
You´ll have to check your cobination.

@cdfgallo can you please try deploying VM with template and offering that do not have dynamic scaling enabled

@shwstppr sorry but if I apply a fixed compute offering how can it scale dinamically?

@cdfgallo please refer http://docs.cloudstack.apache.org/en/latest/adminguide/virtual_machines.html#cpu-and-memory-scaling-for-running-vms
When you create a compute offering there is an option to enable/disable dynamic scaling. Similar option is there when you register a template.
The QEMU error is not seen when you use a fixed offering because for that minRam, maxRam attributes are the same while creating VM definition. To deploy VM with a custom offering when using qemu/kvm version that isn't supporting dynamic scaling you'll have to create a new custom offering which has dynamic scaling option disabled.

@shwstppr if I apply a compute offring without dynamic scaling enabled the dynamic scale option is grayed out in the UI

@shwstppr if I apply a compute offring without dynamic scaling enabled the dynamic scale option is grayed out in the UI

@cdfgallo I didn't understand your comment.
Currently, dynamic scaling option cannot be used with a custom compute offering and the following qemu/kvm config,
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 1.5.3

Dynamic scaling and offering type are two different things here.
When your offering has dynamic scaling enabled and the corresponding template is also dynamic scaling enabled, the user can deploy a dynamic scaling enabled VM. A VM which can be scaled using a new offering in Running state.
The custom offering allows users to deploy VMs with desired CPU, and memory. This is in contrast to fixed offerings where the admin has already set CPU and memory.
Both fixed and custom offerings can be dynamic scaling enabled or not.
Please refer to the documentation for more clarity.

Closing this as we added in the documentation for dynamic scaling behaviour on KVM, https://github.com/apache/cloudstack-documentation/pull/283

authors: DaanHoogland cdfgallo shwstppr

repo: apache/cloudstack


text:
convert protocol names to be found as labels

Description

replace significant chars from string to find protocol labels

Fixes: #3252
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

@blueorangutan package

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-407

reviewers please consider if this is for the right milestone.

needs to be

@DaanHoogland If we consider it as a bug then it makes sense to have it fixed on 4.13.1.0. However, if it is considered an enhancement then I don't see why not keeping the 4.14 milestone.
Looking at #3252 I would consider it as a bug on the UI; +1 on setting it to branch 4.13 and milestone 4.13.1.0.

rebasing on 4.13 in #3747


text:
maven: update dependencies

Updates few maven dependencies
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-996

@blueorangutan test

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-1186)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28258 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3928-t1186-kvm-centos7.zip
Smoke tests completed. 81 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

authors: blueorangutan rhtyd


text:
Application crash when take picture from camera and accept this image

Bug Report
Problem
When I take a picture with the tablet and accept, the application crashes.
What is expected to happen?
The image is successfully added and the alert is displayed.
What does actually happen?
The application is crashed.
Information

When select image from gallery it working fine
When change option to allowEdit to true, no crashes but showed this alert: You can't edit this image.
Command or Code

function setOptions(srcType) {                 var options = {                     // Some common settings are 20, 50, and 100                     quality: 75,                     destinationType: Camera.DestinationType.FILE_URI,                     // In this app, dynamically set the picture source, Camera or photo gallery                     sourceType: srcType,                     encodingType: Camera.EncodingType.JPEG,                     mediaType: Camera.MediaType.PICTURE,                     allowEdit: false,                     correctOrientation: true                 }                 return options;             }
                    var srcType = Camera.PictureSourceType.CAMERA;
                    var options = setOptions(srcType);

                    navigator.camera.getPicture(function cameraSuccess(imageUri) {

                        alert("get picture");

                    }, function cameraError(error) {
                        console.debug("Unable to obtain picture: " + error, "app");

                    }, options);

Environment, Platform, Device

Lenovo tab m10 hd
Android: 11
Version information

Cordova 11.0.0
Cordova platform: Android 10.1.2
Cordova plugins:

Android studio: 2021.1
Android sdk-tools:

Java Jdk:

Gradle:

Operating System: Windows 10 Home
Checklist

[x] I searched for existing GitHub issues
[x] I updated all Cordova tooling to most recent version
[x] I included all the necessary information above

I have the same issue, have you solved it?

Hi @FZampetti,
I change the plugin. Now I use https://github.com/apache/cordova-plugin-media-capture.
I hope this helps to you.

Presumably the same issue as #679

have something similar, any recommended solution

Closing as stale.
If the "crash" involves the webview restarting, then it isn't a crash. It's the android activity that got killed while the camera is operating, which is normal android lifecycle. Your camera response will be in the resume event. Android Quirk
If the app is fatally crashing (completely closes, potentially with "App stopped working" dialog, then it is likely related to RecoverableSecurityException error which is indirectly fixed by https://github.com/apache/cordova-plugin-camera/pull/907

repo: apache/cordova-plugin-camera

license_source: bigquery


text:
7.0.1 patch release updates

Platforms affected
Windows
Motivation and Context

I would like to make a patch release to resolve issue #341 that WinJS  is not copied correctly. I would rather not have to rebuild and update cordova.js in this patch release.
Description

Cherry-pick a limited number of updates from master onto 7.0.x branch to be part of 7.0.1 patch release:

Conditionally skip the one broken test for VS 2017 (#330) ... instead of allowing whole suite to fail
Fix path to winjs dependency (#331)
Support latest Visual Studio 2017 (15.9.x) (#329)
refactor: split tests into suites for Windows 10 and Windows 8.1 (#349)

✅ updated version to 7.0.1-dev in the first commit of this PR
Testing

[x] verified green build on AppVeyor CI & Travis CI
[x] check that I am able to add cordova-windows platform from this branch to a test project and ensure that the test project works correctly

Checklist

[x] I've run the tests to see all new and existing tests pass
I added automated test coverage as appropriate for this change
Commit is prefixed with (platform) if this change only applies to one platform (e.g. (android))
If this Pull Request resolves an issue, I linked to the issue in the text above (and used the correct keyword to close issues using keywords)
I've updated the documentation if necessary

Codecov Report

Merging #372 into 7.0.x will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##            7.0.x     #372   +/-   ##
=======================================
  Coverage   73.53%   73.53%
=======================================
  Files          17       17
  Lines        2116     2116
=======================================
  Hits         1556     1556
  Misses        560      560

Impacted Files
Coverage Δ

template/cordova/lib/MSBuildTools.js
62.83% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e7587a...5adc231. Read the comment docs.

I just merged by pushing the changes to the 7.0.x branch.

FYI I cherry-picked one more change with nyc@14 update in devDependencies (PR #356) to resolve a non-production npm audit warning. I pushed this change directly into the 7.0.x branch in order to make the patch release with zero npm audit warning messages.

authors: brodybits codecov-io

repo: apache/cordova-windows


text:
Update README.md

Fix a few typos and such

+1

authors: DeadZen iilyak

repo: apache/couchdb-couch-epi


text:
Setting ssl ciphers config does not work

Description
If you uncomment ciphers or tls_versions for example in local.ini, starting couchdb fails. This could be something on my end, but I've been toying with this for a couple days with no luck. We have ssl working just fine, but are trying to restrict the ciphers to be used now.
Steps to Reproduce
Uncomment the default ciphers config in local.ini. The http server will fail to start when starting couchdb. If you tweak from the default at all, you'll likely get an error that mochi had an issue parsing
Expected Behaviour
Your Environment

CouchDB version used: 3.1.1
Browser name and version:
Operating system and version: CentOs7

tried several different erlang versions 20, 21, and 22 major versions
Additional Context

Is there a plan to fix this issue in a comping release?

Hi is there anyway for someone to check on this to at least verify it is truly a bug, whether or not that actually involves fixing it at the same time? We would like verification before we put more effort in trying to attempt to get this to work.

repo: apache/couchdb


text:
JS stats test has inconsistent results

Current & Expected Behaviour
Sometimes, the JS test/javascript/tests/stats.js test errors with off-by-one errors such as the following:
test/javascript/tests/stats.js
    Error: expected '8', got '9'
Trace back (most recent call first):

  52: test/javascript/test_setup.js
      T(false,"expected '8', got '9'","Request counts are incremented proper
 321: test/javascript/couch_test_runner.js
      TEquals(8,9,"Request counts are incremented properly.")
 132: test/javascript/tests/stats.js
      (6,9)
  46: test/javascript/tests/stats.js
      runTest([object Array],[object Object])
 131: test/javascript/tests/stats.js
      ()
  37: test/javascript/cli_runner.js
      runTest()
  48: test/javascript/cli_runner.js

fail

Possible Solution
This test was recently re-enabled under the assumption that it was good to have something covering this area of our API, but significant sections had to be commented out because results didn't match up with requests being made.
It would be good to determine whether these tests are deterministic enough - or whether we actually have a bug in our stats handling - since numbers seem off in quite a few places.
Your Environment
Most recent failure is Jenkins, master branch, logs _id jenkins-couchdb-1-2017-05-29T01:59:56.803570, Ubuntu 12.04, Erlang 18.3.

stats.js relies heavily on restartServer which I believe we nixed for 2.0 {

restartServer() works with 2.0 via a call out to dev/run, see my comments here: https://github.com/apache/couchdb/pull/556#issuecomment-305900649

yup, sorry,retract retract

We're still seeing a bunch of these:
https://travis-ci.org/apache/couchdb/jobs/243780115 - stats.js mismatch line 58
https://travis-ci.org/apache/couchdb/jobs/243303510 - stats.js mismatch line 58
I think we have a real set of bugs here, but it may be time to re-disable the entire stats.js test for now. The fact that the test run is failing is a blocker; the fact that our stats are sometimes inconsistent, to me, isn't - though it may be indicative of race conditions as of yet uncovered.
Thoughts?

I have fully disabled this test for now. I'm starting to think that the dynamic behaviour of the clustered setup means these stats don't make as much sense anymore -- and that the test needs a complete rework.
We can revisit after 2.1.0.

authors: janl wohali


text:
Is there a HllMap C++ implementation?

Is the HllMap available here?  I see it on the website documentation and in datasketches-java, but not sure what is equivalent here.

HllMap in Java exists to address the problem of object overhead. Imagine a HashMap from a standard Java library with tens or hundreds of thousands of pairs, say, Integer to HllSketch. Just the object overhead will take a lot of memory. There is no such problem in C++. Try something like std::unordered_map<int, hll_sketch>. I believe it should work fine. Please let us know if you encounter any problem.

HllMap in Java exists to address the problem of object overhead. Imagine a HashMap from a standard Java library with tens or hundreds of thousands of pairs, say, Integer to HllSketch. Just the object overhead will take a lot of memory. There is no such problem in C++. Try something like std::unordered_map<int, hll_sketch>. I believe it should work fine. Please let us know if you encounter any problem.

Thank you, I will try that.

Thank you, I will try that.

authors: AlexanderSaydakov bryanherger

repo: apache/datasketches-cpp


text:
[Bug] [Dependent] Start process instance error when enable dependent mode

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
Start process instance error when enable dependent mode,
I find in dependentCommand = (Command) BeanUtils.cloneBean(command) the command id not modified due to Duplicate entry 'xxx' for key 't_ds_command.PRIMARY', I suspect it relation https://github.com/apache/dolphinscheduler/pull/11765
What you expected to happen
Start process success.
How to reproduce
Use dev code, create two process, one parent, one child dependent parent and start parent process use dependent mode.
Anything else
No response
Version
dev
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I will fix this.

authors: ruanwenjun stalary

repo: apache/dolphinscheduler


text:
After copying the public workflow, all workflows reported errors.

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
全部工作流崩溃了
What you expected to happen
正常运行
How to reproduce
复制一个工作i路
Anything else
No response
Version
3.2.x
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

数据库报错。


text:
[Bug] [dolphinscheduler-sql] create index if not exists #8594

fix #8594

Codecov Report

Merging #8595 (f1e89ef) into dev (a3e2c70) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##                dev    #8595   +/-   ##
=========================================
  Coverage     45.31%   45.31%
  Complexity     4059     4059
=========================================
  Files           696      696
  Lines         26978    26978
  Branches       2889     2889
=========================================
  Hits          12224    12224
  Misses        13602    13602
  Partials       1152     1152

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3e2c70...f1e89ef. Read the comment docs.

@JinyLeeChina @lenboo pls review, tks

These changes have been verified during the upgrade from 2.0.2 to 2.0.4

Hi @ouyangyewei  I have a question, this is called by the upgraded script, it should be called manually once, it doesn't matter how much the api replica is;
BTW, You should also modify the ddl of pgsql.

When deployed through K8S, if the API is multi-replicas, the first Pod will successfully create an index, but other Pods will fail to start and raise an exception: java.sql.SQLSyntaxErrorException: Duplicate key name 'idx_code_version'

Hi @ouyangyewei I have a question, this is called by the upgraded script, it should be called manually once, it doesn't matter how much the api replica is; BTW, You should also modify the ddl of pgsql.

When deployed through K8S, if the API is multi-replicas, the first Pod will successfully create an index, but other Pods will fail to start and raise an exception: java.sql.SQLSyntaxErrorException: Duplicate key name 'idx_code_version'

Hi @caishunfeng , when deployed through K8S, if the dolphinscheduler-api is multi-replicas, the first Pod will call and create indexs successfully, but other Pods will fail to start and raise an exception: java.sql.SQLSyntaxErrorException: Duplicate key name 'idx_code_version'

Can you upgrade ddl manually? The upgrade script and api-server are not related.

@caishunfeng Related. The problem occurs when deploying through kubernetes clustering. Recently tested, it happens when upgrading from 2.0.2 to 2.0.4
After a simple analysis of the code, the tracking is as follows:
Code Trace

docker/build/Dockerfile

Assign args 'api-server' to container

docker/build/startup.sh

Execute CreateDolphinScheduler.main() once

Log Trace
init env variables
generate dolphinscheduler config
try to connect zookeeper ...
zookeeper connection is ok
try to connect mysql ...
mysql connection is ok
import sql data

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.5.6)

log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2022-03-15 21:20:46.418  INFO 34 --- [           main] o.a.d.d.u.shell.CreateDolphinScheduler   : Starting CreateDolphinScheduler v2.0.5 using Java 1.8.0_322 on pre-workflow-api-56c459578b-4jm49 with PID 34 (/opt/apache-dolphinscheduler-2.0.5-bin/lib/dolphinscheduler-dao-2.0.5.jar started by root in /opt/apache-dolphinscheduler-2.0.5-bin)
2022-03-15 21:20:46.421  INFO 34 --- [           main] o.a.d.d.u.shell.CreateDolphinScheduler   : The following profiles are active: shell-create,shell-cli,default,mysql
2022-03-15 21:20:47.801  INFO 34 --- [           main] ru.yandex.clickhouse.ClickHouseDriver    : Driver registered
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2022-03-15 21:20:47.932  INFO 34 --- [           main] com.zaxxer.hikari.HikariDataSource       : DolphinScheduler - Starting...
2022-03-15 21:20:47.937  WARN 34 --- [           main] com.zaxxer.hikari.util.DriverDataSource  : Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-03-15 21:20:48.168  INFO 34 --- [           main] com.zaxxer.hikari.HikariDataSource       : DolphinScheduler - Start completed.
 _ _   |_  _ _|_. ___ _ |    _
| | |\/|_)(_| | |_\  |_)||_|_\
     /               |
                        3.2.0
2022-03-15 21:20:49.977  INFO 34 --- [           main] o.a.d.d.u.shell.CreateDolphinScheduler   : Started CreateDolphinScheduler in 4.171 seconds (JVM running for 4.62)
2022-03-15 21:20:50.005  INFO 34 --- [           main] o.a.d.d.upgrade.DolphinSchedulerManager  : The database has been initialized. Skip the initialization step
2022-03-15 21:20:50.021  INFO 34 --- [           main] o.a.d.d.upgrade.DolphinSchedulerManager  : upgrade DolphinScheduler metadata version from 2.0.3 to 2.0.4
2022-03-15 21:20:50.021  INFO 34 --- [           main] o.a.d.d.upgrade.DolphinSchedulerManager  : Begin upgrading DolphinScheduler's table structure
2022-03-15 21:20:50.021  INFO 34 --- [           main] o.a.d.dao.upgrade.UpgradeDao             : pre_workflow
2022-03-15 21:20:50.024  INFO 34 --- [           main] o.a.d.common.utils.ScriptRunner          : sql: /*  * Licensed to the Apache Software Foundation (ASF) under one or more  * contributor license agreements.  See the NOTICE file distributed with  * this work for additional information regarding copyright ownership.  * The ASF licenses this file to You under the Apache License, Version 2.0  * (the "License"); you may not use this file except in compliance with  * the License.  You may obtain a copy of the License at  *  *    http://www.apache.org/licenses/LICENSE-2.0  *  * Unless required by applicable law or agreed to in writing, software  * distributed under the License is distributed on an "AS IS" BASIS,  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  * See the License for the specific language governing permissions and  * limitations under the License. */ ALTER TABLE `t_ds_task_instance` ADD INDEX `idx_code_version` (`task_code`, `task_definition_version`) USING BTREE
2022-03-15 21:20:50.046 ERROR 34 --- [           main] o.a.d.common.utils.ScriptRunner          : SQLException

java.sql.SQLSyntaxErrorException: Duplicate key name 'idx_code_version'
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:782) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:666) ~[mysql-connector-java-8.0.16.jar:8.0.16]
	at com.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94) ~[HikariCP-4.0.3.jar:na]
	at com.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java) ~[HikariCP-4.0.3.jar:na]
	at org.apache.dolphinscheduler.common.utils.ScriptRunner.runScript(ScriptRunner.java:117) [dolphinscheduler-common-2.0.5.jar:2.0.5]
	at org.apache.dolphinscheduler.common.utils.ScriptRunner.runScript(ScriptRunner.java:72) [dolphinscheduler-common-2.0.5.jar:2.0.5]
	at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.upgradeDolphinSchedulerDDL(UpgradeDao.java:333) [dolphinscheduler-dao-2.0.5.jar:2.0.5]
	at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.upgradeDolphinScheduler(UpgradeDao.java:125) [dolphinscheduler-dao-2.0.5.jar:2.0.5]
	at org.apache.dolphinscheduler.dao.upgrade.DolphinSchedulerManager.upgradeDolphinScheduler(DolphinSchedulerManager.java:105) [dolphinscheduler-dao-2.0.5.jar:2.0.5]
	at org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler$CreateRunner.run(CreateDolphinScheduler.java:57) [dolphinscheduler-dao-2.0.5.jar:2.0.5]
	at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:791) [spring-boot-2.5.6.jar:2.5.6]
	at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:775) [spring-boot-2.5.6.jar:2.5.6]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:345) [spring-boot-2.5.6.jar:2.5.6]
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:143) [spring-boot-2.5.6.jar:2.5.6]
	at org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler.main(CreateDolphinScheduler.java:40) [dolphinscheduler-dao-2.0.5.jar:2.0.5]

@caishunfeng Related. The problem occurs when deploying through kubernetes clustering. Recently tested, it happens when upgrading from 2.0.2 to 2.0.5
After a simple analysis of the code, the tracking is as follows:

Thanks for your feedback, and I think remove initDatabase in api-server is better, what do you think?

@caishunfeng There are two ways I think of, one is to remove the initDatabase as you said, but it means that it needs to be executed manually in the future, and the other can consider abstracting the stored procedure, such as CreateIndexIfNotExists, CreateUniqueKeyIfNotExists, DropIndexIfExists

@caishunfeng I think initDatabase() is a convenience to help users automatically step up from a very low version to a higher version. Otherwise, the user needs to check the current version in t_ds_version, and then manually execute DDL for each subsequent version

@caishunfeng There are two ways I think of, one is to remove the initDatabase as you said, but it means that it needs to be executed manually in the future, and the other can consider abstracting the stored procedure, such as CreateIndexIfNotExists, CreateUniqueKeyIfNotExists, DropIndexIfExists

What about split initDatabase as a alone step in docker sh? like

case "$1" in
(all)
...
waitDatabase
initDatabase
.....
;;
(initDatabase)
initDatabase
;;
(api-server)
...
waitDatabase
...
;;


text:
[Feature] Support array function sequence like presotdb

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
It is used to generate a series of integers or dates
Document Connection: https://prestodb.io/docs/current/functions/array.html?highlight=sequence
we had alreay support array_range to generate int，should also support date/datetime
Use case
No response
Related issues
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I will try, PLZ assign it to me~

authors: HappenLee superdiaodiao

repo: apache/doris


text:
[Enhancement] task worker pool name should be distinguished in top -H

Search before asking

[X] I had searched in the issues and found no similar issues.

Description
When a be is running, we can distinguish the thread name of taskworkerpool. The default name is too long.
be/src/agent/task_worker_pool.cpp
Solution
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

i'd like to try

i'm willing to try

@dataroaring  Hi, I want to try this issue. I read the source file and did not understand clearly the problem. Could you please make a detailed example? (Like, from what to what?) Thanks! :)

@dataroaring Hi, I want to try this issue. I read the source file and did not understand clearly the problem. Could you please make a detailed example? (Like, from what to what?) Thanks! :)

TaskWorkerPool::TaskWorkerPool(std::string_view name, int worker_count,
std::function<void(const TAgentTaskRequest& task)> callback)
: _callback(std::move(callback)) {
auto st = ThreadPoolBuilder(fmt::format("TaskWorkerPool.{}", name))
.set_min_threads(worker_count)
.set_max_threads(worker_count)
.build(&_thread_pool);
CHECK(st.ok()) << name << ": " << st;
}
Here TaskWorkerPool.xxx is name of the worker thread. However, when we use top -H in linux, thread num is cut, so we just see TaskWor. We can name it TWP{name}, names also should be short.

i have no experience, can i try?


text:
fix: fix SemiJoinLogicalJoinTransposeProject.

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

[x] Does it affect the original behavior
[x] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[x] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.76 seconds
stream load tsv:          468 seconds loaded 74807831229 Bytes, about 152 MB/s
stream load json:         35 seconds loaded 2358488459 Bytes, about 64 MB/s
stream load orc:          68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet:          29 seconds loaded 861443392 Bytes, about 28 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230218123727_clickbench_pr_99609.html

authors: hello-stephen jackwener


text:
Fix Fix switch-case fall-through issue in multi-catalog module.

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

[ ] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[ ] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run clickbench

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.95 seconds
stream load tsv:          469 seconds loaded 74807831229 Bytes, about 152 MB/s
stream load json:         39 seconds loaded 2358488459 Bytes, about 57 MB/s
stream load orc:          69 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet:          29 seconds loaded 861443392 Bytes, about 28 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230220133047_clickbench_pr_100319.html

authors: BiteTheDDDDt hello-stephen kaka11chen


text:
[bug][auth]Show grant causes role errors in the memory. #24783

Proposed changes
Issue Number: close #24783

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

build all

run buildall


text:
base-3da7ab541

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run p0


text:
feature enable merge-on-write by default again

Proposed changes
fix #27188, #28096
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run feut

run p0

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

run external


text:
fixFix no auth,but can select count(*)

Proposed changes
Issue Number: close #xxx
when select count(*),cols is empty, should check table priv
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

authors: morrySnow zddr


text:
feature support new function replace_empty (#36283)

#36283

run buildall

run buildall

run buildall


text:
Fix Fix delete on value case due to compaction

Proposed changes
Issue Number: close #xxx

run buildall


text:
Pick some pr to branch 3.0  #41674 #43095 #44164

What problem does this PR solve?
6fdc8a59
https://github.com/apache/doris/pull/41674
7c45912e
https://github.com/apache/doris/pull/43095
7aec6ffb
https://github.com/apache/doris/pull/44164
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall


text:
fixFix incorrect tables in CascadesContext

What problem does this PR solve?
Previously copied the tables used in the last query from connectContext
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Fix incorrect tables in CascadesContext
Release note
Fix incorrect tables in CascadesContext
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall


text:
fix improve error msg

What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.
Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?


text:
opt add summary metric for file scanner

What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[ ] No.
[ ] Yes.

Does this need documentation?

[ ] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.
Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

run buildall

authors: hello-stephen morningman


text:
STDDEV throws exception when combined with a FILTER clause that returns no rows

Trivial example of what I'm talking about:
SELECT STDDEV("field1") FILTER(WHERE 1=2), "field2" FROM my_table GROUP BY 2;
If the FILTER clause happens to filter all rows for a given GROUP BY group, then druid throws a java.lang.IllegalStateException: should not be empty holder exception.  I'd expect it to instead give back a NaN value or something along those lines.

I checked back in on this after a long time, and it seems resolved now.

repo: apache/druid


text:
Log exception while mapping SupervisorSpec

A bad supervisor spec can prevent overlord from becoming leader. This change is to skip and log such spec which couldn't be mapped.
This exception from our prod cluster occurs when the supervisor spec points to a protobuf message which doesn't exist.
Caused by: java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `org.apache.druid.data.input.protobuf.FileBasedProtobufBytesDecoder`, problem: Protobuf message type Record not found in the specified descriptor.  Available messages types are [***]
 at [Source: (byte[])"{"type":"kafka","spec":{"dataSchema":{"dataSource":"***","timestampSpec":{"column":"event_timestamp","format":"iso","missingValue":null},"dimensionsSpec":{"dimensions":[{"type":"string","name":"app_name","multiValueHandling":"SORTED_ARRAY","createBitmapIndex":true},{"type":"string","name":"app_version","multiValueHandling":"SORTED_ARRAY","createBitmapIndex":true},{"type":"string","name":"core_sdk_version","multiValueHandling":"SORTED_ARRAY","createBitmapIndex":true},{"ty"[truncated 35925 bytes]; line: 1, column: 16331] (through reference chain: org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec["spec"]->org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIngestionSpec["ioConfig"]->org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorIOConfig["inputFormat"]->org.apache.druid.data.input.protobuf.ProtobufInputFormat["protoBytesDecoder"])
	at org.apache.druid.metadata.SQLMetadataSupervisorManager$5$2.map(SQLMetadataSupervisorManager.java:262) ~[druid-server-2023.04.0-iap.jar:2023.04.0-iap]
	at org.apache.druid.metadata.SQLMetadataSupervisorManager$5$2.map(SQLMetadataSupervisorManager.java:246) ~[druid-server-2023.04.0-iap.jar:2023.04.0-iap]
	at org.skife.jdbi.v2.Query$4.munge(Query.java:183) ~[jdbi-2.63.1.jar:2.63.1]
	at org.skife.jdbi.v2.QueryResultSetMunger.munge(QueryResultSetMunger.java:43) ~[jdbi-2.63.1.jar:2.63.1]
	at org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1340) ~[jdbi-2.63.1.jar:2.63.1]
	at org.skife.jdbi.v2.Query.fold(Query.java:173) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataSupervisorManager$5.withHandle(SQLMetadataSupervisorManager.java:266) ~[druid-server-2023.04.0-iap.jar:2023.04.0-iap]
	at org.apache.druid.metadata.SQLMetadataSupervisorManager$5.withHandle(SQLMetadataSupervisorManager.java:232) ~[druid-server-2023.04.0-iap.jar:2023.04.0-iap]
	at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
	at org.apache.druid.metadata.SQLMetadataSupervisorManager.getLatest(SQLMetadataSupervisorManager.java:230) ~[druid-server-2023.04.0-iap.jar:2023.04.0-iap]
	at org.apache.druid.indexing.overlord.supervisor.SupervisorManager.start(SupervisorManager.java:127) ~[druid-indexing-service-2023.04.0-iap.jar:2023.04.0-iap]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-processing-2023.04.0-iap.jar:2023.04.0-iap]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-processing-2023.04.0-iap.jar:2023.04.0-iap]
	at org.apache.druid.indexing.overlord.TaskMaster$1.becomeLeader(TaskMaster.java:176) ~[druid-indexing-service-2023.04.0-iap.jar:2023.04.0-iap]
	... 5 more

@findingrish , could you please update the description to indicate why this change is needed?

@findingrish , you might need to merge from latest master to fix the failing tests.

authors: findingrish kfaraz


text:
[3.0] Do not release ByteBuf manually in port unification handler

What is the purpose of the change
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
        at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
        at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
        at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100
)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:288)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.j
ava:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.j
ava:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.jav
a:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)

Brief changelog
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #9869 (7d70703) into 3.0 (225c208) will decrease coverage by 0.38%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                3.0    #9869      +/-   ##
============================================
- Coverage     65.80%   65.41%   -0.39%
- Complexity      297      317      +20
============================================
  Files          1213     1213
  Lines         52847    52810      -37
  Branches       8008     7981      -27
============================================
- Hits          34775    34548     -227
- Misses        14290    14450     +160
- Partials       3782     3812      +30

Impacted Files
Coverage Δ

...bbo/remoting/api/PortUnificationServerHandler.java
55.00% <ø> (+1.34%)
:arrow_up:

.../rpc/protocol/tri/transport/TripleTailHandler.java
50.00% <0.00%> (-50.00%)
:arrow_down:

...o/rpc/protocol/tri/transport/GracefulShutdown.java
50.00% <0.00%> (-35.72%)
:arrow_down:

...dubbo/remoting/zookeeper/ZookeeperTransporter.java
55.55% <0.00%> (-33.34%)
:arrow_down:

...ng/zookeeper/curator5/Curator5ZookeeperClient.java
43.47% <0.00%> (-27.06%)
:arrow_down:

.../remoting/api/SingleProtocolConnectionManager.java
61.53% <0.00%> (-23.08%)
:arrow_down:

...nt/metadata/ServiceInstanceHostPortCustomizer.java
65.78% <0.00%> (-21.06%)
:arrow_down:

...mmon/threadpool/support/AbortPolicyWithReport.java
70.17% <0.00%> (-14.04%)
:arrow_down:

...ache/dubbo/remoting/transport/AbstractChannel.java
75.00% <0.00%> (-12.50%)
:arrow_down:

...in/java/org/apache/dubbo/common/utils/JVMUtil.java
79.59% <0.00%> (-12.25%)
:arrow_down:

... and 55 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 225c208...7d70703. Read the comment docs.

authors: codecov-commenter guohao

repo: apache/dubbo


text:
折线图 坐标轴类型time，使用visualMap分段显示，分段处渐变色

Version
4.1.0-release
Reproduction link
https://gallery.echartsjs.com/editor.html?c=xXp-lCudId&v=1
Steps to reproduce
Gallery 用例中的option即可复现。增加 递增的visualMap范围会使颜色位置发生移动。
What is expected?
期望分段接线明显，切不会随着时间范围外的visualMap中的item增加影响颜色位置
What is actually happening?
分段处为颜色渐变，visualMap增加会使颜色左右移动，甚至整条线为紫色

按照时间分段显示折线图

不知道我理解得对不对，你使用的 piecewise 就是渐变的，如果不要渐变就用 continuous。如果不对的话，能不能再说清楚一点问题是什么？可能需要借助图片标注一下？

https://gallery.echartsjs.com/editor.html?c=xI66JhpOSe&v=2

@Ovilia

不太明白，https://gallery.echartsjs.com/editor.html?c=xI66JhpOSe&v=2 是你要的效果？那不是解决了吗？问题是什么？

有两个问题：
1、piecewise 中多余的项不在显示的时间轴范围内，为什么会影响显示，这个有问题吧。
2、在点数特别多的情况下，即使没有多余的piecewise ，还是会出现渐变色，显示位置不准的问题。

@Ovilia

「piecewise 中多余的项」哪个是多余的项？
例子？

当存在多个 pieces 的时候确实存在这个问题?
https://gallery.echartsjs.com/editor.html?c=xXp-lCudId&v=3
https://gallery.echartsjs.com/editor.html?c=xXp-lCudId&v=4
唯一的区别就是pieces里多了一个值图就糊了

我也遇到了同样的问题，请问解决了嘛？？？

遇到相同问题了，多个pieces导致连接点处出现渐变色。应该不是渐变才对啊。

Encountered the same problem, multiple pieces caused a gradient at the connection point. It's not a gradual change.

when i want to set partial line to different color,  i also encounter this issues. i have manually calculated cross point of two line and generate the right option.piece, but get gradient color change around the cross point.

您的邮件已收到，我会尽快处理！.

您的邮件已收到，我会尽快处理！.

authors: EliotChenKJ Ovilia XDaoHong YouYII Zzb95 ccxgl gqiye

repo: apache/echarts


text:
Doesn't clear "series-line.emphasis.scale" during draging  or zooming

Version
5.0.2
Steps to reproduce
Click on a point and start moving the chart (or scale).
myChart.setOption({
    series:[{
        type: 'line',
        symbolSize: 0.1,
        emphasis: {
            scale: true,
            lineStyle: {
                width: 1,
            },
        },
        lineStyle: {
                width: 1,
        },
        data: Array.from(new Array(100)).map((x, i) => [i, Math.floor(Math.random() * (10 - 1 + 1)) + 1]),
    }],
    xAxis:[{type:'value'}],
    yAxis:[{type:'value'}],
    tooltip: {
        trigger: 'axis',
    },
    dataZoom: [{
        type: 'inside',
    }]
})

What is expected?
Old points are cleared.

What is actually happening?
Old points aren't cleared.

HI guys. Any news?


text:
[Feature] Arbitrary Mesh Display

What problem does this feature solve?
eCharts GL and 3D rendering is very restricted at the moment when wanting to visualise 3D data outside of scatter/plot or a very limited "polygon" (which is really just extruded 2D shape)
As web browsers and WebGL get more powerful,  there is opportunity to visualise data in new ways in 3D.
eCharts would be improved in their 3D/GL area with the ability to have arbitrary mesh support:
Use cases:

Medical application where complex 3D models of organs can be rendered
Mining and other large scale (geographically) projects needing to visualise spatial data
Arbitrary meshes as MarkPoints (a 3D human mesh,  a 3D car mesh on a road,  other 3D markers)
Markpoints and Markareas as sub-meshes ,   think of a medical application where certain surface patches have Tooltip data

What does the proposed API look like?
In the Polygon series type ( https://echarts.apache.org/en/option-gl.html#series-polygons3D.data )
You can only set XY of the data and then a constant height for everything.
It would be great to support rich 3D mesh data,  here is simple proposal of a triangle
data:[
{
vertices: [[0,0,0],  [0,1,0],  [0,0,1]],
triangles: [ [0,1,2] ] //  Indexes of the vertex buffer with a stride of 3
uv:  [[0,0],  [0,0.5],  [0.5,0]   //  UV coordinates of vertices for texture mapping
}
]
How we feed these arrays could then be arbitrary...  from a proceedural process,  or by processing a popular 3D file (STL, FBX, OBJ)
Generally, the mesh data could include

Sub-meshes to denote different materials (  data:[ {meshA},  {meshB} ]  )
Multi-stream Vertex (position, normal, UV, tangent, bitangent, RGBA color)
Tooltip support?  (based on submesh?  or different polygons ,  vertices, edges?)

Hello - was curious if there were any thoughts on this or if there were any plans for the GL component of eCharts looking forward?

sorry to bump again,  would be great to hear of any roadmap or upcoming features for GL and eCharts without needing to move this part of our application to a different library as we have built a nice framework with eCharts

shame there's no comments on this?
the GL pack of eCharts is a promising start and would be great if it were extended rather than having to jump to another package (ie. we can still use much of the eCharts Instance/boilerplate code and tooltips... etc)
is there any roadmap at all for eCharts (not just GL) ?

Bumping this as well, that would be a great feature to have!

authors: astanziola urgrund


text:
[Feature] show minimum value in bar chart

What problem does this feature solve?
when we have bar chart whit outlier data for example green : 1000 , and blue : 1 ,we have serious issue in stacked bar chart ,and bar chart just show green because is too long and doesn't show blue just show label
i don't need markPoint because i need to show integrated chart

What does the proposed API look like?
i think just like minBarWidth and minBarHeight we can have something like that to set default width or height and just set to minimum or very low values.
Thank you

Stacked bar with minHeight may cause logical misleading problems so it's not considered.


text:
堆叠柱状图重合问题，bug？

One-line summary [问题简述]
绘制堆叠柱状图时发现用同样的dataset数据绘制stack bar时，根据数据矩阵各列与series的映射关系不同，会出现本该堆叠的柱状图彼此上下覆盖的现象。求解决方法，多谢！！！
Version & Environment [版本及环境]

ECharts version [ECharts 版本]:4.0.2  或官网demo的在线编辑模式
Browser version [浏览器类型和版本]: chrome Version 63.0.3239.132 (Official Build) (64-bit)
OS Version [操作系统类型和版本]:win10

Expected behaviour [期望结果]
柱状图以堆叠方式绘制，每个图块在前一图块的末尾开始绘制，如图所示

ECharts option [ECharts配置项]

var data = [

    //此数据的绘图结果不正常，将鼠标悬浮于各个图例上可看出各个颜色图块的实际起始，截止位置。可看出并未堆叠，而是以某种方式相互在z轴上覆盖
    ["Twitter_ID", "UserName",  "Like", "Reply","Share"]
    ,["evannex_com", "EVANNEX for Tesla", 6381, 15366, 461]
    ,["TeslaModel3News", "Tesla Model 3 News", 2878, 9876, 422]
    ,["Model3Owners", "Model 3 Owners Club", 1773, 9571, 823]
    ,["TeslaPittsburgh", "Tesla Pittsburgh", 635, 2118, 190]
    ,["XingleiShen", "Xinglei Shen", 585, 2223, 108]
    ,["techeblog", "TechEBlog", 796, 759, 0]
    ,["TeslaClubUK", "Tesla Owners Club UK", 611, 777, 159]
    ,["omg_tesla", "Omg_Tesla", 306, 1128, 97]
    ,["JimHarris", "Jim Harris", 644, 764, 35]
    ,["jassummers", "Jas Summers", 220, 800, 75]
    ,["ubuygas", "UBUYGAS", 347, 510, 34]
    ,["ElectriCarsES", "ElectriCars", 28, 714, 70]
    ,["bmwblog", "BMWBLOG", 188, 578, 24]
    ,["plugshare", "PlugShare", 152, 501, 26]
    ,["Jasonrbirchall", "Jason R Birchall", 96, 332, 170]
    ,["AmazingChevVolt", "Thomas J. Thias", 247, 287, 23]
    ,["kirillklip", "Kirill Klip", 387, 153, 9]
    ];

    //此数据绘制结果正常，可看出各个颜色图块分别是在前一图块绘制截止的位置后开始绘制的，堆叠效果正常
    //与上面数据的区别仅是改变了第一个数组中share, like ,reply的位置
    //使用上面显示正常的数据，在series中改变share like reply的映射顺序也会出现同样的问题
    // ["Twitter_ID", "UserName", "Share", "Like", "Reply"]
    // ,["evannex_com", "EVANNEX for Tesla", 6381, 15366, 461]
    // ,["TeslaModel3News", "Tesla Model 3 News", 2878, 9876, 422]
    // ,["Model3Owners", "Model 3 Owners Club", 1773, 9571, 823]
    // ,["TeslaPittsburgh", "Tesla Pittsburgh", 635, 2118, 190]
    // ,["XingleiShen", "Xinglei Shen", 585, 2223, 108]
    // ,["techeblog", "TechEBlog", 796, 759, 0]
    // ,["TeslaClubUK", "Tesla Owners Club UK", 611, 777, 159]
    // ,["omg_tesla", "Omg_Tesla", 306, 1128, 97]
    // ,["JimHarris", "Jim Harris", 644, 764, 35]
    // ,["jassummers", "Jas Summers", 220, 800, 75]
    // ,["ubuygas", "UBUYGAS", 347, 510, 34]
    // ,["ElectriCarsES", "ElectriCars", 28, 714, 70]
    // ,["bmwblog", "BMWBLOG", 188, 578, 24]
    // ,["plugshare", "PlugShare", 152, 501, 26]
    // ,["Jasonrbirchall", "Jason R Birchall", 96, 332, 170]
    // ,["AmazingChevVolt", "Thomas J. Thias", 247, 287, 23]
    // ,["kirillklip", "Kirill Klip", 387, 153, 9]
    // ];

 //此数据绘制结果不正常，share与like是以堆叠的方式绘制的，reply则是从share的截止位置开始绘制，覆盖了like
//     ['Time', 'Share','Like','Reply']
//     ,['2017-08-10',436,45,45]
//     ,['2017-08-09',46,45,425]
//     ,['2017-08-08',36,45,420]
//     ,['2017-08-07',136,15,13]
//     ,['2017-08-06',36,15,12]

//     ,['2015-06-01',56,55,52]
//     ,['2014-08-10',36,35,35]
//     ,['2014-08-09',26,25,28]
//     ,['2014-08-08',36,454,423]
//     ,['2014-08-07',236,25,24]
//     ,['2014-08-06',26,252,28]
//     ,['2014-06-05',236,255,25]
//     ,['2014-06-04',26,25,27]
//     ,['2014-06-03',536,55,542]
//     ,['2014-06-02',56,55,54]
//     ,['2014-06-01',56,55,53]
// ];

option = {
    legend:{},
    tooltip : {
        trigger: 'axis',
        axisPointer : {            // 坐标轴指示器，坐标轴触发有效
            type : 'shadow'        // 默认为直线，可选为：'line' | 'shadow'
        }
    },

    grid: {
        left: '20%',
        right: '4%',
        bottom: '3%',
        containLabel: true
    },

    xAxis:  {

    },
    yAxis: {
        type: 'category',

    },
    dataset:{
      source:data
    },
    series: [
        {

            type: 'bar',
            stack: '总量',
            label: {
                normal: {
                    show: true,
                    position: 'insideRight'
                }
            },
             name:'Reply',
            encode:{
                x:'Reply',
                y: 'Time'
            }

        },
        {

            type: 'bar',
            stack: '总量',
            label: {
                normal: {
                    show: true,
                    position: 'insideRight'
                }
            },
            name:'Share',
            encode:{
                x:'Share',
                y: 'Time'
            }

        },
        {

            type: 'bar',
            stack: '总量',
            label: {
                normal: {
                    show: true,
                    position: 'insideRight'
                }
            },
            name:'Like',
            encode:{
                x:'Like',
                y: 'Time'
            }

        },

    ]
};

Other comments [其他信息]

dataset stack 不能显示正确堆叠图 这个bug 什么时候可以修好 并发布在线？

如何能实现stack是覆盖的功能呢？需要这个feature

authors: amily4555 lcoder skywalkershen