text:
[BEAM-3362] Added a java example that uses stateful processing.

This pull request adds a new class to the java Gaming example. It uses stateful processing to aggregate team scores per team and outputs team name and it's total score after every N points earned. In real world, N could be the total score a team should earn to level up.
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Hi, @kennknowles, @tgroh, can you please take a look?


text:
Add Portable PostCommit test status to PR template

cc: @tweise @robertwb
Please add a meaningful description for your change here

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---
---

@angoenka I would prefer we add this to the existing table. It already has the cell for Python / Flink.

Thanks!
Updated the PR.

source: gharchive/pull-request

authors: angoenka tweise

repo: apache/beam

license_type: permissive

license_source: github-api


text:
Revert "[BEAM-6553] A Python SDK sink that supports File Loads into B…

…Q (#7655)"
This reverts commit cdea885872b3be7de9ba22f22700be89f7d53766.
r: @tvalentyn

Run Python Postcommit

LGTM, thank you.

authors: pabloem tvalentyn


text:
[BEAM-6114] Handle Unsupported Lookup Joins in BeamSql

Added code in BeamSideInputLookupJoinRel which throws UnsupportedOperationException if "OUTER JOINS" are applied when the OUTER side is a Seekable Table.
Added Unit Tests for BeamSideInputLookupJoinRel
Modified and added Javadoc for Join Rels.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @amaliujia

I was on a vocation and will take a look at this PR soon.

run SQL PostCommit


text:
BIGTOP-4209: Add backend API for AI Chat functionality

https://issues.apache.org/jira/projects/BIGTOP/issues/BIGTOP-4209
intern: https://gitee.com/openeuler/opensource-intern/issues/IAG904

sorry

@kevinw66 Some updates: diff

PTAL @kevinw66

repo: apache/bigtop-manager


text:
buildstream leaves behind a lot of files in ~/.cache/buildstream/cas/tmp

See original issue on GitLab
In GitLab by [Gitlab user @abderrahimk] on Jun 21, 2020, 11:23
Summary
After using bst2 for an afternoon, it left a lot of files in ~/.cache/buildstream/cas/tmp (20.5GiB). It's almost as big as ~/.cache/buildstream/cas/objects (29.7GiB)
Steps to reproduce
No idea how to reproduce exactly, but I just notice it in normal use. Where "normal" includes terminating builds (Ctrl-C then t), and interrupting (Ctrl-C) buildstream during initialization (loading elements, etc.). I also noticed that I sometimes get the following warning
WARNING Buildbox-casd didn't exit in time and has been killed

Other relevant information

BuildStream version affected: /milestone %BuildStream_v1.x

In GitLab by [Gitlab user @abderrahimk] on Jun 21, 2020, 20:30
Turns out I was using an old version of buildbox-fuse. Upgrading to 0.0.9 fixes the issue.

In GitLab by [Gitlab user @abderrahimk] on Jun 21, 2020, 20:30
closed

source: gharchive/issue

repo: apache/buildstream


text:
Plugins can easily corrupt cache

See original issue on GitLab
In GitLab by [Gitlab user @valentindavid] on Nov 8, 2018, 11:28
Summary
In #749, we have seen that the script plugin was able to corrupt the cache. This was a bug in scriptelement.py. However, it is not good that the core of BuildStream was not able to catch that before the corruption happened.
It is easy to write a plugin that has the same issue using the API.
Steps to reproduce
Just mark a directory as non artifact. With sandbox.mark_directory, and stage an artifact anyway in that directory.
What is the current bug behavior?
The modified files in that directory will be modified in the cache directly.
What is the expected correct behavior?
Either we combine staging and marking directory together in the API, or we check during staging that we put files in directories marked for artifacts.
Relevant logs and/or screenshots
Possible fixes
Other relevant information

BuildStream version affected: /milestone %"BuildStream_v1.4"

In GitLab by [Gitlab user @toscalix] on Nov 12, 2018, 09:49
marked this issue as related to #749

In GitLab by [Gitlab user @toscalix] on Nov 12, 2018, 09:50
changed the description

In GitLab by [Gitlab user @tristanvb] on Nov 12, 2018, 09:55
Hopefully we can avoid spending effort on this and trust that BuildBox will handle this safely all the time.
Keeping this open for now, if we do not land BuildBox in BuildStream by 1.4 (see #719), it is a possibility to spend effort here during the 1.4 cycle. That effort would involve identifying cases where artifact files are staged to read/write locations without safe hardlink protection, and raise an error in those cases.


text:
YamlCache reads the entire file's contents to see if the file is in the cache

See original issue on GitLab
In GitLab by [Gitlab user @jonathanmaw] on Feb 1, 2019, 15:00
Background
YamlCache is a class that speeds up the parsing and provenancing of yaml by maintaining a cache of yaml parsing that is persistent across BuildStream executions.
Currently, it reads all of a file's contents to decide whether it has changed since it was put in the cache.
There should be an easier way to do that - statting a file and comparing its mtime works in most cases.
There are currently three distinct cases where we load yaml, listed in order of most to least common:

Element file from a checked-out project. mtime is very suitable for this.
Element file from a junctioned project. mtime is not suitable for this because these files are a checked-out Source each buildstream invocation.
Element file from a workspace of a junctioned project. mtime should be suitable for this, but I am not sure whether it actually loads the file from the workspace.

Task description

[ ] Write a test case to check whether changes in a workspace of a junction are being picked up.

[ ] If not being read, fix this.

[ ] Use the file's mtime when checking for changes in an element in a checked-out project, or a workspaced junctioned project.
[ ] Use the junctioning element's cache key when checking for changes in an element in a junctioned project - plus potentially changes to the Element API if acquiring this information breaks encapsulation.
[ ] Ensure that the yamlcache is being exercised for element checkouts, junctions and workspaced junctions with tests.

Acceptance Criteria
Unless completely unavoidable, YamlCache does not need to read the contents of a file to know whether it is cached.

In GitLab by [Gitlab user @jonathanmaw] on Feb 1, 2019, 15:56
I've tested these changes on linux, and on WSL, and any difference is extremely marginal.
In the files below, "no-read" is with my changes, and "with-read" is from master.
no-read-no-cache-show.cprofile
no-read-with-cache-show.cprofile
with-read-with-cache-show.cprofile
with-read-no-cache-show.cprofile
I suspect this just indicates that the major slowdown isn't from WSL, but is from not using an SSD.
I will retry these changes later on spinning rust over USB and see if there is a more pronounced improvement.

In GitLab by [Gitlab user @juergbi] on Feb 1, 2019, 16:00
Have you tried this with cold page/dentry cache?
echo 3 > /proc/sys/vm/drop_caches
This might not be supported on WSL, though, in which case you may have to reboot before each test to ensure a cold page/dentry cache.

In GitLab by [Gitlab user @jonathanmaw] on Feb 4, 2019, 15:54
mentioned in merge request !1126

In GitLab by [Gitlab user @jonathanmaw] on Feb 4, 2019, 17:24
I've tested these changes on WSL with the project directory on the external disk over USB, and the results are no better.
master-no-yamlcache.cprofile
master-with-yamlcache.cprofile
no-read-no-yamlcache.cprofile
no-read-with-yamlcache.cprofile
It would seem that there's an equal amount of latency to read the contents of the file compared to statting the file.
Next, I will test whether a difference exists on native linux (i.e. what if there is equal latency because of the ext4 emulation layer). If I fail to see a performance improvement there, then I will close this issue, as there is no meaningful performance gain from reading the inode over reading the contents.

In GitLab by [Gitlab user @jonathanmaw] on Feb 5, 2019, 12:49
On native linux, with the project on spinning rust over USB, with drop_caches, the time saving is minor (~2s of a ~40s runtime)
master-no-cache.cprofile
master-with-cache.cprofile
no-read-no-cache.cprofile
no-read-with-cache.cprofile

In GitLab by [Gitlab user @jonathanmaw] on Feb 5, 2019, 15:11
unassigned [Gitlab user @jonathanmaw]

In GitLab by [Gitlab user @jonathanmaw] on Feb 5, 2019, 15:12
Given the benefits of this optimisation are very small, I'm putting this issue down and looking at #895 instead

In GitLab by [Gitlab user @tristanvb] on Feb 5, 2019, 17:03

Given the benefits of this optimisation are very small, I'm putting this issue down and looking at #895 instead.

Fair enough, I have a suspicion that the benefits of this optimization are only very small because the hugest bst projects are fairly small (perhaps the cost of a stat() is similar to the full read of a file that is small enough to be transfered in a single buffer ?).
Judging from the performance gained by fixing workspaces to use mtimes instead of exhaustively (and quite needlessly) hashing file content, we will gain a lot of performance here for any project which consists of about 6GB of .bst files (e.g., like a built WebKit workspace).
Perhaps it's still worth swapping out the hashing algorithm for a local mtime approach too, just for the peace of mind that this wont cost more for larger files.

In GitLab by [Gitlab user @jonathanmaw] on Feb 15, 2019, 18:02
I had another look, this time about whether there are improvements when loading yaml from junctioned elements. I had the suspicion it might be a useful improvement because we don't need to make any syscalls to know whether it's cached (we already have the junction element's cache key).
master-with-cache.cprofile
no-read-with-cache.cprofile
There is no appreciable improvement on my linux laptop. It might still be worthwhile if there's a decent improvement on WSL.

In GitLab by [Gitlab user @danielsilverstone-ct] on Mar 29, 2019, 08:33
Since !1257 merged, we no longer have a YamlCache so this issue can be closed.

In GitLab by [Gitlab user @danielsilverstone-ct] on Mar 29, 2019, 08:33
closed


text:
Retrying failed builds is important, but difficult for new users

See original issue on GitLab
In GitLab by [Gitlab user @aevri] on Mar 20, 2019, 16:20
Summary
In theory, users should have fully reproducible builds, and it is not necessary to retry builds.
In practice there are numerous issues that lead to a need for retrying:

Race conditions in the build.
Limited memory.
Limited disk space.
etc.

Here are the difficulties with retrying as I see them, will create separate tickets depending on how this discussion goes:

Retrying a build with 'r' does not try to rebuild, it uses the cached failed build. (It retries the 'build job' not the build).
A subsequent 'bst build' fails immediately, perhaps it could offer 'retry' and 'logs' options as with first run.
'bst build' failure doesn't signpost commands that can help from there, e.g. 'artifact logs' and 'artifact delete'.

How to repro:
git clone https://gitlab.com/buildstream/buildstream.git
cd buildstream/doc/examples/running-commands
bst build hello.bst
# -- Should be ok up to here --

echo "  - sleep 10" >> elements/hello.bst
echo "  - 'false'" >> elements/hello.bst

bst build hello.bst
# .. wait 10 seconds for it to fail ..

# Choose:
#  (r)etry     - Retry this job
# .. instantly fail again, no 10 second wait ..

bst build hello.bst
# .. instantly fail again, with only the failed command mentioned, logs would be nice ..

bst artifact log hello.bst
# Ok, now I can see the log, but I didn't get a signpost from the 'bst build' instafail.

bst artifact delete hello.bst
bst build hello.bst
# Hurrah, I retried :)

In GitLab by [Gitlab user @aevri] on Mar 20, 2019, 16:27
[Gitlab user @cs]-shadow has expressed some interested in this

In GitLab by [Gitlab user @aevri] on Oct 28, 2019, 13:34
unassigned [Gitlab user @aevri]

In GitLab by [Gitlab user @abderrahimk] on Jun 23, 2020, 12:13
mentioned in merge request !1970

In GitLab by [Gitlab user @marge-bot123] on Jul 29, 2020, 16:48
mentioned in commit 7f0e0205c066f6ad3e4cb2cb8d719c8022d68ab9


text:
Add spring-boot example in the examples directory

It would be good if there was a Spring Boot @SpringBootApplication java file which you can quickly demo via the new spring boot support.

Spring boot support is very limited and we do not support adding your own main, what we do is running your route inside a spring boot application so you can leverage all the component auto configuration options that camel provides.

spring-boot support is gogin to be removed #534

authors: davsclaus lburgazzoli

repo: apache/camel-k


text:
fix: Consistent integration platform settings hierarchy

Users are allowed to create namespace local IntegrationPlatform resources in order to overwrite settings used in the operator.
These namespace local platform settings should derive from the IntegrationPlatform being used by the operator by default. This means that only explicitly specified settings do overwrite the platform defaults used in the operator.
This PR makes sure that we use a consistent platform settings hierarchy where the global operator platform settings always represent the basis for user specified platform settings.
The changes in this PR:

Makes sure to apply global integration platform settings when creating namespace local platform
Global platform settings resist when the setting is not explicitly overwritten in the given user namespace integration platform

Release Note
fix: Consistent integration platform settings hierarchy

Please, mind the conflict. Some other PR changing the same file was merged in the while.

@squakez thx, rebase is done. Now we need to run the full workflow jobs once again 😅

authors: christophd squakez


text:
CAMEL-19749: Introduce variables to Camel

Description

Target

[ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ ] I checked that each commit in the pull request has a meaningful subject line and body.

[ ] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

This is the initial work to get the basics of variables into Camel. I wanted to get a PR in shape before too much activity happens.
What can come in new PRs is work to make variables easier to be used from a few common EIPs such as from / to / toD and other feedback

A little example at
https://github.com/apache/camel-kamelets-examples/tree/main/jbang/variables

repo: apache/camel


text:
CAMEL-10498 Update Salesforce component to support approval REST API

This PR adds support for getting and sending approvals for
processing via Salesforce REST API[1].
Two new operations were added approvals to fetch any approvals already
in progress, and approval to initiate approval process on the supplied
record or records (batch).
For instance, approvals can be simply used as:
...to("salesforce:approvals")
    .split().body()
    .log("${body.entityId} - ${body.instanceStatus}")

And to send a record for approval you can use:
...to("salesforce:approval?approval.actionType=Submit&...")
    .log("${body.id} - ${body.instanceStatus}")

The approval operation has the ability to set properties on the
endpoint (let's call that template), via message headers and message
body. These can be combined, to place default values on the endpoint
(template), and runtime values trough headers or message body.
If the message body is an Iterable of ApprovalRequest objects then
they will be submitted in a batch.
[1]
https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/resources_process_approvals.htm

@dhirajsb here is another one for you to review, thank you for your time :smile_cat:

is the other kinds that allow setting options like approval.xxx in the salesforce component?
In general its a bit harder for end users to use this, and also for tooling. IMHO its better to have approvalXXX approvalYYY as options, then its just like any other option.
Also is the other parts of camel-salesforce DTOs auto generated or something? It seems the approval is hand coded.
Also you cannot have different UriPath in the endpoint. There must only be the default ones.
@zregvart I think its best you move the approval stuff as regular endpoint options. eg on OperationName you need to add your new approval/approvals as new actions you can use.

Don't merge, still on the list:

[ ] rebase on current master
[ ] make changes @davsclaus mentioned in review

@zregvart  github shows me that there is a conflict in a file. I wonder if you could see check this and push a commit with a fix.

@davsclaus freshly squashed and rebased, thank you for tackling this :tada:

Thanks @zregvart.
Do you have an ICLA on file with ASF so you can edit the Camel docs yourself. It would be good if you could update the release notes with some comments about all this salesforce stuff you work on
http://camel.apache.org/camel-219-release.html
Here is how to do that ICLA and how to edit the wiki docs
http://camel.apache.org/how-do-i-edit-the-website.html

Thanks for the PR it has been merged. Do you mind closing this?

Thanks @davsclaus :+1:

authors: davsclaus zregvart


text:
Update PathVisitor.java

Fix for mapping from verb HEAD.
Description

Target

[ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses camel-3.x, whereas Camel 4 uses the main branch)

Tracking

[ ] If this is a large change, bug fix, or code improvement, I checked there is a JIRA issue filed for the change (usually before you start working on it).

Apache Camel coding standards and style

[ ] I checked that each commit in the pull request has a meaningful subject line and body.

[ ] I have run mvn clean install -DskipTests locally and I have committed all auto-generated changes

Thanks

authors: egekaraosmanoglu oscerd


text:
CAMEL-14146 : camel-hdfs - Consumer that splits with ChunkSize should add similar header info like the Splitter

CAMEL-14146 : camel-hdfs - Consumer that splits with ChunkSize should add similar header info like the Splitter

Oh I get test errors so I had to revert this, can you take a look
[ERROR] Tests run: 8, Failures: 6, Errors: 0, Skipped: 0, Time elapsed: 1.082 s <<< FAILURE! - in org.apache.camel.component.hdfs.HdfsConsumerTest
[ERROR] org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFile  Time elapsed: 1.041 s  <<< FAILURE!
java.lang.AssertionError:
Expected: is <1>
but: was <0>
at org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFile(HdfsConsumerTest.java:168)
[ERROR] org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFileWithStreamDownload  Time elapsed: 0.005 s  <<< FAILURE!
java.lang.AssertionError:
Expected: is <1>
but: was <0>
at org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFileWithStreamDownload(HdfsConsumerTest.java:219)
[ERROR] org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFile  Time elapsed: 0.004 s  <<< FAILURE!
java.lang.AssertionError:
Expected: is <1>
but: was <0>
at org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFile(HdfsConsumerTest.java:168)
[ERROR] org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFileWithStreamDownload  Time elapsed: 0.005 s  <<< FAILURE!
java.lang.AssertionError:
Expected: is <1>
but: was <0>
at org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFileWithStreamDownload(HdfsConsumerTest.java:219)
[ERROR] org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFile  Time elapsed: 0.004 s  <<< FAILURE!
java.lang.AssertionError:
Expected: is <1>
but: was <0>
at org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFile(HdfsConsumerTest.java:168)
[ERROR] org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFileWithStreamDownload  Time elapsed: 0.004 s  <<< FAILURE!
java.lang.AssertionError:
Expected: is <1>
but: was <0>
at org.apache.camel.component.hdfs.HdfsConsumerTest.doPollFromExistingLocalFileWithStreamDownload(HdfsConsumerTest.java:219)

Also when you work on PRs then maybe separate them into two when you do some cleanup of the code but keep currently logic vs adding new features. As when you mix them together its harder to follow and review PRs when you look at much more code changes. Just a tip

Ah okay that unit test error was also there before on my local osx laptop. Works on CI so its likely a test glitch on mac or something. I have undo this so this PR is merged and included on master.

authors: davsclaus marius-cornescu


text:
CAMEL-15792 + camel-couchbase polishing

This PR migrates existing integration tests to use test-infra and test-containers. Also small polishing was done, such as added parameters such as connectTimeout and fullDocument.
fullDocument if enabled then in consumer is returned full document instead of result defined in view. This was default behavior before. so it wasn't possible to get view result.

Very nice use of the new test infra, @mmelko!

authors: mmelko orpiske


text:
Correct order of returned columns of ResultSetIterator

The columns returned by ResultSetIterator.getColumnNames() were not in the same order as they appeared in the query projection.
To fix this the HashSet used to return the values was changed to a LinkedHashSet.

Thanks for the PR. It has been merged. Please close this

authors: davsclaus ftoresan


text:
Prevent broken concurrent schema pulls

patch by Dimitar Dimitrov and Berenguer Blasi; reviewed by  for CASSANDRA-16856
Co-authored-by: Dimitar Dimitrov dmtrndmtrv@gmail.com
Co-authored-by: Berenguer Blasi berenguerblasi@gmail.com

CI LGTM

repo: apache/cassandra


text:
Cassandra 18911 5.0

The test is failing because, after the refactoring of the SSTable API, key cache entries of the old SSTable are invalidated when the SSTable is deleted. In 4.1, the key cache entries are invalidated as a part of early open operation.
The moment when there is no reference to the SSTable in the system is more relevant for key cache invalidation.

Can you add the CI link please?

https://app.circleci.com/pipelines/github/jacek-lewandowski/cassandra/968/workflows/08fddcc5-ddd2-4d52-94cf-99344cd19c74

+1

authors: bereng jacek-lewandowski


text:
feat: custom image name for init container

What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?

Thanks. This pr is helpful. Celeborn's most PR is connected with a Jira ticket before merging into the main.
Do you need assistance to create a new Jira ticket?

authors: FMX aagumin

repo: apache/celeborn


text:
Issue #27: Implement Tree-view based Domain List View

Hi @rhtyd, @svenvogel. This is the design of the new tree-view. Please let me know your comments.

@utchoang @svenvogel here are some issues from my testing:

On refreshing the view loses context and shows me the root domain, even if the nested domain may be selected (the refresh can happen if I add a new domain or perform an action like update count)
The + plus button the tree should only be shown the domain has haschild set to true (no need to show it otherwise)
In the src/config/section/iam.js for the createDomain action, you can add a mapping rule for parentdomainid (see compute.js for example) to return the parentdomain.id as the record.id, i.e. id of the selected domain
As in the info-card component, can we show additional details about the domain, or at least the button to view account under that domain (see old UI)
Would be great if the file icon could be replace with the domain icon in the tree if that's possible for leaf domains with no children
Remove the settings tab for the account, as it will be dealt by #26

Screenshots may help:

Old UI (refreshing does not reset the view)

Info card for a domain per master branch, see comments and the view accounts button:

screenshot that shows different in the list of rendered keys:

@rhtyd Thanks for your response. I will check and fix it.

@rhtyd @svenvogel It is fixed. Please check. Regarding to the info card, I'm going to make a tab as in the image below. Please let me know what you think

Thanks @utchoang I'll review and merge, maybe post merging do few fixes.

@utchoang @svenvogel Overall good work!
Here a few outstanding issues:

Go to a resource that links to a domain, for example a VM -> click on the domain -> it takes you to the domain resource view (path is something like http://localhost:5050/#/domain/d150293a-eb52-11e9-b6c2-1e00970107cd). Here we see two infocards.
There could be a middleground to
On clicking a domain the URL router does not change, maybe if this is implemented as a view component (instead of a new tree component) it would simplify a lot -> let me know if you think it would be a lot of work for you I can takeover to complete it

@rhtyd Thank for your response. I made change so that the view of the domain resource will not display domain's two info cards

Here a few outstanding issues:

Go to a resource that links to a domain, for example a VM -> click on the domain -> it takes you to the domain resource view (path is something like http://localhost:5050/#/domain/d150293a-eb52-11e9-b6c2-1e00970107cd). Here we see two infocards.
There could be a middleground to
On clicking a domain the URL router does not change, maybe if this is implemented as a view component (instead of a new tree component) it would simplify a lot -> let me know if you think it would be a lot of work for you I can takeover to complete it
@rhtyd lets us change alls things completely so its ready for merge...
@utchoang there are things to do?

alright let me handle and takeover this, I think if we reimplement this as a resource view component we can handle all cases. @utchoang in that case you can move to the next task/ticket like #46 etc. and grant me push access to your PR repo. I'll refactor and merge this, this weekend.

I've made some fixes @utchoang @svenvogel and will merge this. Any further changes or fixes can be done in a new PR.

authors: rhtyd svenvogel utchoang

repo: apache/cloudstack-primate


text:
VR failing health checks on certain checks 4.18

ISSUE TYPE

Bug Report

COMPONENT NAME

VR, UI

CLOUDSTACK VERSION

4.18

CONFIGURATION
advanced networking
OS / ENVIRONMENT
CS management server running on ubuntu 22.04 and hosts running on ubuntu 20.04
SUMMARY

I keep getting ROUTER.HEALTH.CHECKS indicating "has failing checks: dns_check.py, dhcp_check.py". Everything seems to be working. My VMs are online so not sure what to make of it. I'm not sure if I should just exclude those 2 checks from global config or will there be any issues excluding those 2 checks from global config.
STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

this looks same as #8177 #8569
just to confirm, did you update the name of vms ?

this looks same as #8177 #8569
just to confirm, did you update the name of vms ?

I did not update the name of the VM. As a matter of fact, I didn't really do anything. I made VMs, assign it the network and that's about it.

this looks same as #8177 #8569
just to confirm, did you update the name of vms ?

I did not update the name of the VM. As a matter of fact, I didn't really do anything. I made VMs, assign it the network and that's about it.

ok. @Chih80
do the VMs have multiple nics ?

this looks same as #8177 #8569
just to confirm, did you update the name of vms ?

I did not update the name of the VM. As a matter of fact, I didn't really do anything. I made VMs, assign it the network and that's about it.

ok. @Chih80 do the VMs have multiple nics ?

Some does but I doubt this matters. This has been happening from the get-go when all the VMs had one nic.

I checked some similar tickets #8569 #7653 #8177 #8158

all issues happens with 4.18.0 or 4.18.1
shared network (#8177, #7653, #8158). network type is not specified in #8569 and #8761
I suspect it has been fixed by #8741

@Chih80
are you still facing the issue ?
what type of network do you use ?

I recently upgraded my environment to 4.15.2 and I'm seeing a similar issue. I spent some time digging into this a bit more on my system. Here's what I have found.
Works as expected
Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts are fine
Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine
Step 3) Create VM in guest network - DHCP/DNS alerts are fine
- VM is added to "VirtualMachines" in /root/health_checks_data.json
- DNS check - entry for VM is added into /etc/hosts
- DHCP check - entry for VM exists in /etc/dhcphosts.txt file
** Causes failed healthchecks **
Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS checks pass
Step 2) Create VPC Guest Network ** without DHCP **  - DHCP/DNS checks pass
Step 3) Create VM in guest network - DHCP/DNS checks fail
- Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json
- DNS check - Entry for this VM is not added in /etc/hosts
- DHCP check - No /etc/dhcphosts.txt file created
** Causes failed healthchecks **
Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts pass
Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine
Step 3) Create VM in guest network - DHCP/DNS alerts are fine
- Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json
- DNS check - Entry for this VM is added into /etc/hosts
- DHCP check -Entry for this VM exists in /etc/dhcphosts.txt file
Step 4) Create VPC Guest Network ** without DHCP **  - DHCP/DNS alerts pass
Step 5) Create VM in guest network - DHCP/DNS checks fail
- Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json
- DNS check - Entry for this VM is not added in /etc/hosts
- DHCP check - Entry for this VM does not exist in the /etc/dhcphosts.txt file
Does it make sense to add entries for machines in non-DHCP enabled networks to /root/health_checks_data.json ??

I recently upgraded my environment to 4.15.2 and I'm seeing a similar issue. I spent some time digging into this a bit more on my system. Here's what I have found.
Works as expected Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts are fine Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine Step 3) Create VM in guest network - DHCP/DNS alerts are fine - VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - entry for VM is added into /etc/hosts - DHCP check - entry for VM exists in /etc/dhcphosts.txt file
** Causes failed healthchecks ** Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS checks pass Step 2) Create VPC Guest Network ** without DHCP ** - DHCP/DNS checks pass Step 3) Create VM in guest network - DHCP/DNS checks fail - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is not added in /etc/hosts - DHCP check - No /etc/dhcphosts.txt file created
** Causes failed healthchecks ** Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts pass Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine Step 3) Create VM in DHCP enabled guest network - DHCP/DNS alerts are fine - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is added into /etc/hosts - DHCP check -Entry for this VM exists in /etc/dhcphosts.txt file Step 4) Create VPC Guest Network ** without DHCP ** - DHCP/DNS alerts pass Step 5) Create VM in non-DHCP enabled guest network - DHCP/DNS checks fail - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is not added in /etc/hosts - DHCP check - Entry for this VM does not exist in the /etc/dhcphosts.txt file
Does it make sense to add entries for machines in non-DHCP enabled networks to /root/health_checks_data.json ??

@zenbiking
I tried to reproduce the issue, however, I was not able to.
root@r-337-VM:~# cat /root/health_checks_data.json
{
...
        "virtualMachines": "vmName=VPC-natted-001,macAddress=02:01:02:05:00:01,ip=172.21.0.90,dhcp=true,dns=true;vmName=VPC-natted-002-no-dhcp-dns,macAddress=02:01:02:c2:00:01,ip=172.21.1.214,dhcp=false,dns=false;"
    }
}

The health check data indicates that dhcp/dns are not supported (dhcp=false,dns=false) for vm on the network without Dhcp/Dns (refer to #5554).
@zenbiking
can you share the file /root/health_checks_data.json  in the vpc vr ?

authors: Chih80 weizhouapache zenbiking

repo: apache/cloudstack


text:
Deletion of User Data in a Project fails

ISSUE TYPE

Bug Report

COMPONENT NAME

UserData

CLOUDSTACK VERSION

4.19 (Would be present from 4.18 when managed userdata support was added)

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

Try creating a userdata in a project and then attempt to delete it. It fails with the following error:

STEPS TO REPRODUCE

As described above

EXPECTED RESULTS

Should successfully delete userdata created in a project.

ACTUAL RESULTS

Deletion of userdata in a project fails

@Pearl1594 , projectId and account/domainId have never been allowed together. Therefore I am assuming this is a UI bug. Do you agree?
I will look into it.


text:
UI: Fix backup offering buttons for instances in data view

Description
This PR fixes backup offering buttons for instances in data view

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)
[ ] build/CI
[ ] test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[x] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[x] Minor
[ ] Trivial

Screenshots (if appropriate):
Image before the changes for a instance with a backup offering

Image after the changes for a instance with a backup offering

How Has This Been Tested?
Assign a backup offering to an instance and check the buttons in the data view.

How did you try to break this feature and the system with this change?

@blueorangutan ui

@vishesh92 a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/9391 (QA-JID-403)

Verified fix in qa simulator env.

code looks good in qa but removing the offering doesn't work:

I am not sure if the icon should be added if the removal of the offering is not allowed.

authors: DaanHoogland blueorangutan sureshanaparti vishesh92


text:
CLOUDSTACK-8964: Ovm3HypervisorGuru wrong answer and snapshot to volume implementation.

Fixes the wrong answer that is given for host delegation related to CLOUDSTACK-8964. The snapshot to volume is implemented and a small logger is added to the XenServerGuru to make host delegation visible.

@snuf Thanks! LGTM, based on a set of tests that I run on this branch:
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
component/test_vpc_redundant.py \
component/test_routers_iptables_default_policy.py \
component/test_routers_network_ops.py \
component/test_vpc_router_nics.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py

Result:
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Stop existing router, add a PF rule and check we can access the VM ... === TestName: test_isolate_network_FW_PF_default_routes | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_RVR_Network_FW_PF_SSH_default_routes | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 21 tests in 9470.662s

OK

And:
nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_privategw_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_vpc_vpn.py \
smoke/test_service_offerings.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

Result:
Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : EXCEPTION ===
ERROR
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : EXCEPTION ===
ERROR
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : EXCEPTION ===
ERROR
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 40 tests in 6968.642s

FAILED (errors=3)

The 3 errors are due to cleanup problems that are were broken on master when this was branched off. It was fixed in PR #967.

@snuf We may want to run a specific test against OVM3, I assume you did that? The above tests just show you didn't break anything else :-)

@remi: yep did the OVM3 dance, together with KVM and separately from OVM3.

@DaanHoogland Could you please do a code review of this PR? Thx!

one comment, code looks fine otherwise

Pinging @snuf to respond to the comment here

Ping @remibergsma @snuf
Just started testing this PR.
Cheers,
Wilder

Ping @remibergsma @snuf
I did not test any OVM3 specifics, but @snuf did. The PR doesn't break existing functionality.
Although the "hardware required=false" tests are still running, I would like to give my LGTM! :+1:
Tests results:

Hardware required: true
Management Server + MySQL on CentOS 7.1
One KVM host on CentOS 7.1
Agent and Common RPMs built from source

Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Test to verify access to loadbalancer haproxy admin stats page ... === TestName: test02_internallb_haproxy_stats_on_all_interfaces | Status : SUCCESS ===
ok
Test create, assign, remove of an Internal LB with roundrobin http traffic to 3 vm's ... === TestName: test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 | Status : SUCCESS ===
ok
Stop existing router, add a PF rule and check we can access the VM ... === TestName: test_isolate_network_FW_PF_default_routes | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_RVR_Network_FW_PF_SSH_default_routes | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 27 tests in 13849.872s

OK
(END)

Cheers,
Wilder

@snuf Can you respond to the comment by @dahn? Then we can proceed to merge this. Thanks!

Sorry guys been a bit in flight, yes @DaanHoogland has point but tbh we should fix that everywhere in one go for all the hypervisors.... not to be a dick, but just saying :)

@dahn are you OK to merge this as-is?

#1015 made to apply my remarks

authors: DaanHoogland remibergsma snuf wilderrodrigues


text:
COMPRESS-407 - Just Warn

Warn if an invalid record size is set.
I would not recommend using this approach.

Coverage decreased (-0.02%) to 84.704% when pulling abab94288a8b8de05ecfade6517456a2c8dc6fb2 on sesuncedu:COMPRESS-407-WARN into 4be9979b45ceadc50dc24607884d34613fead1f5 on apache:master.

authors: coveralls sesuncedu

repo: apache/commons-compress


text:
CouchDB 3.1.0 not starting because it uses wrong DLLs on Windows 10

Description
We are using CouchDB as local database within a project. Basically it is just a small program which accesses the database. That's all working fine on my machine, but on an second machine we are having problems with starting up CouchDB at all. Installation works fine but then the CouchDB does not start on the machine.
The problem seems to be very similar to the one described here: https://github.com/apache/couchdb/issues/2680
Exactly the same DLLs are causing the same problems. We are already using CouchDB version 3.1.0 so we already use the fix mentioned in the above issue.
As far as I have analyzed the problem seems to be that the mentioned DLLs are also stored in the C:/Windows/System32 folder and CouchDB uses them instead of the files in the CouchDB/bin folder. If I remove the DLLs from System32 folder CouchDB starts as expected.
The Fix shipped out with CouchDB 3.0.1 sets the Systems32 to the path so I think this is causing the problem?
Steps to Reproduce
I cannot reproduce it on my machine. The problem only appears on the second machine.
Expected Behaviour
CouchDB starting on both machines
Your Environment

CouchDB version used: 3.1.0
Browser name and version: Irrelevant, CouchDB does not start at all
Operating system and version: Windows 10 Version 1909 (Build 18363.1198)

HI there,
https://github.com/apache/couchdb/pull/2710 was merged into 3.x a while ago. It should place %COUCHDB_BIN_DIR% ahead of %SystemRoot%\system32 in the path.
Can you confirm that your couchdb.cmd file has this change in place?

Hey,
yes I can confirm that the 'couchdb.cmd' has the changes from https://github.com/apache/couchdb/pull/2710
I also tried it with CouchDB 3.1.1 as well but the problem still occurs.

Hey,
yes I can confirm that the 'couchdb.cmd' has the changes from https://github.com/apache/couchdb/pull/2710
I also tried it with CouchDB 3.1.1 as well but the problem still occurs.

@magel20 Unfortunately, I can't confirm this in our development environment.
That PATH environment variable change explicitly puts the CouchDB binary directory ahead of the C:\Windows\System32 directory. The Windows dynamic linker searches PATH in order to find the necessary DLLs it needs. It can't run with only `%COUCHDB_BIN_DIR%, it needs the system's DLLs as well to run.
What happens if you open an Administrator Command Prompt and run the couchdb.cmd rather than using the automatic Windows Service? Does the problem still occur?
Do you know which program placed those DLLs in your %SYSTEMROOT% directory? Arguably, that's the bug here, because with those DLLs in there, if the %PATH% tweak doesn't prevent those from being accessed, there's little we can do here to fix it. Can you move those DLLs to the directory of the program that actually needs them?

@magel20 Unfortunately, I can't confirm this in our development environment.
That PATH environment variable change explicitly puts the CouchDB binary directory ahead of the C:\Windows\System32 directory. The Windows dynamic linker searches PATH in order to find the necessary DLLs it needs. It can't run with only `%COUCHDB_BIN_DIR%, it needs the system's DLLs as well to run.
What happens if you open an Administrator Command Prompt and run the couchdb.cmd rather than using the automatic Windows Service? Does the problem still occur?
Do you know which program placed those DLLs in your %SYSTEMROOT% directory? Arguably, that's the bug here, because with those DLLs in there, if the %PATH% tweak doesn't prevent those from being accessed, there's little we can do here to fix it. Can you move those DLLs to the directory of the program that actually needs them?

I tried the couchdb.cmd in an Administrator Command Prompt as well as the Windows Service. Problem occurred both times.
So the set PATH=... command inside the couchdb.cmd sets  the CouchDB binary directory ahead of the others. Should I be able to see that change in the Environment Variables UI in Windows 10? Because there I do not see the %COUCHDB_BIN_DIR% in the PATH at all or is this set done by the script only temporary?
If I understand it correctly the problem I encounter should not occur because the couchdb.cmd first looks inside the CouchDB binary directory finds the DLLs and should then not care about the DLLs in the %SYSTEMROOT% directory? Might there be a possibility that the set command is not working? Or is it more likely that the %PATH% tweak does not prevent the DLLs from being accessed if they are placed in the %SYSTEMROOT% ?
Unfortunately I have not found out which program placed the DLLs there. Removing them from %SYSTEMROOT% made the CouchDB work and so far no other program I use on a daily basis has complained about it. But if I got you correct it should not be the case that a program places its DLLs in the %SYSTEMROOT% directory and that is the actual bug?

I tried the couchdb.cmd in an Administrator Command Prompt as well as the Windows Service. Problem occurred both times.
So the set PATH=... command inside the couchdb.cmd sets  the CouchDB binary directory ahead of the others. Should I be able to see that change in the Environment Variables UI in Windows 10? Because there I do not see the %COUCHDB_BIN_DIR% in the PATH at all or is this set done by the script only temporary?
If I understand it correctly the problem I encounter should not occur because the couchdb.cmd first looks inside the CouchDB binary directory finds the DLLs and should then not care about the DLLs in the %SYSTEMROOT% directory? Might there be a possibility that the set command is not working? Or is it more likely that the %PATH% tweak does not prevent the DLLs from being accessed if they are placed in the %SYSTEMROOT% ?
Unfortunately I have not found out which program placed the DLLs there. Removing them from %SYSTEMROOT% made the CouchDB work and so far no other program I use on a daily basis has complained about it. But if I got you correct it should not be the case that a program places its DLLs in the %SYSTEMROOT% directory and that is the actual bug?

Closing this old issue, the windows installer of CouchDB doesn't put anything into system32.

repo: apache/couchdb


text:
Switch away from passing funs between nodes when uploading attachments

Passing closures around is fragile and prevents smooth upgrading. Instead pass
a tuple with a data from the receiver closure explicitly and convert to back to
a local fun locally on each node.
Config options [fabric] attachment_receiver_tuple controls how receivers are
encoded. To upgrade a cluster to use the new format, first upgrade all nodes to
the new code, then flip the value to true. In the future, the option and the
old closure handling code will be removed.

I can't review the code here, but I insist you add something to the documentation about this new flag, even if it's just in rel/overlay/etc/default.ini.
Speaking of which, I don't see any changes in this PR to that file :disappointed:  ALL new config:get entries should be paired with a change to default.ini.
When we change the default value and remove the config option, we will need a big screaming warning in the upgrade docs that says rolling upgrades from older CouchDBs that have the value set to false will fail. (It might be worth describing exactly how those upgrades will fail, too.)

@wohali sorry, this is still a WIP, forgot to mark it as such

authors: nickva wohali


text:
Fix displaying WORK or TRANSACTION after BEGIN

Fixes  #1553

🚀

authors: alamb mvzink

repo: apache/datafusion-sqlparser-rs


text:
[feat] Add token as authentication for python gateway

Brief Summary of The Change

separate from apache/dolphinscheduler#6407. Authentication, add secret to ensure only trusted people could
connect to gateway.
fix: apache/dolphinscheduler#8255
ref: apache/dolphinscheduler#12893
Pull Request checklist
I confirm that the following checklist has been completed.

[x] Add/Change test cases for the changes.
[x] Add/Change the related documentation.
[x] (Optional) Add your change to UPDATING.md when it is an incompatible change.

Codecov Report

Merging #13 (05b8f77) into main (20b2a2b) will decrease coverage by 0.00%.
The diff coverage is 90.00%.

@@            Coverage Diff             @@
##             main      #13      +/-   ##
==========================================
- Coverage   90.60%   90.59%   -0.01%
==========================================
  Files          56       56
  Lines        2032     2042      +10
==========================================
+ Hits         1841     1850       +9
- Misses        191      192       +1

Flag
Coverage Δ

unittests
90.59% <90.00%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/pydolphinscheduler/configuration.py
97.01% <90.00%> (-1.24%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Brief Summary of The Change
separate from apache/dolphinscheduler#6407. Authentication, add secret to ensure only trusted people could connect to gateway.
fix: apache/dolphinscheduler#8255 ref: apache/dolphinscheduler#12893
Pull Request checklist
I confirm that the following checklist has been completed.

[x]  Add/Change test cases for the changes.
[x]  Add/Change the related documentation.
[x]  (Optional) Add your change to UPDATING.md when it is an incompatible change.

released after apache/dolphinscheduler:3.1.2

authors: codecov-commenter zhongjiajie

repo: apache/dolphinscheduler-sdk-python


text:
The workflow defines that both data sources are mysql

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
datax plugin, The workflow defines that both data sources are mysql
What you expected to happen
Displays the data type I specified
How to reproduce
The workflow defines that both data sources are mysql
Anything else
No response
Version
3.0.0
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Could you please provide more detail?

authors: luojinhao ruanwenjun

repo: apache/dolphinscheduler


text:
[Bug] Fix StringUtils import

Purpose of the pull request

Related pr: #12495 (which run CI success)

@caishunfeng +1


text:
[Fix-11583][UI] Fix the API usage of gantt graph

Purpose of the pull request
Fixes https://github.com/apache/dolphinscheduler/issues/11583, the backend passed wrong data that cannot be parsed by the frontend.
(The import items were changed because of the spotless check)
Verify this pull request
This change added tests and can be verified as follows: Manually verified the change by testing locally.

Please review the backend @SbloodyS

@caishunfeng Please review the backend

Codecov Report

Merging #13778 (dfb5836) into dev (69e7449) will decrease coverage by 0.05%.
The diff coverage is 0.00%.

:exclamation: Current head dfb5836 differs from pull request most recent head ab8a8a0. Consider uploading reports for the commit ab8a8a0 to get more accurate results

@@             Coverage Diff              @@
##                dev   #13778      +/-   ##
============================================
- Coverage     39.02%   38.97%   -0.05%
+ Complexity     4450     4442       -8
============================================
  Files          1133     1133
  Lines         42353    42316      -37
  Branches       4789     4769      -20
============================================
- Hits          16528    16494      -34
+ Misses        23999    23998       -1
+ Partials       1826     1824       -2

Impacted Files
Coverage Δ

...r/api/service/impl/ProcessInstanceServiceImpl.java
59.92% <0.00%> (ø)

... and 33 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter lxorc


text:
[Bug]be can't start

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
1.1.2
What's Wrong?
Sep 13 18:05:37 be01 kernel: traps: doris_be[15696] trap invalid opcode ip:20f0685 sp:7ffd6baf1160 error:0 in doris_be[400000+6fc1000]
Sep 13 18:05:38 be01 systemd-coredump[15698]: Process 15696 (doris_be) of user 0 dumped core.
                                          Stack trace of thread 15696:
                                          #0  0x00000000020f0685 _Z27check_required_instructionsv (doris_be)
                                          #1  0x000000000601751d __libc_csu_init (doris_be)
                                          #2  0x000014a504f5973e __libc_start_main (libc.so.6)
                                          #3  0x00000000020ee8aa _start (doris_be)

-- Subject: Process 15696 (doris_be) dumped core
What You Expected?
can started
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

kernel version?

centos 8 4.18.0-240.1.1.el8_3.x86_64

Can you add me on WeChat(jnsimba) and let's take a look?

Is there a solution to this problem

repo: apache/doris


text:
feature enable  quick compaction  by default

Proposed changes
Issue Number: close #xxx
Problem summary
i think the quick compaction enabled by default is more beneficial for the system
more details see  issue(#9804)
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[x] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@chenlinzhong  rebase


text:
fix fix meta replay bug when upgrading from 1.1.x to 1.2.x

Proposed changes
Issue Number: close #xxx
Problem summary
Follow up #12016
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[x] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[x] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.79 seconds
load time: 457 seconds
storage size: 17180036989 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221107184137_clickbench_pr_41408.html

authors: hello-stephen morningman


text:
fix Check whether the in bitmap contains correlated subqueries

Currently in bitmap does not support correlated subquery, like: a.k1 in (select bitmap_col from b where a.k2 = b.k2).

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.65 seconds
load time: 630 seconds
storage size: 17122997150 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221219145636_clickbench_pr_65243.html

authors: hello-stephen luozenglin


text:
[typo][doc]Delete the extra "".

Proposed changes
Delete the extra "".

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
fixshouldn't use arena to alloc memory for SingleValueDataString

Proposed changes
==1682746==ERROR: AddressSanitizer: alloc-dealloc-mismatch (malloc vs operator delete []) on 0x7f4640d16800
    #0 0x5634d3357e9d in operator delete[](void*) (/mnt/ssd01/doris-master/VEC_ASAN/be/lib/doris_be+0x17050e9d) (BuildId: 3e28e26775df56d2)
    #1 0x5634d7958783 in doris::vectorized::SingleValueDataString::reset() /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_min_max.h:331:13
    #2 0x5634d79518d0 in doris::vectorized::AggregateFunctionsSingleValue<doris::vectorized::AggregateFunctionMaxData<doris::vectorized::SingleValueDataString>>::reset(char*) const /home/zcp/repo_center/doris_master/doris/be/src/vec/aggregate_functions/aggregate_function_min_max.h:535:75
    #3 0x5634f5cf2d9c in doris::vectorized::BlockReader::_update_agg_value(std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>>>&, int, int, bool) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:492:23
    #4 0x5634f5cf170f in doris::vectorized::BlockReader::_update_agg_data(std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>>>&) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:426:9
    #5 0x5634f5cf08e4 in doris::vectorized::BlockReader::_agg_key_next_block(doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.cpp:317:5
    #6 0x5634d388a9af in doris::vectorized::BlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/olap/block_reader.h:51:16
    #7 0x5634e9215dc5 in doris::vectorized::NewOlapScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/new_olap_scanner.cpp:466:5
    #8 0x5634e95e0a2e in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vscanner.cpp:76:17
    #9 0x5634e9289721 in doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler*, doris::vectorized::ScannerContext*, std::shared_ptr<doris::vectorized::VScanner>) /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:372:27
    #10 0x5634e928ce6e in doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::'lambda1'()::operator()() const /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:243:35
    #11 0x5634e928cd24 in void std::__invoke_impl<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::'lambda1'()&>(std::__invoke_other, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::'lambda1'()&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #12 0x5634e928ccc4 in std::enable_if<is_invocable_r_v<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::'lambda1'()&>, void>::type std::__invoke_r<void, doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::'lambda1'()&>(doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::'lambda1'()&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #13 0x5634e928cb2c in std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext*)::$_1::operator()() const::'lambda1'()>::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
    #14 0x5634d3503372 in std::function<void ()>::operator()() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560:9
    #15 0x5634e9297d24 in doris::PriorityWorkStealingThreadPool::work_thread(int) /home/zcp/repo_center/doris_master/doris/be/src/util/priority_work_stealing_thread_pool.hpp:135:17
    #16 0x5634e929c5b2 in void std::__invoke_impl<void, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_memfun_deref, void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14
    #17 0x5634e929c484 in std::__invoke_result<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>::type std::__invoke<void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&>(void (doris::PriorityWorkStealingThreadPool::* const&)(int), doris::PriorityWorkStealingThreadPool*&, int&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #18 0x5634e929c454 in decltype(std::__invoke((*this)._M_pmf, std::forward<doris::PriorityWorkStealingThreadPool*&>(fp), std::forward<int&>(fp))) std::_Mem_fn_base<void (doris::PriorityWorkStealingThreadPool::*)(int), true>::operator()<doris::PriorityWorkStealingThreadPool*&, int&>(doris::PriorityWorkStealingThreadPool*&, int&) const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:131:11
    #19 0x5634e929c424 in void std::__invoke_impl<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::__invoke_other, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #20 0x5634e929c394 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&>(std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)>&, doris::PriorityWorkStealingThreadPool*&, int&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
    #21 0x5634e929c303 in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::__call<void, 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570:11
    #22 0x5634e929c16d in void std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>::operator()<>() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:629:17
    #23 0x5634e929c084 in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
    #24 0x5634e929c044 in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>(std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>&&) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14
    #25 0x5634e929c01c in void std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253:13
    #26 0x5634e929bff4 in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>::operator()() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260:11
    #27 0x5634e929bf78 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (doris::PriorityWorkStealingThreadPool::*)(int)> (doris::PriorityWorkStealingThreadPool*, int)>>>>::_M_run() /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13
    #28 0x5634f970b4ef in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../libstdc++-v3/src/c++11/thread.cc:82:18
    #29 0x7f56a77a4608 in start_thread /build/glibc-sMfBJT/glibc-2.31/nptl/pthread_create.c:477:8
    #30 0x7f56a7a33162 in __clone /build/glibc-sMfBJT/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.73 seconds
stream load tsv:          545 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17161964981 Bytes

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.66 seconds
stream load tsv:          541 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162360948 Bytes

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.83 seconds
stream load tsv:          540 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select:          29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162228676 Bytes

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.03 seconds
stream load tsv:          542 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select:          29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162225819 Bytes

authors: hello-stephen starocean999 yiguolei


text:
feat Enable auto sample by default

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.26 seconds
stream load tsv:          573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17163856179 Bytes

authors: Kikyou1997 hello-stephen xiaokang


text:
fixfix start time can not be earlier than the current time

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
[Fix] fix compile problem

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
refactorRename workload schedule policy

Proposed changes
Rename workload schedule policy grammar to workload policy.

run buildall

update document?

authors: morrySnow wangbo


text:
[Pick](Row store)pick #37145

Proposed changes
pick #37145

run compile

run buildall


text:
[Regression](inverted index) fix build index case for wrong debug point #38112

cherry pick from #38112

run buildall


text:
[fix] _cur_reader  can be null in exception cases in VFileScanner

Proposed changes
Issue Number: close #xxx
cur_reader pointer can be null in VFileScanner. can cause BE crash
[ RUN      ] VfileScannerExcepTest.failure_case
AddressSanitizer:DEADLYSIGNAL
==1892247==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5cdbe26eb5f8 bp 0x7ffed6728610 sp 0x7ffed67277a0 T0)
==1892247==The signal is caused by a READ memory access.
==1892247==Hint: address points to the zero page.
#0 0x5cdbe26eb5f8 in doris::vectorized::VFileScanner::_get_next_reader() /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:980:9
#1 0x5cdbe26e3512 in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:286:25
#2 0x5cdbe26e2e46 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vfile_scanner.cpp:252:17
#3 0x5cdbe28e05ce in doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/workspace/doris/be/src/vec/exec/scan/vscanner.cpp:117:17
#4 0x5cdbc405a922 in doris::vectorized::VfileScannerExcepTest_failure_case_Test::TestBody() /root/doris/workspace/doris/be/test/vec/exec/vfile_scanner_excep_test.cpp:309:24
#5 0x5cdbff5b191a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51eb491a) (BuildId: 21d41a2d207823b9)
#6 0x5cdbff59f989 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::)(), char const) (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51ea2989) (BuildId: 21d41a2d207823b9)
#7 0x5cdbff57a9c2 in testing::Test::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7d9c2) (BuildId: 21d41a2d207823b9)
#8 0x5cdbff57b708 in testing::TestInfo::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7e708) (BuildId: 21d41a2d207823b9)
#9 0x5cdbff57bec3 in testing::TestSuite::Run() (/root/doris/workspace/doris/be/ut_build_ASAN/test/doris_be_test+0x51e7eec3) (BuildId: 21d41a2d207823b9)
After fix:
I20240902 09:11:07.722273 1946048 run_all_tests.cpp:67] init config 1
Note: Google Test filter = VfileScannerE*
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from VfileScannerExceptionTest
[ RUN      ] VfileScannerExceptionTest.failure_case
msg = [INTERNAL_ERROR]cur path: . Failed to create reader for  file format: 11
[       OK ] VfileScannerExceptionTest.failure_case (3 ms)
[----------] 1 test from VfileScannerExceptionTest (3 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (3 ms total)
[  PASSED  ] 1 test.
=== Finished. Gtest output: /root/doris/workspace/doris/be/ut_build_ASAN/gtest_output

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall

authors: Vallishp yiguolei


text:
opt Refactor memory maintenance thread

step 1. Refresh process memory metrics.
step 2. Refresh allocator memory metrics.
step 3. Update and print memory stat when the memory changes by 256M.
step 4. Asyn Refresh cache capacity
step 5. Cancel top memory task when process memory exceed hard limit.
step 6. Refresh weighted memory ratio of workload groups.
step 7. Analyze blocking queries.
step 8. Flush memtable.
step 9. Jemalloc purge all arena dirty pages.
memory_maintenance_thread execute once cost:

3ms (cluster idle)
20ms (cluster high concurrency, CPU full)

memory_maintenance_thread CPU usage:

10%-20% (default memory_maintenance_sleep_time_ms=20ms)
20%-30% (memory_maintenance_sleep_time_ms=10ms)
30%+ (memory_maintenance_sleep_time_ms=5ms)

run buildall

run buildall


text:
[fix] (bloom filter) Fix the bloom filter calculation for date and datetime

What problem does this PR solve?

Problem Summary:
After adding a bloom filter to the date type, queries return errors following the upgrade.
Check List (For Committer)

Test

[ ] Regression test
[x] Unit Test
[x] Manual test (add detailed scripts or steps below)
[ ] No need to test or manual test. Explain why:

[ ] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No colde files have been changed.
[ ] Other reason

Manual Test

Create a table with a date type column and a corresponding bloom filter.
Upgrade to the 2.1
Queries are correct.

Behavior changed:

[x] No.
[ ] Yes.

Does this need documentation?

[x] No.
[ ] Yes.

Release note

None

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall

run buildall

run buildall


text:
Reduce method invocation of reservoir sampling

Motivation
The periodic task of the Coordinator service could be slow in a large cluster. The periodic task consists of several serial subtasks. According to the profiling result, the segment balance task has some performance issue. I found that the root cause is that the current implementation invokes the sampling method too many times. We can reduce the number of method invocations by increasing the sample size in each invocation.

Proposed changes
Adding a new Reservoir Sampling method to sample K elements each time instead of only one element each time.
A default method pickSegmentsToMove will be added to interface BalancerStrategy to pick K segments to move in a single method invocation.
Rationale
The current implementation picks up only one segment each time iterating all segments. When there are a lot segments need to be rebalanced or need to be decommissioned, the load balance calculation will be really slow. By picking up K segments each time will significantly reduce the number of iteration and thus speed up the process.
Operational impact
There will be no impact in operation
Test plan (optional)
Ensure test coverage and test it in test cluster

Adding a new Reservoir Sampling method to sample K elements each time instead of only one element each time.

I'm not sure how this can improve the performance. Does the new sampling method need to loop to sample K segments anyway? I guess I'm probably missing something. Would you please add more details on the proposed changes?

Adding a new Reservoir Sampling method to sample K elements each time instead of only one element each time.

I'm not sure how this can improve the performance. Does the new sampling method need to loop to sample K segments anyway? I guess I'm probably missing something. Would you please add more details on the proposed changes?

Thanks for having a look at this. The default implementation samples only one segment in an iteration of all segments. Let's assume that:

the list of server holders contain 1 million segments
1000 segments need to be picked up from these server holders

Then the current implementation needs to call the sampling method 1000 times and each time needs to iterate 1 million segments.
I found that the Reservoir Sampling actually can sample K elements a single pass over the items. So in this case, the new method can sample 1000 segments in a method invocation, which means it needs to iterate 1 million segments only once.

@yuanlihan thanks. I think I was confused about what is expensive. I think what is expensive is not the sample method invocation, but is the algorithm that calls the sample method over again. I looked over your PR and seems reasonable to me.

@jihoonson

But I want to think about it more because we have no test coverage on the long term effect of this change.

As this change will be enabled by default, I understand your concern. I tried to confirm the correctness based on the following facts:

the set of segments in each server holder is immutable in each cycle
the sampling method ensures(without mathematical proof) that getting k segments in a single pass is equal to getting k segments in k passes(1 segment per pass)

It would be great if you can provide some long term test results.

Actually I have shipped this change into our on-premise cluster(which contains millions of segments) in Jan 2021 and it works well. In the past several months, the default cost-based balancer manages segment loading among 5 different historical tiers well. During this period, we expanded the cluster by adding new historical nodes and decommissioned historical nodes for server maintenance. It seems hard to provide visual long term function testing result about this change.

@yuanlihan thank you for adding more details. I think the basic idea makes sense since every segment has the same probability of being picked up. I will review your PR soon.

authors: jihoonson yuanlihan

repo: apache/druid


text:
bump to latest of same version node and npm versions, bump frontend-maven-plugin

Updates the node/npm versions to latest v10.x version per https://nodejs.org/en/download/releases/. I wasn't adventurous enough to try to go any higher...
Also, more consistently uses node.version in pom.xml.

+1 after CI passes. This PR seems to fix the broken CI (for example, https://travis-ci.com/github/apache/druid/builds/221570150).

authors: clintropolis jihoonson


text:
num of rows in a segment dimensions

Adds the ability to see the number of rows per segment.

Description
Adds metrics to help users of druid to understand how many rows there are in a segment. As part of this we report the average num of rows in a segment and also a distribution of segments in predefined buckets.

Adding metrics
The use of buckets was chosen because average wasn't good enough to understand the number of rows that are there in segments since its possible to have outliers and such. I looked into using min/max/median but that would involve too much memory keeping track of the number of rows in every segment for every datasource. Using buckets gives us a good idea of the distribution of rows in segments that only requires keeping track of a fixed number of values.
doing this at SegmentManager whenever we load or drop a segment allows for not counting the number of rows(which can be an expensive operation as it scans the segment) more than once. The downside to this is handling lazily loaded segments. These metrics will not make sense if lazy loading is enabled.
We chose to not handle lazy loading with this as it can be hard to understand if a segment is loaded 2x once lazily then actually how to keep track of that. And also it is hard to know if a dropped segment was originally lazy loaded or not. So instead this will throw an exception if SegmentStatsMonitor is enabled and lazyLoadOnStart is true

Key changed/added classes in this PR

SegmentRowCountDistribution.java
SegmentStatsMonitor.java

This PR has:

[x] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[x] added documentation for new or modified features or behaviors.
[x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[x] been tested in a test Druid cluster.

The bucket sizing hasn't been decided yet so please give lots of input on those

@suneet-s made changes to get tests to pass. Please take a look

Thanks @TSFenwick !

authors: TSFenwick suneet-s


text:
有计划支持JDK 11吗

目前我们线上服务器全部都是jdk 11，但好像dubbo-admin还只能用在jdk 8，请问有计划支持或者有解决方案吗

可以使用 11 试一下。

jdk11在解析token的时候会抛找不到javax.xml.bind.DatatypeConverter的异常

可以使用 11 试一下。

jdk11在解析token的时候会抛找不到javax.xml.bind.DatatypeConverter的异常

添加依赖试一下。
<dependency>
    <groupId>javax.xml.bind</groupId>
    <artifactId>jaxb-api</artifactId>
    <version>2.3.0</version>
</dependency>

closed by #886.

authors: chen796916 haoyann ivancxj

repo: apache/dubbo-admin


text:
[Bug Report]Addresses in 2.7.1 cannot be removed from Registry properly.

Recently, we keep receiving issues from the community reporting duplicate URL addresses in Registry, mostly zookeeper, after upgrading to 2.7.1. Those URLs are all pushed to the Consumer and because only one of the duplicated URLs is valid, it causes the consumer to fail when the consumer chooses the invalid one during load balance.
The following are some typical issues: #3785 #3770 #3920 #4013
After troubleshooting, we found the reason for the problem: in version 2.7.1, the URL addresses will be registered as a persistent node in zookeeper, which is different from the behavior in 2.7.0 or before, in which they are registered as ephemeral nodes. The meaning of the ephemeral node is that it will get removed from zookeeper when the server process went for a while. Now, after changed to persistent, the node cannot be removed in some exceptional scenarios, such as process crashed, process forced killed with kill -9, etc, eventually leading to the appearance of dirty data.
Advices to solve or avoid the problem:

Never forcibly terminating the process to ensure that Dubbo enters it's elegant shutdown process.
Wait for 2.7.2 to fix this issue, going to be released in early June.
For dirty data that has already been generated, consider manual or ZK scripts to clean up, but be careful of the deletion criteria.
Add the following configuration for 2.7.1: <dubbo:provider dynamic="true"/> or <dubbo:service dynamic="true"/>

近期，关于升级到2.7.1版本后注册中心（多数是zookeeper）出现重复URL地址数据无法删除的情况，我们持续收到来自社区的issue报告。以下issue是几个典型的异常现象：#3785 #3770 #3920 #4013
经过问题排查我们定位到了问题的原因：在2.7.1版本中URL地址在zookeeper中被注册为持久persistent节点(2.7.0及之前注册的是临时emperal节点)，这样当server进程异常终止无法进入正常优雅下线流程时，如进程崩溃、强行kill -9等，导致zookeeper已失效持久节点无法清空，最终导致脏数据的出现。
解决及规避方法：

上下线过程避免强行终止进程，保证让Dubbo进入优雅停机流程
官方2.7.2版本将会修复此问题，预计在6月初发布
对于已经产生的脏数据，考虑手动或ZK脚本予以清理，但此时要注意严格控制删选条件
2.7.1版本增加配置：<dubbo:provider dynamic="true"/> 或<dubbo:service dynamic="true"/>

终于修复了这个问题。

repo: apache/dubbo


text:
Triple http limiting the size of the HTTP request and response

What is the purpose of the change
refer: https://github.com/apache/dubbo/issues/14068

When an HttpStatusException occurs during serialization, It does not need to be wrapped as an EncodeException
Use LimitedByteBufOutputStream to limit the size of the write data. Throws an HttpOverPayloadException when the limit is exceeded
Fix some typos

For example:
        tripleProtocol.setTriple(TripleBuilder.newBuilder()
                .maxBodySize(50)
                .maxResponseBodySize(100)
                .build());

Success result:

Request body limit result:

Response body limit result:

Brief changelog
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

@oxsean PTAL

I thinkd h2 respone also need limit respone size.

I think h2 respone also need limit respone size.

HTTP/2 also limits the response body size and uses tri-status and tri-message identifiers.

In http2:

In case of serialization failure, add tri-status and tri-message in trailers to indicate the reason for failure, and keep the header code as 200, or serialize first and then send the header frame for unary calls?
Can content-length be placed in trailers? Is it necessary to place it? Or provide configuration to write content-length?

A problem encountered is that errors may occur during serialization. At this time, if the header is written first, the status code will be 200, but the result is failure.

Http2 no longer requires a content-length header, but some clients may use content-length to determine whether the data stream is complete. Therefore, refer to some common implementations in the industry to decide whether to add a content-length header to the http2 trailers, such as spring boot, tomcat, etc.

New results
http2:

http1:

LGTM.

@EarthChen PTAL

authors: AlbumenJ finefuture oxsean