text:
Improve object_store crate documentation

Which issue does this PR close?
Closes https://github.com/apache/arrow-rs/issues/2202
Rationale for this change
More docs == more better
What changes are included in this PR?

[x] Update main crate readme (What appears in https://crates.io/crates/object_store)
[x] Add main example (front page of https://docs.rs/object_store/latest/object_store/)

Example front page:

Are there any user-facing changes?
Better docs

Codecov Report

Merging #2260 (990f81b) into master (2c09ba4) will increase coverage by 0.00%.
The diff coverage is 100.00%.

:exclamation: Current head 990f81b differs from pull request most recent head 88e2060. Consider uploading reports for the commit 88e2060 to get more accurate results

@@           Coverage Diff           @@
##           master    #2260   +/-   ##
=======================================
  Coverage   82.28%   82.28%
=======================================
  Files         245      245
  Lines       62670    62688   +18
=======================================
+ Hits        51567    51585   +18
  Misses      11103    11103

Impacted Files
Coverage Δ

object_store/src/aws.rs
0.00% <ø> (ø)

object_store/src/azure.rs
0.00% <ø> (ø)

object_store/src/gcp.rs
0.00% <ø> (ø)

object_store/src/lib.rs
86.75% <ø> (ø)

arrow/src/datatypes/types.rs
97.93% <100.00%> (+0.46%)
:arrow_up:

arrow/src/datatypes/datatype.rs
62.30% <0.00%> (-0.32%)
:arrow_down:

parquet_derive/src/parquet_field.rs
65.98% <0.00%> (+0.22%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us.

Thanks @Ted-Jiang  !

authors: alamb codecov-commenter


text:
Zero-copy conversion from Vec to PrimitiveArray

Which issue does this PR close?

Closes #.
Rationale for this change

Avoids a memory copy
What changes are included in this PR?

Are there any user-facing changes?

Aah, this breaks the mutable kernels...

source: gharchive/pull-request

repo: apache/arrow-rs

license_type: permissive

license_source: github-api


text:
Update twox-hash requirement from 1.6 to 2.0

Replaces #6677.

Ahh, MSRV for that crate is now Rust 1.81. I guess that's too new?

MSRV for that crate is now Rust 1.81

That is quite aggressively recent, having only been released in September. Perhaps we should sit on this for a bit?

See also https://github.com/apache/arrow-rs/pull/6583 from @shepmaster

Closing in favor of #6583.

authors: alamb crepererum tustvold


text:
[R] CRAN release version 10.0.1

Packaging checklist for CRAN release
For full documentation of the steps in this checklist, see the
Apache Arrow Release Management Guide.

[x] When a release candidate is created
Create a GitHub issue
entitled [R] CRAN release version X.X.X and copy this checklist to the issue

[x] Confirm that all
nightly tests and nightly packaging builds are passing.

Will be at https://github.com/apache/arrow/pull/14772

[x] Check current CRAN check results

[x] Ensure the contents of the README is accurate

[x] urlchecker::url_check()

[x] Polish NEWS

[x] Prepare tweet thread highlighting new features

Make pull requests into the autobrew and
rtools-packages repositories
used by the configure script on MacOS and Windows. These pull requests will
use the release candidate as the source.

[x] Pull request to modify
the apache-arrow autobrew formula
to update the version, SHA, and any changes to dependencies and build steps that have changed in the
copy of the formula we have of that formula in the Arrow repo
[x] Pull request to modify
the apache-arrow-static autobrew formula
to update the version, SHA, and any changes to dependencies and build steps that have changed in the
copy of the formula we have of that formula in the Arrow repo
[x] Pull request to modify the
autobrew script
to include any additions made to
r/tools/autobrew.
[x] Pull request to modify the
PKGBUILD script
to reflect changes in
ci/PKGBUILD,
uncommenting the line that says "uncomment to test the rc".

Prepare and check the .tar.gz that will be released to CRAN.

[x] git fetch upstream && git checkout release-X.X.X-rcXX
[x] make build (copies Arrow C++ into tools/cpp, prunes some unnecessary
components, and runs R CMD build)
[x] devtools::check("arrow_X.X.X.tar.gz") locally
[x] Run reverse dependency checks

Wait for the official release...

[x] Release vote passed!
[x] Create a CRAN-release branch from the official release commit
[x] Pick any commits that were made to master since the release commit that
were needed to fix CRAN-related submission issues (e.g., fixed URLs, fixes
to )
[x] Remove badges from README.md
[x] Run urlchecker::url_check() on the R directory
[x] Bump the Version in DESCRIPTION to X.X.X
[x] Create a PR entitled WIP: [R] Verify CRAN release-10.0.1-rc0. Add
a comment @github-actions crossbow submit --group r to run all R crossbow
jobs against the CRAN-specific release branch.
[x] Regenerate arrow_X.X.X.tar.gz (i.e., make build)

Update autobrew and r-windows PRs to use the release instead of the
release candidate; ensure linux binary packages are available:

[x] PR into autobrew/homebrew-core
[x] PR into autobrew/homebrew-core
[x] PR into autobrew/scripts
[x] PR into r-windows/rtools-packages
[x] Ensure linux binaries are available in the artifactory:
https://apache.jfrog.io/ui/repos/tree/General/arrow/r

Check binary Arrow C++ distributions specific to the R package:

[x] Upload the .tar.gz to win-builder
and confirm that the check is clean

done; Neal will get an email to confirm. Failed. Resubmit after Jeroen has uploaded binaries.

[x] Upload the .tar.gz to MacBuilder
and confirm that the check is clean

Results will be at https://mac.r-project.org/macbuilder/results/1669915267-2fc3871d8987a385/

[x] Check install.packages("arrow_X.X.X.tar.gz") on Ubuntu and ensure that the
hosted binaries are used
[x] devtools::check("arrow_X.X.X.tar.gz") locally one more time (for luck)

Submit!

[x] Upload arrow_X.X.X.tar.gz to the
CRAN submit page
[x] Confirm the submission email

Wait for CRAN...

[x] Accepted!
[ ] Tag the tip of the CRAN-specific release branch
[x] Add a new line to the matrix in the backwards compatability job
[x] Update the packaging checklist template to reflect any new realities of the
packaging process.
[x] Wait for CRAN-hosted binaries on the
CRAN package page to reflect the
new version
[ ] Tweet!

Closing this now, as the release is done.

source: gharchive/issue

repo: apache/arrow


text:
[Release] Calculate the committer statistics from the master branch

Out current workflow for patch releases starts with cherry picking the selected patches from the master branch to a maint-* branch which means that the original committer data gets lost.
Reporter: Krisztian Szucs / @kszucs
Note: This issue was originally created as ARROW-8850. Please see the migration documentation for further details.

Committee data is lost due to API merge anyway and needs to be extracted from commit message.

authors: asfimport assignUser


text:
ARROW-12771: [C++][Compute] Fix MaybeReserve parameter in the Consume function of GroupedCountImpl

The parameter of MaybeReserve in GroupedCountImpl's Consume function is incorrect.
counts_.length() means the buffer's byte size, it should be replaced with num_groups_.  Otherwise, subsequent chunked arrays with more groups are skipped as counts_.length() is actually num_groups_*sizeof(int64_t)

CI failures look unrelated. I've restarted the failed jobs, will merge when everything's green

authors: Crystrix bkietz


text:
MINOR: [C++] Fix MSVC compiler error

Downcast explicitly from double to long.
This fixes a regression introduced in ARROW-16521.

CI passed on my fork, will merge.


text:
ARROW-2313: [C++] Add -NDEBUG flag to arrow.pc

Arrow C++ users should use the same -NDEBUG flag as Arrow C++ itself.

@pitrou @xhochy on some more contemplation about this issue, I think we should make Buffer::mutable_data always inline -- right now depending on whether libarrow is built in debug or release mode, the symbol will be present in the shared library or not. This means that downstream linkers must remember to define NDEBUG before including Arrow headers if they are using an Arrow release build. Does this sound OK (while it is nice to assert that mutable_data() has not been used on a non-mutable buffer)?

That sounds right to me.

Hmm, I see. A workaround would be to have the inline function call an always-defined non-inline checker if NDEBUG isn't defined.

OK, patch incoming

@pitrou I agree -- we should work out an equivalent DCHECK strategy that can be included in header files for debug assertions

Something like:
class Array {
  uint8_t* mutable_data() const {
#ifndef NDEBUG
    CheckMutable();
#endif
    return mutable_data_;
  }

protected:
  void CheckMutable();
};

// in array.cpp
void Array::CheckMutable() {
  DCHECK(mutable()) << "array not mutable";
}

Sure, I will add that to my patch https://github.com/apache/arrow/pull/1756

authors: kou pitrou wesm xhochy


text:
GH-36513: [Dev][C#] Add Dependabot configuration for NuGet

Rationale for this change
We don't want to adjust PR titles from Dependabot manually.
What changes are included in this PR?
Set the commit message prefix for /csharp/ from Dependabot.
Are these changes tested?
No. I hope that this works well...
Are there any user-facing changes?
No.

Closes: #36513

@eerhardt This causes the following PRs:

#36525
#36526
#36527
#36528
#36529
#36530

Are they useful for us? Or should we revert this configuration?

Are they useful for us? Or should we revert this configuration?

Some are useful. But some I would suggest not to take. For example https://github.com/apache/arrow/pull/36530.
I would suggest adding this section:
https://github.com/microsoft/semantic-kernel/blob/8d705b30adf0ef94792dacb86424b172b613ffcd/.github/dependabot.yml#L25-L31
For things like System., Microsoft.Bcl., etc. we want to consciously upgrade major versions. There are reasons why customers want to stay on older versions, for example 6.0 which is "LTS - long term support", and not upgrade to 7.0, which goes out of support sooner than 6.0.
See https://github.com/microsoft/semantic-kernel/pull/1799 for reasoning why staying on lower major versions is preferred in some cases.

OK. I've opened #36577. Could you review PRs from Dependabot and merge/close them?

authors: eerhardt kou


text:
Add Arrow Flight SQL ODBC driver

Rationale for this change
An ODBC driver uses the Open Database Connectivity (ODBC) interface that allow applications to access data in DBMS like environment using SQL.
As Arrow Flight provides JDBC and ADBC drivers, similarly Arrow Flight can provide ODBC driver.
What changes are included in this PR?
This PR adds an ODBC driver implementation.
Are these changes tested?
This ODBC driver is coming from a production system (SGA) that is tested/ran for a while.
Are there any user-facing changes?
No change, but new user option to use Arrow Flight.

@ianmcook @lidavidm @assignUser as discussed during the community meetings, here's the ODBC driver PR draft.
This PR includes:

sources files with legal fixes (I'm checking whoami source dual license/MIT, eventually alternatives)
build convenient files (like Brewfile)

I'm still working on:

finalize Arrow update (the original code was using Arrow 9.x)
finalize the build and cleanup the CMakeLists.txt files

finalize Arrow update (the original code was using Arrow 9.x)

I fear I can't really help on that front but once that's done I'd be happy to continue working on the CMake!

Hi @jduo !

Some of the code in odbc_impl should be updated to use the same naming conventions as the arrow project (eg ODBCConnection -> odbc_connection).

Yes, I'm doing that (that's why the PR is still a draft).

This covers the flightsql-odbc repo, but there is additional code needed for the ODBC entry points to build a functional driver. That was in warpdrive, but that code isn't Apache-compatible.

Yes, locally I fixed/updated it, but I'm looking for alternative.

Need to add the license for whereami to the Arrow project itself.

I'm checking if we can't find an alternative, else I will update NOTICE file.

Are the separate brewfile and vcpkg.json necessary or can this utilize the ones already at the top-level?

Yes, that's the plan, I'm doing in two steps: fixing the build and integrating in the Arrow ecosystem.

Should the top-level CMakeLists now include the driver build? Or is that on hold as part of the donation process?

Agree, same the same as for brewfile/vcpkg.json: I'm doing that in two steps.
I will push new changes to this PR and "remove" the draft state as soon as it's clean to review (build ok, etc).

This covers the flightsql-odbc repo, but there is additional code needed for the ODBC entry points to build a functional driver. That was in warpdrive, but that code isn't Apache-compatible.

Yes, locally I fixed/updated it, but I'm looking for alternative.

There was another group that did some work to build a full driver from flightsql-odbc. I haven't seen it though, but they've mentioned it here:
https://github.com/apache/arrow/issues/30622#issuecomment-1925690421

Following up here - do you want any help?

@jbonofre checking in on this. Were you still continuing work on this? Did you need help?
Thanks

@lidavidm @jduo hey guys. I'm very very sorry to have been quiet for long. Between vacation and ASF tasks, I was not able to be active enough on Arrow. I'm now back and resuming my work on Arrow, including ODBC.

No worries! Glad to see you back around :)

Hey @jbonofre  we're excited about the AFS ODBC driver! We want to start using it as soon as we can for a project we're working on and was wondering whether you had a sense of timing where it might be ready to be in someone's hands for development. We would be happy to be early adopters :)

Thanks @jbonofre . We're around if you need assistance moving this forward.

Hi @jbonofre , just checking in on if we can help to move this forward.

Hey @jduo
I started the recase. I will push this.
Would you have time for a Quick sync ?
Thanks !

I'm curious, does this version of the driver allow passing arbitrary parameters? I tried using the Dremio ODBC driver but it can't seem to be able to set arbitrary parameters. I have the following in a custom PowerBI connector:
        // This record contains all of the connection string properties we
        // will set for this ODBC driver. The 'Driver' field is required for
        // all ODBC connections. Other properties will vary between ODBC drivers,
        // but generally take Server and Database properties. Note that
        // credential related properties will be set separately.
        ConnectionString = [
            Driver = "Arrow Flight SQL ODBC Driver",
            host = server,
            port = 443,
            environmentId = "200532"
        ],

But i get the following error: ODBC: ERROR [HY000] [Apache Arrow][Flight SQL] (100) Flight returned invalid argument error, with message: environmentId is a required connection parameter
For our use-case we need to be able to pass various parameters as extra headers, which is supported by both ADBC and JDBC.

I'm curious, does this version of the driver allow passing arbitrary parameters? I tried using the Dremio ODBC driver but it can't seem to be able to set arbitrary parameters. I have the following in a custom PowerBI connector:
        // This record contains all of the connection string properties we
        // will set for this ODBC driver. The 'Driver' field is required for
        // all ODBC connections. Other properties will vary between ODBC drivers,
        // but generally take Server and Database properties. Note that
        // credential related properties will be set separately.
        ConnectionString = [
            Driver = "Arrow Flight SQL ODBC Driver",
            host = server,
            port = 443,
            environmentId = "200532"
        ],

But i get the following error: ODBC: ERROR [HY000] [Apache Arrow][Flight SQL] (100) Flight returned invalid argument error, with message: environmentId is a required connection parameter
For our use-case we need to be able to pass various parameters as extra headers, which is supported by both ADBC and JDBC.

Last I tried, it supported this. I believe it lower-cases headers though (the C++ grpc client itself requires this) -- is your server looking for environmentId case-insensitively?

Hey @jduo I started the recase. I will push this. Would you have time for a Quick sync ? Thanks !

Hi @jbonofre , yes we can have a chat. Will send meeting info offline.

Last I tried, it supported this. I believe it lower-cases headers though (the C++ grpc client itself requires this) -- is your server looking for environmentId case-insensitively?

Yeah it's case-insensitive. I also tried setting environmentid in all lower-case and still get the same error

Interesting, I do see:
TEST(PopulateCallOptionsTest, GenericOption) {
  FlightSqlConnection connection(odbcabstraction::V_3);
  connection.SetClosed(false);

  Connection::ConnPropertyMap properties;
  properties["Foo"] = "Bar";
  auto options = connection.PopulateCallOptions(properties);
  auto headers = options.headers;
  ASSERT_EQ(1, headers.size());

  // Header name must be lower-case because gRPC will crash if it is not lower-case.
  ASSERT_EQ("foo", headers[0].first);

  // Header value should preserve case.
  ASSERT_EQ("Bar", headers[0].second);
}

Maybe there's a bug with call options not getting set correctly for the HANDSHAKE request? I remember that being an issue with the JDBC driver: https://github.com/apache/arrow/pull/33946

Last I tried, it supported this. I believe it lower-cases headers though (the C++ grpc client itself requires this)

This is actually part of the HTTP/2 spec.

Ok, I think I know what it is... PopulateCallOptions gets called AFTER Authenticate in FlightSqlConnection::Connect. We'd probably need that before.

@aiguofer good call. Let me update the test.

authors: CurtHagenlocher aiguofer assignUser jbonofre jduo lidavidm rpourzand


text:
ARROW-9986: allow to_timestamp to parse local times without fractional seconds

As pointed out on @jhorstmann: https://github.com/apache/arrow/pull/8161#issuecomment-691464193

One (not directly related) issue I noticed while trying this out, is that the local patterns seem to require the millisecond part, while for utc timestamps with "Z" they are optional:

> select to_timestamp('2020-09-12T10:30:00') from test limit 1;
ArrowError(ExternalError(General("Error parsing \'2020-09-12T10:30:00\' as timestamp")))

> select to_timestamp('2020-09-12T10:30:00Z') from test limit 1;
+-------------------------------------------+
| totimestamp(Utf8("2020-09-12T10:30:00Z")) |
+-------------------------------------------+
| 1599906600000000000                       |
+-------------------------------------------+

This PR allows parsing local timestamp patterns without the fractional seconds and tests for same

@alamb these timestamp conversions could be useful as compute kernels (https://github.com/apache/arrow/blob/master/rust/arrow/src/compute/kernels/temporal.rs), so perhaps something to consider in future :)

these timestamp conversions could be useful as compute kernels

Thanks @nevi-me  -- I will add that to my list of potentials!
Andrew

authors: alamb nevi-me


text:
[Bug]: Website, quickstart-go looks not up-to-date

What happened?
I try to follow the steps on https://beam.apache.org/get-started/quickstart-go/ to run a wordcount with Go SDK and have some difficulties.
The command go get seems is not working anymore with latest versions of Go:
$ go version
go version go1.20.5 darwin/amd64

$ go get -u github.com/apache/beam/sdks/v2/go/pkg/beam
go: go.mod file not found in current directory or any parent directory.
	'go get' is no longer supported outside a module.
	To build and install a command, use 'go install' with a version,
	like 'go install example.com/cmd@latest'
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.

When I try to install the latest version with install command then it fails:
$ go install github.com/apache/beam/sdks/v2/go/pkg/beam@latest
package github.com/apache/beam/sdks/v2/go/pkg/beam is not a main package

When I try to specify a version explicitly I have the same result:
$ go install github.com/apache/beam/sdks/v2/go/pkg/beam@v2.47.0
package github.com/apache/beam/sdks/v2/go/pkg/beam is not a main package
$ echo $?
1

I'm a Go noobie, so, please let me know if I'm doing something wrong.
Issue Priority
Priority: 3 (minor)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[X] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

Ping @lostluck

Per the instructions produced by vy the Go  there, the user needs to be in a module for our instructions to work as intended.
See https://go.dev/doc/tutorial/getting-started for the basics of Go.
We try not to spend time teaching language operations in the beam docs since the point of Beam is to use the language you like already, not to force you to learn a new one.
I can't check and fix the quickstart instructions at present to clarify the module expectation however. In a car for several hours.

Yes, I agree about not teaching the basics of Go there, but I'd add more details on Quickstart Go page about how to run Beam Go pipeline properly and with all needed commands/steps (especially, for people like me, who just started their journey with Go).

Oh, I agree!
I just can't make the change from my phone.

authors: aromanenko-dev lostluck

repo: apache/beam


text:
[BEAM-8871] Support trySplit for ByteKeyRangeTracker

r: @lukecwik
cc: @youngoli

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Java PreCommit

(Sorry for adding this but cannot find a better moment ot ask). Curious question related to fractional splits (but not to this PR). Would testing trySplit for a RestrictionTracker would be enough to test that a SDF based IO that uses this kind of Restriction supports DWR correctly?
I am asking because in HBaseIO based on SDF it is one of the missing tests compared with the tests we had in the past for splitAtFranction https://github.com/apache/beam/blob/ec67a9374671ea9ae670fb0f3935ead2ebed7981/sdks/java/io/hbase/src/test/java/org/apache/beam/sdk/io/hbase/HBaseIOTest.java#L359
If it is not the case, any suggestions on how to test this?

(Sorry for adding this but cannot find a better moment ot ask). Curious question related to fractional splits (but not to this PR). Would testing trySplit for a RestrictionTracker would be enough to test that a SDF based IO that uses this kind of Restriction supports DWR correctly?
I am asking because in HBaseIO based on SDF it is one of the missing tests compared with the tests we had in the past for splitAtFranction
https://github.com/apache/beam/blob/ec67a9374671ea9ae670fb0f3935ead2ebed7981/sdks/java/io/hbase/src/test/java/org/apache/beam/sdk/io/hbase/HBaseIOTest.java#L359-L384
If it is not the case, any suggestions on how to test this?

Testing the restriction tracker is almost all that is needed for the most part for testing DWR.
The last bit would be to make sure that the DoFn doesn't assume its processing the initial restriction and interacts with the restriction tracker correctly. An example of a splittable DoFn that would work a lot of the time but be incorrect when splitting occurs would be something like:
@ProcessElement
public void process(RestrictionTracker<OffsetRange, Long> foo) {
  int start = foo.getCurrentRestriction().getFrom();
  int end = foo.getCurrentRestriction().getTo();
  from (int i = start; i < end; ++i) {
    ... produce output ...
  }
  tracker.tryClaim(end);
}

All the examples I know of stem from producing output before claiming. Also, moving the tryClaim before the loop would make the splittable DoFn correct but would prevent it from being able to effectively split.

Run Java PreCommit

Run Java PreCommit

Testing that output is only produced after the DoFn has done the claim call makes a lot of sense and ensuring that only the part that has been claimed was produced as output would cover correctness.

Thanks for the answer (again). It does not seem trivial to be able to test this from the outside of the DoFn without having to play with DoFnRunners and friends. Or is there something that would help to do this?

Testing that output is only produced after the DoFn has done the claim call makes a lot of sense and ensuring that only the part that has been claimed was produced as output would cover correctness.

Thanks for the answer (again). It does not seem trivial to be able to test this from the outside of the DoFn without having to play with DoFnRunners and friends. Or is there something that would help to do this?

Users should rely on PAssert/TestStream for correctness testing. As a simpler test, if the user doesn't use StartBundleContext/FinishBundleContext/ProcessContext, it should be easy for the user to instantiate their DoFn and call the methods on it directly if needs be.
Runners that support dynamic work rebalancing can tell users that their SDFs aren't behaving well by seeing that the SDF is not splitting or making a lot of unbalanced splits.

authors: boyuanzz iemejia lukecwik


text:
[BEAM-12128] replace usage of snippets_test_py3.py to snippets_test.py

Fixed reference to snippets_test_py3.py deleted at #14444

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

---

XLang

---

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @tvalentyn

Thanks for a quick fix, @lazylynx !

authors: lazylynx tvalentyn


text:
Avoid pickling unstable reference to moved proto classes. (#23739)

CloudPickle notices that schema_pb2 (from the closure of the locally defined reduce method) is not importable under its declared name (org.apache.beam...) and tries to pickle it (which fails due to proto classes themselves being unpicklable).  This avoids that error by moving it out.
Cherry-picking https://github.com/apache/beam/pull/23739 to the 2.43.0 release branch.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @robertwb

Run Python PreCommit

Run Python PreCommit


text:
Add LogElements as a Beam PTransform

Closes #23016
Adds LogElements as a valid PTransform with unit tests and updates the previous references to the standalone transform in the katas section.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @tvalentyn

Re Lint errors, you cand find them in Details -> ConsoleOutput, e.g. https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Commit/16316/consoleText
See also: s.apache.org/beam-python-dev-wiki that mentions lint/autoformatting tools

@tvalentyn Thank you for the quick response! All points have been fully addressed except for this one, where I kindly asked you to clarify a piece of information before proceeding.

Thanks, this looks good to me. Test failures are unrelated.

cc: @damccorm fyi.

authors: nivaldoh tvalentyn


text:
Add region to dataflowOptions as well.

@herohde sorry, apparently this didn't make it in here.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand:

[ ] What the pull request does
[ ] Why it does it
[ ] How it does it
[ ] Why this approach

[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Run go precommit


text:
Fix various lint errors in Python presubmit.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

Thanks for fixing this. Still seems to be failing.
11:23:39 Running isort for module apache_beam  gen_protos.py  setup.py  test_config.py:
11:23:45 ERROR: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/sdks/python/apache_beam/io/parquetio_it_test.py Imports are incorrectly sorted.
11:23:45 --- /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/sdks/python/apache_beam/io/parquetio_it_test.py:before	2018-12-11 10:20:43.606281
11:23:45 +++ /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/sdks/python/apache_beam/io/parquetio_it_test.py:after	2018-12-11 10:23:41.687212
11:23:45 @@ -23,9 +23,8 @@
11:23:45  import unittest
11:23:45  from collections import Counter
11:23:45
11:23:45 +import pyarrow as pa
11:23:45  from nose.plugins.attrib import attr
11:23:45 -
11:23:45 -import pyarrow as pa
11:23:45
11:23:45  from apache_beam import Create
11:23:45  from apache_beam import DoFn
11:23:45 ERROR: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/sdks/python/apache_beam/io/parquetio_test.py Imports are incorrectly sorted.
11:23:45 --- /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/sdks/python/apache_beam/io/parquetio_test.py:before	2018-12-11 10:20:43.606281
11:23:45 +++ /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/sdks/python/apache_beam/io/parquetio_test.py:after	2018-12-11 10:23:42.026171
11:23:45 @@ -25,13 +25,12 @@
11:23:45  import unittest
11:23:45
11:23:45  import hamcrest as hc
11:23:45 -from parameterized import param
11:23:45 -from parameterized import parameterized
11:23:45 -
11:23:45  import pandas
11:23:45  import pyarrow as pa
11:23:45  import pyarrow.lib as pl
11:23:45  import pyarrow.parquet as pq
11:23:45 +from parameterized import param
11:23:45 +from parameterized import parameterized
11:23:45
11:23:45  from apache_beam import Create
11:23:45  from apache_beam import Map
11:23:45 Command exited with non-zero status 1

isort can't seem to make up its mind... I'll revert that change.

ERROR: beam7 is offline; cannot locate JDK 1.8 (latest)

Run RAT PreCommit

authors: mxm robertwb


text:
[BEAM-8496] remove SDF translators from flink translator

fix the issue mentioned in the "beam_PostCommit_Java_Nexmark_Flink are failing continuously" thread
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Flink Runner Nexmark Tests

R: @mxm @tweise

Run Flink ValidatesRunner

It's odd. I am investigating this.

Run Flink ValidatesRunner

Run Flink ValidatesRunner

Run Flink Runner Nexmark Tests

reopen in another PR since jenkins trigger does not work.

Run Flink ValidatesRunner

Run Flink Runner Nexmark Tests

Run Flink ValidatesRunner

Run SQL Postcommit

Run SQL Postcommit

Run Java PreCommit

@mxm could you help to review again and run jenkins jobs?

Run Flink ValidatesRunner

retest this please

obbey your master Jenkins !!!

Run Flink ValidatesRunner

Run Flink Runner Nexmark Tests

Run SQL Postcommit

Thanks @iemejia! I have no idea how you did that :)

Run Flink Runner Nexmark Tests

Run Flink Runner Nexmark Tests

The test seems to have passed already, but it is not visible here: https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Flink_PR/25/

@vectorijk Could you squash the commits?

squashed the commits. thanks for the review! @mxm

authors: iemejia mxm tvalentyn vectorijk


text:
BROOKLYN-528: same proxy class for all entities of type

Fixes https://issues.apache.org/jira/browse/BROOKLYN-528
As well as fixing the space leak and decreasing the memory usage, this improves performance of entity creation from about 90 per second to about 230 per second (on my laptop).

LGTM, and the integration tests pass for me too. Will merge.

authors: aledsage geomacy

repo: apache/brooklyn-server


text:
BROOKLYN-244: Fix blueprint composer loading issue

This loads straight away CodeMirror when launching the composer from the composer tab, removing the requirement on loading the full list of applications contained within the catalog.
When coming from the catalog (editing a catalog item YAML), it will temporarily disable the editor, add an overlay displaying Loading item... until the item is full loaded from the REST server and added to the editor. At that point, the overlay will fade out and the editor enable again.

LGTM
Tested with a simulated 50 kb/s connection ... worked as expected

Thanks @tbouron and @m4rkmckenna
Merging now. Sounds like something to get into the next 0.9.0 release candidate.

Great, thank @aledsage!
If it gets into 0.9.0, you might also want #24

@aledsage +1 to including in the 0.9.0 release

authors: aledsage m4rkmckenna tbouron

repo: apache/brooklyn-ui


text:
Improve bst init, including 2.x upgrade assistance

See original issue on GitLab
In GitLab by [Gitlab user @devcurmudgeon] on Apr 28, 2020, 09:52
Currently the story for helping 1.x projects to migrate is not well-defined. Based on IRC discussion bst init could be extended to provide some help for users as follows:

running bst init where no project.conf exists should create project.conf as usual, but include all default configuration with comments describing what each config does

running bst init where there is an existing project.conf should attempt to upgrade the project.conf file to match current bst versioning, include any missing configuration, and report any known incompatibilities ideally with guiadance on how to fix

In GitLab by [Gitlab user @tristanvb] on Nov 4, 2020, 08:58
marked this issue as related to #1093

In GitLab by [Gitlab user @tristanvb] on Nov 4, 2020, 08:58
marked this issue as related to #1093

repo: apache/buildstream


text:
Documentation: example on how to run make check, or ninja test, etc

Documentation: example on how to run make check, or ninja test, etc
To run the test suite of an element.
I could read in details the BuildStream documentation, and find the way to run these additional commands, but small examples would be handy.

Perhaps this would be better suited for https://github.com/apache/buildstream-plugins since the plugins are now there.

authors: nanonyme swilmet


text:
Symlinks in the sandbox are broken by the path to the buildstream cache containing symlinks

See original issue on GitLab
In GitLab by [Gitlab user @jonathanmaw] on Jul 3, 2017, 10:39
The specific use-case found is that during a build that tried to execute rst2man, it turned out to be a broken symlink.
The symlink turned out to be
../../../../../../../../../home/jonathanmaw/.cache/buildstream/build/debian-python-x9gp05kc/root/etc/alternatives/rst2man
instead of /etc/alternatives/rst2man
According to [Gitlab user @tristanvb], the problem is utils.py:_relative_symlink_path() needs to be fixed, so that the realpath of the passed 'root' directory is used as well

In GitLab by [Gitlab user @jonathanmaw] on Jul 4, 2017, 11:16
created branch 44-symlinks-in-the-sandbox-are-broken-by-the-path-to-the-buildstream-cache-containing-symlinks

In GitLab by [Gitlab user @jonathanmaw] on Jul 4, 2017, 11:16
mentioned in merge request !47

In GitLab by [Gitlab user @tristanvb] on Jul 4, 2017, 13:20
Branch was merged, closing !

In GitLab by [Gitlab user @tristanvb] on Jul 4, 2017, 13:20
closed


text:
Google storage component doesn't work

I can't get the simple Google Storage route below to work in Camel K.
<routes id="google-storage-routes" xmlns="http://camel.apache.org/schema/spring">
    <route id="google-cloud-storage-route">
        <from uri="direct:google-cloud-storage-queue" />
        <convertBodyTo type="java.lang.String" />
        <to uri="google-storage:xxxxx" />
    </route>
</routes>

When run in Camel K using the command below, the deploy succeeds but an error is thrown since the google component isn't available.
kamel run google-storage-routes.xml --dev
...
[1] 2021-08-16 18:54:12,534 ERROR [org.apa.cam.qua.mai.CamelMainRuntime] (main) Failed to start application: org.apache.camel.FailedToCreateRouteException: Failed to create route google-cloud-storage-route at: >>> To[google-storage:xxxxxx] <<< in route: Route(google-cloud-storage-route)[From[direct:google-cloud-s... because of No endpoint could be found for: google-storage://xxxxx, please check your classpath contains the needed Camel component jar.

If I add the dependency, the deploy fails.
google-storage git:(master) ✗ kamel run -d camel-google-storage google-storage-routes.xml --dev
integration "google-storage-routes" created
Condition "IntegrationPlatformAvailable" is "True" for Integration google-storage-routes: default/camel-k
Progress: integration "google-storage-routes" in phase Initialization
Progress: integration "google-storage-routes" in phase Building Kit
Integration google-storage-routes in phase "Initialization"
Integration google-storage-routes in phase "Building Kit"
Condition "IntegrationKitAvailable" is "False" for Integration google-storage-routes: creating a new integration kit
Integration google-storage-routes subresource kit-c4dba6esrkndflf45f8g (Integration Kit) changed phase to "Build Submitted"
Integration google-storage-routes subresource kit-c4dba6esrkndflf45f8g (Build) changed phase to "Scheduling"
Integration google-storage-routes subresource kit-c4dba6esrkndflf45f8g (Build) changed phase to "Pending"
Integration google-storage-routes subresource kit-c4dba6esrkndflf45f8g (Build) changed phase to "Running"
Integration google-storage-routes subresource kit-c4dba6esrkndflf45f8g (Integration Kit) changed phase to "Build Running"
Integration google-storage-routes subresource kit-c4dba6esrkndflf45f8g (Build) changed phase to "Failed"

Why isn't Camel K automatically pulling the google-storage dependency, and why is it failing when the dependency is explicitly added? How should this be updated so that the google-storage component may be used within Camel K?

Hi @jeffgbradley2, I can't reproduce using the master branch. What version of camel-k are you using ? The latest release is v1.5.0, Thanks !

It's main branch now.
Il mar 17 ago 2021, 10:52 John Poth @.***> ha scritto:

Hi @jeffgbradley2 https://github.com/jeffgbradley2, I can't reproduce
using the master branch. What version of camel-k are you using ? The
latest release is v1.5.0
https://github.com/apache/camel-k/releases/tag/v1.5.0, Thanks !
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/camel-k/issues/2576#issuecomment-900116051,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABG6XV326QGM743U3RGPDQLT5IPOHANCNFSM5CIKEHUQ
.

We've tried with 1.4.0 and 1.5.0, same results.

I've also tested the same with 1.5.0 and it's working as expected. The only thing is that it requires a --build-property quarkus.google.cloud.project-id=test to startup.
The dependency should be detected automatically. In order to troubleshoot it you can execute kamel describe integration route and see if the dependency was indeed included. Also, you may have a look at the camel-k-operator pod log in order to check problems during the assembly phase. Finally, you may check the Build that is in charge to create a Kit via k describe builds kit-c4ec3jk3k6d8tsit3bvg

authors: jeffgbradley2 johnpoth oscerd squakez

repo: apache/camel-k


text:
feat(doc): network architecture

Release Note
feat(doc): network architecture

@oscerd @lburgazzoli wdyt?

Always good to provide more information. I think we should have some kind of best practices collection for camel-k, like tuning and this kind of information. +1. Thanks.

authors: oscerd squakez


text:
PDF integration tests failing with Camel 3.7.0-SNAPSHOT

The initial PDF creation step fails with:
2020-11-01T01:49:00.5763116Z 2020-11-01 01:49:00,570 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /pdf/createFromText failed, error id: da60e408-6679-45cb-b42d-51c2545cee4a-1: org.jboss.resteasy.spi.UnhandledException: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: pdf://create?font=Courier&fontSize=6&pageSize=PAGE_SIZE_A5 due to: Error binding property (fontSize=6) with name: fontSize on bean: pdf://create?font=Courier&fontSize=6&pageSize=PAGE_SIZE_A5 with value: 6
2020-11-01T01:49:00.5768188Z 	at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
2020-11-01T01:49:00.5770944Z 	at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
2020-11-01T01:49:00.5773593Z 	at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
2020-11-01T01:49:00.5776100Z 	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
2020-11-01T01:49:00.5778406Z 	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
2020-11-01T01:49:00.5780758Z 	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
2020-11-01T01:49:00.5786426Z 	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
2020-11-01T01:49:00.5789787Z 	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
2020-11-01T01:49:00.5791990Z 	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
2020-11-01T01:49:00.5794368Z 	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
2020-11-01T01:49:00.5797170Z 	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:131)
2020-11-01T01:49:00.5800355Z 	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.access$000(VertxRequestHandler.java:37)
2020-11-01T01:49:00.5802969Z 	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:94)
2020-11-01T01:49:00.5805490Z 	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
2020-11-01T01:49:00.5807766Z 	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
2020-11-01T01:49:00.5809634Z 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
2020-11-01T01:49:00.5811405Z 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
2020-11-01T01:49:00.5813130Z 	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
2020-11-01T01:49:00.5815001Z 	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
2020-11-01T01:49:00.5816459Z 	at java.base/java.lang.Thread.run(Thread.java:834)
2020-11-01T01:49:00.5817299Z 	at org.jboss.threads.JBossThread.run(JBossThread.java:479)
2020-11-01T01:49:00.5819293Z Caused by: org.apache.camel.ResolveEndpointFailedException: Failed to resolve endpoint: pdf://create?font=Courier&fontSize=6&pageSize=PAGE_SIZE_A5 due to: Error binding property (fontSize=6) with name: fontSize on bean: pdf://create?font=Courier&fontSize=6&pageSize=PAGE_SIZE_A5 with value: 6
2020-11-01T01:49:00.5822097Z 	at org.apache.camel.impl.engine.AbstractCamelContext.doGetEndpoint(AbstractCamelContext.java:902)
2020-11-01T01:49:00.5824731Z 	at org.apache.camel.impl.engine.AbstractCamelContext.getEndpoint(AbstractCamelContext.java:783)
2020-11-01T01:49:00.5827882Z 	at org.apache.camel.impl.engine.DefaultProducerTemplate.resolveMandatoryEndpoint(DefaultProducerTemplate.java:573)
2020-11-01T01:49:00.5830763Z 	at org.apache.camel.impl.engine.DefaultProducerTemplate.requestBody(DefaultProducerTemplate.java:411)
2020-11-01T01:49:00.5893185Z 	at org.apache.camel.quarkus.component.pdf.it.PdfResource.createFromText(PdfResource.java:53)
2020-11-01T01:49:00.5895986Z 	at org.apache.camel.quarkus.component.pdf.it.PdfResource_ClientProxy.createFromText(PdfResource_ClientProxy.zig:234)
2020-11-01T01:49:00.5898307Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-11-01T01:49:00.5900604Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-11-01T01:49:00.5903309Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-11-01T01:49:00.5905476Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
2020-11-01T01:49:00.5906999Z 	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
2020-11-01T01:49:00.5909022Z 	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
2020-11-01T01:49:00.5911763Z 	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:643)
2020-11-01T01:49:00.5915242Z 	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:507)
2020-11-01T01:49:00.5918456Z 	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:457)
2020-11-01T01:49:00.5994401Z 	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
2020-11-01T01:49:00.5997886Z 	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:459)
2020-11-01T01:49:00.6000431Z 	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:419)
2020-11-01T01:49:00.6002571Z 	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:393)
2020-11-01T01:49:00.6004696Z 	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:68)
2020-11-01T01:49:00.6006803Z 	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
2020-11-01T01:49:00.6007969Z 	... 17 more
2020-11-01T01:49:00.6009111Z Caused by: org.apache.camel.PropertyBindingException: Error binding property (fontSize=6) with name: fontSize on bean: pdf://create?font=Courier&fontSize=6&pageSize=PAGE_SIZE_A5 with value: 6
2020-11-01T01:49:00.6011909Z 	at org.apache.camel.support.PropertyBindingSupport.setSimplePropertyViaConfigurer(PropertyBindingSupport.java:849)
2020-11-01T01:49:00.6014881Z 	at org.apache.camel.support.PropertyBindingSupport.doSetPropertyValue(PropertyBindingSupport.java:609)
2020-11-01T01:49:00.6017638Z 	at org.apache.camel.support.PropertyBindingSupport.doBuildPropertyOgnlPath(PropertyBindingSupport.java:481)
2020-11-01T01:49:00.6020492Z 	at org.apache.camel.support.PropertyBindingSupport.doBindProperties(PropertyBindingSupport.java:376)
2020-11-01T01:49:00.6023155Z 	at org.apache.camel.support.PropertyBindingSupport.access$100(PropertyBindingSupport.java:87)
2020-11-01T01:49:00.6025296Z 	at org.apache.camel.support.PropertyBindingSupport$Builder.bind(PropertyBindingSupport.java:1877)
2020-11-01T01:49:00.6027346Z 	at org.apache.camel.support.DefaultEndpoint.setProperties(DefaultEndpoint.java:435)
2020-11-01T01:49:00.6029549Z 	at org.apache.camel.support.DefaultEndpoint.configureProperties(DefaultEndpoint.java:403)
2020-11-01T01:49:00.6031795Z 	at org.apache.camel.support.DefaultComponent.setProperties(DefaultComponent.java:387)
2020-11-01T01:49:00.6034193Z 	at org.apache.camel.component.pdf.PdfComponent.createEndpoint(PdfComponent.java:38)
2020-11-01T01:49:00.6036523Z 	at org.apache.camel.support.DefaultComponent.createEndpoint(DefaultComponent.java:156)
2020-11-01T01:49:00.6060870Z 	at org.apache.camel.impl.engine.AbstractCamelContext.doGetEndpoint(AbstractCamelContext.java:868)
2020-11-01T01:49:00.6062510Z 	... 37 more
2020-11-01T01:49:00.6064355Z Caused by: java.lang.ClassCastException: class java.lang.Double cannot be cast to class java.lang.Float (java.lang.Double and java.lang.Float are in module java.base of loader 'bootstrap')
2020-11-01T01:49:00.6066596Z 	at org.apache.camel.component.pdf.PdfEndpointConfigurer.configure(PdfEndpointConfigurer.java:46)
2020-11-01T01:49:00.6069835Z 	at org.apache.camel.support.PropertyBindingSupport.setSimplePropertyViaConfigurer(PropertyBindingSupport.java:847)
2020-11-01T01:49:00.6071918Z 	... 48 more

Should be fixed in camel-upstream. Let's check in the next camel-master build.

Ok, I was able to run pdf itests against the last camel SNAPSHOT.

authors: aldettinger jamesnetherton

repo: apache/camel-quarkus


text:
CAMEL-17121: present the awaitConsumer from blocking indefinitely

@davsclaus this should fix the block (in case we want to retain the patch instead of reverting it).

repo: apache/camel


text:
[CARBONDATA-1910]do not allow tupleid, referenceid and positionReference as columns names

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[X] Any interfaces changed?
NA

[X] Any backward compatibility impacted?
NA

[X] Document update required?
NA

[X] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
UTs are added to test the scenarios

[X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2133/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2422/

retest this please

retest sdv please

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/929/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2158/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2443/

LGTM

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2318/

retest this please

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1102/

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2550/

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1106/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2323/

LGTM

authors: CarbonDataQA akashrn5 ravipesala sraghunandan

repo: apache/carbondata


text:
[CARBONDATA-2145] Refactored PreAggregate functionality for dictionary include

Description: Add the count to measure column only when the column is dictionary type in maintable.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[x] Any interfaces changed? No

[x] Any backward compatibility impacted? No

[x] Document update required? No

[x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
Ran already wriiten Unit Test case to test the functionality.
Added Unit Test Case to check count on string type.
 - How it is tested? Please attach test report.
 - Is it a performance related change? Please attach the performance test report.
 - Any additional information to help reviewers in testing this change.

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.  (N/A)

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3573/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2335/

@kumarvishal09 please review.

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3422/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3036/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4280/

authors: CarbonDataQA SangeetaGulia ravipesala


text:
[HOTFIX] Set default header for load data in legacy enviroment

Why is this PR needed?
The old code has been commented, this PR will revert the code.
What changes were proposed in this PR?
Set default header for load data in legacy enviroment
Does this PR introduce any user interface change?

No

Is any new testcase added?

No

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1274/

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2993/

LGTM

authors: CarbonDataQA1 QiangCai ajantha-bhat


text:
CLOUDSTACK-8608: [VMware] System VMs failed to start due to permissions issue. Provide permissions to template folder when mounted on management server.

[VMware] System VMs failed to start due to permissions issue. Provide permissions to template folder when mounted on management server.
This closes #555

@rhtyd Can you run VMware CI on this PR.

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-511

@blueorangutan test centos7 vmware-55u3

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-55u3) has been kicked to run smoke tests

Trillian test result (tid-867)
Environment: vmware-55u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 48400 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1875-t867-vmware-55u3.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Test completed. 46 look ok, 3 have error(s)

Test
Result
Time (s)
Test File

test_04_rvpc_privategw_static_routes
Failure
872.15
test_privategw_acl.py

test_02_redundant_VPC_default_routes
Error
237.39
test_vpc_redundant.py

test_02_list_snapshots_with_removed_data_store
Error
75.67
test_snapshots.py

test_02_list_snapshots_with_removed_data_store
Error
80.75
test_snapshots.py

test_01_vpc_site2site_vpn
Success
375.63
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
186.40
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
596.74
test_vpc_vpn.py

test_02_VPC_default_routes
Success
368.38
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
750.75
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
665.98
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1527.05
test_vpc_redundant.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Success
691.66
test_vpc_redundant.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1373.64
test_vpc_redundant.py

test_09_delete_detached_volume
Success
20.62
test_volumes.py

test_06_download_detached_volume
Success
70.58
test_volumes.py

test_05_detach_volume
Success
100.22
test_volumes.py

test_04_delete_attached_volume
Success
15.17
test_volumes.py

test_03_download_attached_volume
Success
25.26
test_volumes.py

test_02_attach_volume
Success
58.86
test_volumes.py

test_01_create_volume
Success
519.45
test_volumes.py

test_change_service_offering_for_vm_with_snapshots
Success
534.17
test_vm_snapshots.py

test_03_delete_vm_snapshots
Success
275.17
test_vm_snapshots.py

test_02_revert_vm_snapshots
Success
227.10
test_vm_snapshots.py

test_01_test_vm_volume_snapshot
Success
191.26
test_vm_snapshots.py

test_01_create_vm_snapshots
Success
161.62
test_vm_snapshots.py

test_deploy_vm_multiple
Success
216.81
test_vm_life_cycle.py

test_deploy_vm
Success
0.02
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.02
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
26.68
test_vm_life_cycle.py

test_09_expunge_vm
Success
125.18
test_vm_life_cycle.py

test_08_migrate_vm
Success
101.03
test_vm_life_cycle.py

test_07_restore_vm
Success
0.06
test_vm_life_cycle.py

test_06_destroy_vm
Success
10.14
test_vm_life_cycle.py

test_03_reboot_vm
Success
5.10
test_vm_life_cycle.py

test_02_start_vm
Success
20.17
test_vm_life_cycle.py

test_01_stop_vm
Success
10.11
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
241.44
test_templates.py

test_08_list_system_templates
Success
0.02
test_templates.py

test_07_list_public_templates
Success
0.03
test_templates.py

test_05_template_permissions
Success
0.04
test_templates.py

test_04_extract_template
Success
10.16
test_templates.py

test_03_delete_template
Success
5.08
test_templates.py

test_02_edit_template
Success
90.18
test_templates.py

test_01_create_template
Success
120.87
test_templates.py

test_10_destroy_cpvm
Success
236.61
test_ssvm.py

test_09_destroy_ssvm
Success
268.53
test_ssvm.py

test_08_reboot_cpvm
Success
366.63
test_ssvm.py

test_07_reboot_ssvm
Success
308.41
test_ssvm.py

test_06_stop_cpvm
Success
176.51
test_ssvm.py

test_05_stop_ssvm
Success
173.30
test_ssvm.py

test_04_cpvm_internals
Success
1.02
test_ssvm.py

test_03_ssvm_internals
Success
3.32
test_ssvm.py

test_02_list_cpvm_vm
Success
0.09
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.09
test_ssvm.py

test_01_snapshot_root_disk
Success
66.15
test_snapshots.py

test_04_change_offering_small
Success
92.44
test_service_offerings.py

test_03_delete_service_offering
Success
0.04
test_service_offerings.py

test_02_edit_service_offering
Success
0.06
test_service_offerings.py

test_01_create_service_offering
Success
0.08
test_service_offerings.py

test_02_sys_template_ready
Success
0.14
test_secondary_storage.py

test_01_sys_vm_start
Success
0.13
test_secondary_storage.py

test_09_reboot_router
Success
175.86
test_routers.py

test_08_start_router
Success
155.82
test_routers.py

test_07_stop_router
Success
25.21
test_routers.py

test_06_router_advanced
Success
0.04
test_routers.py

test_05_router_basic
Success
0.03
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.53
test_routers.py

test_03_restart_network_cleanup
Success
150.81
test_routers.py

test_02_router_internal_adv
Success
0.88
test_routers.py

test_01_router_internal_basic
Success
0.47
test_routers.py

test_router_dns_guestipquery
Success
76.65
test_router_dns.py

test_router_dns_externalipquery
Success
0.06
test_router_dns.py

test_router_dhcphosts
Success
175.51
test_router_dhcphosts.py

test_router_dhcp_opts
Success
21.30
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
0.05
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
30.28
test_reset_vm_on_reboot.py

test_createRegion
Success
0.03
test_regions.py

test_create_pvlan_network
Success
5.31
test_pvlan.py

test_dedicatePublicIpRange
Success
0.33
test_public_ip_range.py

test_03_vpc_privategw_restart_vpc_cleanup
Success
1021.87
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Success
690.34
test_privategw_acl.py

test_01_vpc_privategw_acl
Success
212.22
test_privategw_acl.py

test_01_primary_storage_nfs
Success
38.41
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.16
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
15.32
test_portable_publicip.py

test_isolate_network_password_server
Success
93.92
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.09
test_over_provisioning.py

test_oobm_zchange_password
Success
30.59
test_outofbandmanagement.py

test_oobm_multiple_mgmt_server_ownership
Success
16.27
test_outofbandmanagement.py

test_oobm_issue_power_status
Success
10.21
test_outofbandmanagement.py

test_oobm_issue_power_soft
Success
15.30
test_outofbandmanagement.py

test_oobm_issue_power_reset
Success
15.25
test_outofbandmanagement.py

test_oobm_issue_power_on
Success
10.26
test_outofbandmanagement.py

test_oobm_issue_power_off
Success
15.28
test_outofbandmanagement.py

test_oobm_issue_power_cycle
Success
15.69
test_outofbandmanagement.py

test_oobm_enabledisable_across_clusterzones
Success
82.17
test_outofbandmanagement.py

test_oobm_enable_feature_valid
Success
5.11
test_outofbandmanagement.py

test_oobm_enable_feature_invalid
Success
0.28
test_outofbandmanagement.py

test_oobm_disable_feature_valid
Success
5.15
test_outofbandmanagement.py

test_oobm_disable_feature_invalid
Success
0.07
test_outofbandmanagement.py

test_oobm_configure_invalid_driver
Success
0.06
test_outofbandmanagement.py

test_oobm_configure_default_driver
Success
0.06
test_outofbandmanagement.py

test_oobm_background_powerstate_sync
Success
23.30
test_outofbandmanagement.py

test_extendPhysicalNetworkVlan
Success
15.25
test_non_contigiousvlan.py

test_01_nic
Success
468.92
test_nic.py

test_releaseIP
Success
503.86
test_network.py

test_reboot_router
Success
610.18
test_network.py

test_public_ip_user_account
Success
10.20
test_network.py

test_public_ip_admin_account
Success
40.38
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
76.47
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.34
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
124.88
test_network.py

test_delete_account
Success
302.46
test_network.py

test_02_port_fwd_on_non_src_nat
Success
55.51
test_network.py

test_01_port_fwd_on_src_nat
Success
111.72
test_network.py

test_nested_virtualization_vmware
Success
328.08
test_nested_virtualization.py

test_nic_secondaryip_add_remove
Success
191.89
test_multipleips_per_nic.py

login_test_saml_user
Success
17.82
test_login.py

test_assign_and_removal_lb
Success
148.37
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
207.25
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
207.61
test_loadbalance.py

test_03_list_snapshots
Success
0.06
test_list_ids_parameter.py

test_02_list_templates
Success
0.03
test_list_ids_parameter.py

test_01_list_volumes
Success
0.05
test_list_ids_parameter.py

test_07_list_default_iso
Success
0.04
test_iso.py

test_05_iso_permissions
Success
0.04
test_iso.py

test_04_extract_Iso
Success
5.27
test_iso.py

test_03_delete_iso
Success
95.19
test_iso.py

test_02_edit_iso
Success
0.06
test_iso.py

test_01_create_iso
Success
20.68
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
569.14
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
388.49
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
1073.04
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
824.05
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.18
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.13
test_global_settings.py

test_rolepermission_lifecycle_update
Success
5.82
test_dynamicroles.py

test_rolepermission_lifecycle_list
Success
5.71
test_dynamicroles.py

test_rolepermission_lifecycle_delete
Success
5.58
test_dynamicroles.py

test_rolepermission_lifecycle_create
Success
5.60
test_dynamicroles.py

test_rolepermission_lifecycle_concurrent_updates
Success
5.68
test_dynamicroles.py

test_role_lifecycle_update_role_inuse
Success
5.61
test_dynamicroles.py

test_role_lifecycle_update
Success
10.75
test_dynamicroles.py

test_role_lifecycle_list
Success
5.94
test_dynamicroles.py

test_role_lifecycle_delete
Success
10.68
test_dynamicroles.py

test_role_lifecycle_create
Success
5.78
test_dynamicroles.py

test_role_inuse_deletion
Success
5.58
test_dynamicroles.py

test_role_account_acls_multiple_mgmt_servers
Success
7.07
test_dynamicroles.py

test_role_account_acls
Success
7.12
test_dynamicroles.py

test_default_role_deletion
Success
5.73
test_dynamicroles.py

test_04_create_fat_type_disk_offering
Success
0.05
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.03
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.03
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.05
test_disk_offerings.py

test_01_create_disk_offering
Success
0.06
test_disk_offerings.py

test_deployvm_userdispersing
Success
55.56
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
80.72
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
231.49
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
35.45
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
156.22
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
5.57
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
5.58
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
5.65
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
201.71
test_deploy_vm_iso.py

test_3d_gpu_support
Success
451.01
test_deploy_vgpu_enabled_vm.py

test_DeployVmAntiAffinityGroup
Success
227.64
test_affinity_groups.py

test_08_resize_volume
Skipped
5.08
test_volumes.py

test_07_resize_fail
Skipped
15.24
test_volumes.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_static_role_account_acls
Skipped
0.02
test_staticroles.py

test_11_ss_nfs_version_on_ssvm
Skipped
0.02
test_ssvm.py

test_01_scale_vm
Skipped
66.29
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.03
test_primary_storage.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
0.76
test_deploy_vgpu_enabled_vm.py

@rhtyd Thanks for running tests. The test failures/errors above are failing in other PRs as well, not related to the changes in this PR.

@sateesh-chodapuneedi @rhtyd  Please review the changes.

LGTM on the code changes.

LGTM.

LGTM, @sureshanaparti since this is a critical bugfix wher systemvm can fail to start, can you change PR's base branch to 4.9?

@rhtyd Changed PR base branch to 4.9.

Thanks @sureshanaparti
@blueorangutan package

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-675

@blueorangutan test centos7 vmware-55u3

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-55u3) has been kicked to run smoke tests

Trillian test result (tid-1033)
Environment: vmware-55u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 52073 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1875-t1033-vmware-55u3.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Test completed. 46 look ok, 3 have error(s)

Test
Result
Time (s)
Test File

test_04_rvpc_privategw_static_routes
Failure
1082.38
test_privategw_acl.py

test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers
Error
116.09
test_vpc_redundant.py

test_02_list_snapshots_with_removed_data_store
Error
95.72
test_snapshots.py

test_02_list_snapshots_with_removed_data_store
Error
100.79
test_snapshots.py

test_01_vpc_site2site_vpn
Success
475.62
test_vpc_vpn.py

test_01_vpc_remote_access_vpn
Success
246.55
test_vpc_vpn.py

test_01_redundant_vpc_site2site_vpn
Success
762.79
test_vpc_vpn.py

test_02_VPC_default_routes
Success
379.72
test_vpc_router_nics.py

test_01_VPC_nics_after_destroy
Success
891.05
test_vpc_router_nics.py

test_05_rvpc_multi_tiers
Success
905.32
test_vpc_redundant.py

test_04_rvpc_network_garbage_collector_nics
Success
1697.64
test_vpc_redundant.py

test_02_redundant_VPC_default_routes
Success
899.25
test_vpc_redundant.py

test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL
Success
1607.98
test_vpc_redundant.py

test_09_delete_detached_volume
Success
35.84
test_volumes.py

test_06_download_detached_volume
Success
115.63
test_volumes.py

test_05_detach_volume
Success
105.23
test_volumes.py

test_04_delete_attached_volume
Success
20.18
test_volumes.py

test_03_download_attached_volume
Success
20.23
test_volumes.py

test_02_attach_volume
Success
63.77
test_volumes.py

test_01_create_volume
Success
526.17
test_volumes.py

test_03_delete_vm_snapshots
Success
280.22
test_vm_snapshots.py

test_02_revert_vm_snapshots
Success
239.32
test_vm_snapshots.py

test_01_create_vm_snapshots
Success
161.73
test_vm_snapshots.py

test_deploy_vm_multiple
Success
337.28
test_vm_life_cycle.py

test_deploy_vm
Success
0.02
test_vm_life_cycle.py

test_advZoneVirtualRouter
Success
0.02
test_vm_life_cycle.py

test_10_attachAndDetach_iso
Success
26.69
test_vm_life_cycle.py

test_09_expunge_vm
Success
185.22
test_vm_life_cycle.py

test_08_migrate_vm
Success
80.85
test_vm_life_cycle.py

test_07_restore_vm
Success
0.08
test_vm_life_cycle.py

test_06_destroy_vm
Success
10.10
test_vm_life_cycle.py

test_03_reboot_vm
Success
5.13
test_vm_life_cycle.py

test_02_start_vm
Success
25.20
test_vm_life_cycle.py

test_01_stop_vm
Success
5.08
test_vm_life_cycle.py

test_CreateTemplateWithDuplicateName
Success
487.18
test_templates.py

test_08_list_system_templates
Success
0.02
test_templates.py

test_07_list_public_templates
Success
0.02
test_templates.py

test_05_template_permissions
Success
0.04
test_templates.py

test_04_extract_template
Success
30.29
test_templates.py

test_03_delete_template
Success
5.07
test_templates.py

test_02_edit_template
Success
90.09
test_templates.py

test_01_create_template
Success
211.03
test_templates.py

test_10_destroy_cpvm
Success
266.53
test_ssvm.py

test_09_destroy_ssvm
Success
304.61
test_ssvm.py

test_08_reboot_cpvm
Success
156.27
test_ssvm.py

test_07_reboot_ssvm
Success
158.23
test_ssvm.py

test_06_stop_cpvm
Success
196.61
test_ssvm.py

test_05_stop_ssvm
Success
233.57
test_ssvm.py

test_04_cpvm_internals
Success
1.03
test_ssvm.py

test_03_ssvm_internals
Success
3.64
test_ssvm.py

test_02_list_cpvm_vm
Success
0.08
test_ssvm.py

test_01_list_sec_storage_vm
Success
0.09
test_ssvm.py

test_01_snapshot_root_disk
Success
76.17
test_snapshots.py

test_04_change_offering_small
Success
92.02
test_service_offerings.py

test_03_delete_service_offering
Success
0.03
test_service_offerings.py

test_02_edit_service_offering
Success
0.05
test_service_offerings.py

test_01_create_service_offering
Success
0.08
test_service_offerings.py

test_02_sys_template_ready
Success
0.09
test_secondary_storage.py

test_01_sys_vm_start
Success
0.14
test_secondary_storage.py

test_09_reboot_router
Success
165.76
test_routers.py

test_08_start_router
Success
135.72
test_routers.py

test_07_stop_router
Success
20.16
test_routers.py

test_06_router_advanced
Success
0.04
test_routers.py

test_05_router_basic
Success
0.03
test_routers.py

test_04_restart_network_wo_cleanup
Success
5.53
test_routers.py

test_03_restart_network_cleanup
Success
185.92
test_routers.py

test_02_router_internal_adv
Success
0.86
test_routers.py

test_01_router_internal_basic
Success
0.45
test_routers.py

test_router_dns_guestipquery
Success
76.65
test_router_dns.py

test_router_dns_externalipquery
Success
0.06
test_router_dns.py

test_router_dhcphosts
Success
166.39
test_router_dhcphosts.py

test_router_dhcp_opts
Success
21.30
test_router_dhcphosts.py

test_01_updatevolumedetail
Success
5.09
test_resource_detail.py

test_01_reset_vm_on_reboot
Success
100.57
test_reset_vm_on_reboot.py

test_createRegion
Success
0.03
test_regions.py

test_create_pvlan_network
Success
5.16
test_pvlan.py

test_dedicatePublicIpRange
Success
0.27
test_public_ip_range.py

test_03_vpc_privategw_restart_vpc_cleanup
Success
1186.14
test_privategw_acl.py

test_02_vpc_privategw_static_routes
Success
810.66
test_privategw_acl.py

test_01_vpc_privategw_acl
Success
267.45
test_privategw_acl.py

test_01_primary_storage_nfs
Success
38.15
test_primary_storage.py

test_createPortablePublicIPRange
Success
15.13
test_portable_publicip.py

test_createPortablePublicIPAcquire
Success
20.36
test_portable_publicip.py

test_isolate_network_password_server
Success
95.47
test_password_server.py

test_UpdateStorageOverProvisioningFactor
Success
0.09
test_over_provisioning.py

test_oobm_zchange_password
Success
30.51
test_outofbandmanagement.py

test_oobm_multiple_mgmt_server_ownership
Success
16.26
test_outofbandmanagement.py

test_oobm_issue_power_status
Success
10.20
test_outofbandmanagement.py

test_oobm_issue_power_soft
Success
15.24
test_outofbandmanagement.py

test_oobm_issue_power_reset
Success
15.26
test_outofbandmanagement.py

test_oobm_issue_power_on
Success
15.25
test_outofbandmanagement.py

test_oobm_issue_power_off
Success
15.24
test_outofbandmanagement.py

test_oobm_issue_power_cycle
Success
15.25
test_outofbandmanagement.py

test_oobm_enabledisable_across_clusterzones
Success
87.20
test_outofbandmanagement.py

test_oobm_enable_feature_valid
Success
5.12
test_outofbandmanagement.py

test_oobm_enable_feature_invalid
Success
0.10
test_outofbandmanagement.py

test_oobm_disable_feature_valid
Success
0.11
test_outofbandmanagement.py

test_oobm_disable_feature_invalid
Success
0.08
test_outofbandmanagement.py

test_oobm_configure_invalid_driver
Success
0.05
test_outofbandmanagement.py

test_oobm_configure_default_driver
Success
0.05
test_outofbandmanagement.py

test_oobm_background_powerstate_sync
Success
23.29
test_outofbandmanagement.py

test_extendPhysicalNetworkVlan
Success
15.23
test_non_contigiousvlan.py

test_01_nic
Success
534.03
test_nic.py

test_releaseIP
Success
357.58
test_network.py

test_reboot_router
Success
739.30
test_network.py

test_public_ip_user_account
Success
10.22
test_network.py

test_public_ip_admin_account
Success
40.24
test_network.py

test_network_rules_acquired_public_ip_3_Load_Balancer_Rule
Success
76.45
test_network.py

test_network_rules_acquired_public_ip_2_nat_rule
Success
61.31
test_network.py

test_network_rules_acquired_public_ip_1_static_nat_rule
Success
124.92
test_network.py

test_delete_account
Success
307.22
test_network.py

test_02_port_fwd_on_non_src_nat
Success
55.56
test_network.py

test_01_port_fwd_on_src_nat
Success
111.78
test_network.py

test_nic_secondaryip_add_remove
Success
252.26
test_multipleips_per_nic.py

test_list_zones_metrics
Success
0.18
test_metrics_api.py

test_list_volumes_metrics
Success
5.34
test_metrics_api.py

test_list_vms_metrics
Success
236.54
test_metrics_api.py

test_list_pstorage_metrics
Success
0.21
test_metrics_api.py

test_list_infrastructure_metrics
Success
0.32
test_metrics_api.py

test_list_hosts_metrics
Success
0.27
test_metrics_api.py

test_list_clusters_metrics
Success
0.26
test_metrics_api.py

login_test_saml_user
Success
17.85
test_login.py

test_assign_and_removal_lb
Success
148.64
test_loadbalance.py

test_02_create_lb_rule_non_nat
Success
207.20
test_loadbalance.py

test_01_create_lb_rule_src_nat
Success
208.11
test_loadbalance.py

test_03_list_snapshots
Success
0.06
test_list_ids_parameter.py

test_02_list_templates
Success
0.03
test_list_ids_parameter.py

test_01_list_volumes
Success
0.02
test_list_ids_parameter.py

test_07_list_default_iso
Success
0.04
test_iso.py

test_05_iso_permissions
Success
0.05
test_iso.py

test_04_extract_Iso
Success
5.17
test_iso.py

test_03_delete_iso
Success
95.20
test_iso.py

test_02_edit_iso
Success
0.04
test_iso.py

test_01_create_iso
Success
20.73
test_iso.py

test_04_rvpc_internallb_haproxy_stats_on_all_interfaces
Success
629.46
test_internal_lb.py

test_03_vpc_internallb_haproxy_stats_on_all_interfaces
Success
470.32
test_internal_lb.py

test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80
Success
1137.59
test_internal_lb.py

test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80
Success
911.07
test_internal_lb.py

test_dedicateGuestVlanRange
Success
10.20
test_guest_vlan_range.py

test_UpdateConfigParamWithScope
Success
0.11
test_global_settings.py

test_rolepermission_lifecycle_update
Success
5.79
test_dynamicroles.py

test_rolepermission_lifecycle_list
Success
5.69
test_dynamicroles.py

test_rolepermission_lifecycle_delete
Success
5.59
test_dynamicroles.py

test_rolepermission_lifecycle_create
Success
5.62
test_dynamicroles.py

test_rolepermission_lifecycle_concurrent_updates
Success
5.71
test_dynamicroles.py

test_role_lifecycle_update_role_inuse
Success
5.60
test_dynamicroles.py

test_role_lifecycle_update
Success
10.69
test_dynamicroles.py

test_role_lifecycle_list
Success
5.61
test_dynamicroles.py

test_role_lifecycle_delete
Success
10.63
test_dynamicroles.py

test_role_lifecycle_create
Success
5.68
test_dynamicroles.py

test_role_inuse_deletion
Success
5.71
test_dynamicroles.py

test_role_account_acls_multiple_mgmt_servers
Success
7.28
test_dynamicroles.py

test_role_account_acls
Success
7.40
test_dynamicroles.py

test_default_role_deletion
Success
5.71
test_dynamicroles.py

test_04_create_fat_type_disk_offering
Success
0.05
test_disk_offerings.py

test_03_delete_disk_offering
Success
0.03
test_disk_offerings.py

test_02_edit_disk_offering
Success
0.03
test_disk_offerings.py

test_02_create_sparse_type_disk_offering
Success
0.04
test_disk_offerings.py

test_01_create_disk_offering
Success
0.07
test_disk_offerings.py

test_deployvm_userdispersing
Success
135.91
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userconcentrated
Success
65.61
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_firstfit
Success
251.40
test_deploy_vms_with_varied_deploymentplanners.py

test_deployvm_userdata_post
Success
30.35
test_deploy_vm_with_userdata.py

test_deployvm_userdata
Success
186.12
test_deploy_vm_with_userdata.py

test_02_deploy_vm_root_resize
Success
5.57
test_deploy_vm_root_resize.py

test_01_deploy_vm_root_resize
Success
5.59
test_deploy_vm_root_resize.py

test_00_deploy_vm_root_resize
Success
5.68
test_deploy_vm_root_resize.py

test_deploy_vm_from_iso
Success
236.89
test_deploy_vm_iso.py

test_DeployVmAntiAffinityGroup
Success
216.47
test_affinity_groups.py

test_08_resize_volume
Skipped
10.12
test_volumes.py

test_07_resize_fail
Skipped
15.21
test_volumes.py

test_06_copy_template
Skipped
0.00
test_templates.py

test_static_role_account_acls
Skipped
0.01
test_staticroles.py

test_01_scale_vm
Skipped
66.39
test_scale_vm.py

test_01_primary_storage_iscsi
Skipped
0.05
test_primary_storage.py

test_06_copy_iso
Skipped
0.00
test_iso.py

test_deploy_vgpu_enabled_vm
Skipped
0.00
test_deploy_vgpu_enabled_vm.py

The above intermitten failures are not related to this PR changes.

@sureshanaparti please rebase/fix the branch

@sureshanaparti ping, can you fix the conflicts?

@sureshanaparti please rebase and re-open if still relevant

authors: DaanHoogland SudharmaJain blueorangutan rhtyd sureshanaparti

repo: apache/cloudstack


text:
Text 174: ScriptStringLookup does not accept ":"

The usage of ":" can be pretty usefull for inside a ScriptStringLookup
For example ternary operations :
${script:javascript:'${myKey:-}' == '' ? 'Not Present' : 'Present' }
Today it's not possible to do that because the lookup only expect one ":"
final String[] keys = key.split(SPLIT_STR);
But we can limit the number of split to two:
final String[] keys = key.split(SPLIT_STR, 2);
Thanks a lot

Coverage decreased (-0.02%) to 97.962% when pulling 776f95b7b78c1737ca6a93830f724425075f1399 on furkilic:TEXT-174-ScriptStringLookup into e72bec3c84c582c4abbe02e1b81eec3c1373d365 on apache:master.

authors: coveralls furkilic

repo: apache/commons-text

license_source: bigquery


text:
Try catch finally cursors

Because lollipop can return null cursors here I try catch them!

Hi @manugb, is this the reason of why I'm getting

android.database.CursorWindowAllocationException: Cursor window could not be created from binder.

Thanks

Cordova CI Build has completed successfully.
Commit     - Link
Dashboard - Link

Builder Name
Console Output
Test Report
Device Logs

Windows 8.1 Store
Link
Link
Link

Windows 10  Store
Link
Link
Link

Windows 8.1 Phone
Link
Link
Link

iOS
Link
Link
Link

Android Mac
Link
Link
Link

Can one of the admins verify this patch?

There are merge conflicts in this pull request, and I believe a more complete and up-to-date implementation is in #146 (along with try/catch protection when interacting with the cursor).
Will close.

authors: cordova-qa filmaj manugb nicoabie

repo: apache/cordova-plugin-contacts


text:
64560 add metrics bulk docs count

Add a new counter "bulk_request_docs" to track the number of documents being handled as part of the bulk request.
FogBugz 64560.

Why? When you have a line above: couch_stats:update_histogram([couchdb, httpd, bulk_docs], length(DocsArray)) which is more informative than just constantly incremented counter.

@kxepal ? I have asked the same question in 64560 and discussed it with Aaron. As per my understanding, counter gives a different perspective and probably easy to identify the spikes and trends? Can you add a comment to Aaron/Russel/Ops in 64560 and get his perspective on why this metric would help Operations?

@brkolla Sorry, I can't since I'm not from Cloudant team. Can you ping them here or share the reason why? because I'm -1 on having two metrics that does the same thing while one of them (histogram) is actually what is really useful.

I am actually asking Aaron about this in Slack. To be honest, I don't fully understand the difference between the histogram and the regular counter and if its possible to have one metric that does the both. Will ask around and find out more.

@brkolla Histogram uses sliding time window and provides percentile-wise information about the recorded numbers. Just compare output of both these metrics to see the difference.

Yeah I agree with @kxepal on this, using a histogram is the correct choice here. It looks like there was also some confusion on this issue as @rnewson snuck in the actual fix for this issue back in https://github.com/cloudant/couchdb-chttpd/commit/a49c47417cc9190f8b5afe0c058c9a742a6cf587.
I do think it's worthwhile to also track the doc count in _bulk_get and that should be switched to a histogram like the above commit. The big question there is whether to use a different histogram to be able to distinguish between _bulk_get reads and _bulk_docs writes. I think having them separate would be reasonable.
Also, we should get similar stats for quantify of docs returned and provided in _all_docs and views and what not.

Oh, it also counts _bulk_get. I miss that moment, sorry. But that should be a different metric and should have a correct name anyway.

I think I have to revisit this and come up with a new metric to track the bulk gets and probably use one metric that tracks all the request that would result in bulk gets (_all_docs etc..) . We are still discussing on how we can do that.
I guess I will have to close this PR as we are not going to go ahead with this.

Yeah agreed, _bulk_get is sufficiently different we should track it separately.

@chewbranca I have updated the code to add a new metric bulk_reads to track the number of doc reads done as part of the bulk_get. Can you review this code?

@kxepal I am not really sure, but this seems to be only used during the replication. I guess, we could change the name to bulk_get_docs? @chewbranca?

@brkolla Not only. Replication is just a client for CouchDB. Other users may be this endpoint in own way (and for profit!).

authors: brkolla chewbranca kxepal

repo: apache/couchdb-chttpd


text:
77984 upgrade mrargs record phase2

Second phase for #mrargs{} record upgrade.

I'll use a +1 from PR #76.

+1 for posterity.

authors: davisp iilyak

repo: apache/couchdb-fabric


text:
Performance regression querying changes using _doc_ids filter

When upgrading from v1.7.1 to v2.2.0 I noticed our replication was taking longer. I investigated further and found the problem was specifically in relation to the initial request for changes. We use the _doc_ids filter so we can replicate only certain documents and this has been stable and performant on the 1.x versions.
Steps to Reproduce
Use this node script to create a database and fill it with 1 million docs and then query it for specific IDs.
Context
In my testing using the above script I got responses in 1 to 2ms on v1.7.1 and 2500 to 2600ms on 2.2.0.
In our production database with real world data and about 8 million docs it takes less than a second on v1.7.1 and around 40 seconds on v2.2.0.
This has affected real world performance for users trying to replicate their data.
Your Environment

Version used: 2.2.0 (from 1.7.1)
Operating System: Ubuntu 1804
Link to your project: https://github.com/medic/medic-webapp/

What I've tried

changing clustering from n=3 q=8 to n=1 q=1 - no improvement
changing the changes_doc_ids_optimization_threshold at 1, 100, and very large - no improvement
using a very large seq_interval parameter - no improvement
using a mango selector - worse performance

ok, I think I get this finally. Before 2.0, we had an optimization for _doc_ids and _design filter;
commit bfa0a8900163edd4f85c7bbf5b595de9885cfbf9
Author: Filipe David Borba Manana <fdmanana@apache.org>
Date:   Tue Sep 20 22:55:29 2011 +0000

    Efficient implementation of builtin filters

    Currently, the builtin changes filters "_doc_ids" and "_design"
    are not very efficient because they fold the entire seq btree
    and then filter the values by document ID.
    This implementation avoids that by doing direct lookups against
    the id btree, and then, for continuous changes requests, it
    just listens for database update events and does partial seq
    btree folds.

    COUCHDB-1288

The clustered code for _changes does not use it.

we pass {default,main_only} down as the filter, so the optimization does not fire.
It's not yet clear if this can be restored given how the clustered version of changes feed has to work, but at least this is progress.

so fabric_rpc:changes always calls couch_db:fold_changes .

@garethbowen This is fixed by #1771 and will appear in CouchDB 2.3.0.

Awesome. Thanks for the quick turnaround!

I ran the test script above against 2.3.0-RC1 and got responses in 4 to 14ms which is slightly slower than 1.7.1 (1 to 2ms) and massively faster than 2.2.0 (2500 to 2600ms). Great work @janl !

authors: garethbowen rnewson wohali

repo: apache/couchdb


text:
Add convenience remsh bash script

To make this work, I had to change the default -name from the old
couchdb@localhost to couchdb@127.0.0.1. This matches the advice
we already had in vm.args to use FQDN or IP address, anyway.
Once this merges I'll look at doing a Windows version, if possible.
Checklist

[X] Code is written and works correctly;
[ ] Changes are covered by tests;
[ ] Documentation reflects the changes;

Note: I don't know if we actually want to document this just yet. It's for advanced users for now; if they need it, they will have it.
If we add ferd/recon to our dependencies, as we've discussed on and off, then we might include some shortcuts for basic stats gathering to the Troubleshooting section of the documentation.


text:
fix: limit with offset should return correct results

Which issue does this PR close?

Closes #358.
Rationale for this change

What changes are included in this PR?

How are these changes tested?

cc @sunchao @andygrove

Merged. Thanks.

repo: apache/datafusion-comet