text:
test: Expand tests for built-in functions

Which issue does this PR close?
Closes #128
Rationale for this change
Improve test coverage for buit-in functions
What changes are included in this PR?

Expand tests for math functions
Expand tests for string functions
Expand tests for hash functions
Add tests for temporal functions

Are there any user-facing changes?
No

Thanks for running the GitHub Action, saw it was failing yesterday.

Rebased branch to latest master
Fix failing unit tests (change in data types)
Ran formatter & linter so that GitHub Action job should pass now

@simicd I don't have permissions to officially review but wanted to say changes look good. More test coverage is always a good thing

Thanks @jdye64! One question: When you want to update a branch after submitting a PR, do you rebase or just merge main into your branch? Asking because it seems like Actions require someone to trigger the workflow again, hope I didn't pick the wrong option by rebasing.

I always prefer using git rebase for Pull Request branches. This way the commit history is cleaner.
Rebasing also makes it easier  to use Squash and merge Github UI functionality.
The workflows should be triggered manually for first time contributors. Once your first PR is merged all following PRs will execute the checks automatically!

Thanks for the review @andygrove and for the explanation @martin-g!

authors: jdye64 martin-g simicd

repo: apache/arrow-datafusion-python


text:
Release a new version of DataFusion to crates.io

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The current version of datafusion on crates.io is 4.0.0 (which among other things doesn't work with the newly released arrow 5.0.0)
Describe the solution you'd like
It would be great if we released a DataFusion 5.0.0 (or some other number if we want to diverge from arrow) -- doing so would likely involve porting the arrow-rs release scripts, from https://github.com/apache/arrow-rs/tree/master/dev/release and then sending it to the dev mailing list for a formal vote
Starting this ticket to gather some feedback

@andygrove @jorgecarleitao I couldn't find the exact release for the 4.0.0 version we have in crates.io, Do you know where can we find it? It would be good to retroactively create the 4.0.0 tag for change log automation. I am guessing the commit is around ddaea81f9f46e918b5ab4e6257f1963b2a8a0f15?
@jorgecarleitao do we want to bundle the python release with the main datafusion release as well? Or do we want to release python binding in its own cadence?

I think 4.0.0 was released when datafusion was in the https://github.com/apache/arrow repo, specifically https://github.com/apache/arrow/tree/f959141ece4d660bce5f7fa545befc0116a7db79
I dug that version out of the email thread for the 4.0.0 RC3 release:
https://lists.apache.org/thread.html/rfb4e0065347235571cd0e0e71a7fa38c9652f07e27f98f89b4398feb%40<dev.arrow.apache.org>

We also need to consider what to do about Ballista release cadence and version numbers. My preference would be to release DataFusion and Ballista together. Ballista version is currently 0.5.0-SNAPSHOT and I think that releasing as 0.5.0 would be appropriate.

That would be awesome, I would +1 on aligning to 0.5.0, also for Python Datafusion (i.e. bump from 0.2.1 to 0.5.0); I think we do have a problem with DataFusion, that is already 5.0.0 on crates.io?

FWIW, the most recent release to crates.io for datafusion is 4.0.0: https://crates.io/crates/datafusion
Given that DataFusion doesn't have the associated arrow ecosystem I think using an entirely different versioning scheme is fine for DataFusion and Ballista - I don't really have a strong opinion on versions for DataFusion.

BTW @houqp  / @Jimexist  I am happy to help here / run whatever scripts / share my experience with doing the release process with arrow-rs -- just let me know what I can do to help

I also prefer we go with our own versioning scheme to better adhere to semantic versioning at the very least for ballista and datafusion python binding.
For datafusion the ship has already sailed unfortunately. If we are to release datafusion and ballista together, how do we manage the version differences? Do we just name the same tarball with different names? Or do we actually produce two different release tarballs with project specific sources?
Regardless what versioning scheme we use, the datafusion python documentation would still be needed. @Jimexist perhaps you could hep start building the automation for that?

Here is what I think could work for us, release all project sources in a single signed repo source tarball, which gets uploaded to apache svn. Then from within that same source release, we publish datafusion crate, ballista crate and datafusion python wheels with different versions. For the next release, we could have datafusion-5.0.0, ballista-0.5.0 and datafusion python 0.3.0 (or 0.5.0 if we want to align it with ballista). Name of the source tarball won't matter much here since I don't expect downstream consumers of these projects to use it directly. Instead they would use crates.io or pypi with the proper project name and version.
Consequence of this is every time we need to release a new version of either datafusion, python binding or ballista, we would need to vote and release a new version the datafusion repo as a whole.The repo source release should be pretty light weight given we don't need to do maintenance releases in current state of the project. All we have to do is to run create-tarball.sh , send to dev list for vote, then finally run release-tarball.sh.
WDYT?

on a side note, turns out the tree we used for 4.0.0 release (https://github.com/apache/arrow/commits/f959141ece4d660bce5f7fa545befc0116a7db79) is not in our repo.
Should I push this tree to our repo and tag it with 4.0.0? Alternatively, that particular commit maps to 31dd3cdfc9493352805e7bdbff69615e0a3290b2 in our master branch, so we could also just tag that commit as 4.0.0. But I think this might not align with Apache's release policy.

Regardless what versioning scheme we use, the datafusion python documentation would still be needed. @Jimexist perhaps you could help start building the automation for that?

@jorgecarleitao any previous work already done this area?
also there's how and where docs are supposed to be hosted, e.g.

jax uses read the docs, so is arrow which is likely the easiest
pandas has its own
requests has a different style but is also hosted by read the docs

Then from within that same source release, we publish datafusion crate, ballista crate and datafusion python wheels with different versions.

I think that would work nicely and I think it makes a lot of sense.

Should I push this tree to our repo and tag it with 4.0.0?

I would recommend not doing this (because it will effectively bring along the (large) history of the arrow repo with it, effectively making the arrow-datafusion repo several times larger).

Alternatively, the state of Rust code in that release maps to 31dd3cd in our master branch, so we could also just tag that commit as 4.0.0. But I think this might not align with Apache's release policy.

I think tagging 31dd3cd as 4.0.0 is fine (and I did something similar in arrow-rs) -- the official apache release policy, at least as I understand and discussed on the arrow-dev mailing lists, was centered around the tarballs as the artifacts. Since the 4.0.0 release / announcement was built from https://github.com/apache/arrow/commits/f959141ece4d660bce5f7fa545befc0116a7db79 if anyone wants to know exactly what was in the release they can use that reference. tagging 31dd3cd in this repo to compute the changelog seems like it would be fine

@Jimexist there were some previous discussion on how to host a website for datafusion on the dev list: https://lists.apache.org/thread.html/r0ed76cc60cdf651e8cf5c82a21cc64114c1f6d174dc5487434bd32ef%40<dev.arrow.apache.org>.
Read the docs is certainly the route with the least amount of work, but I am not 100% sure if it's something allowed for apache projects. The dev mailing list recommended hosting our docs as a sub path under the main arrow website. This incurs more work since this is not a turnkey solution. But it has the added benefit of us leveraging the arrow brand for marketing Datafusion and potentially improve SEO. We might be able to reuse the same automation that's used to generate https://arrow.apache.org/docs/python/api.html for datafusion too.

Quick update on my end, I have pushed the 4.0.0 tag with a reference to 31dd3cd to help with change log generation. Next up, I will look into subproject changelog automation using PR labels while waiting for more feedbacks on the release process proposed in https://github.com/apache/arrow-datafusion/issues/771#issuecomment-888042993.

Ballista is now at a point where it supports the TPC-H queries with
reasonable performance and I am now working on improving the developer
documentation and the user guide. I am keen on getting an official release
soon so that we can reserve the crate names. Let me know how I can help
with the release work. I generally only have time at weekends though.
On Wed, Jul 28, 2021 at 10:43 AM QP Hou @.***> wrote:

Quick update on my end, I have pushed the 4.0.0 tag with a reference to
31dd3cd
https://github.com/apache/arrow-datafusion/commit/31dd3cdfc9493352805e7bdbff69615e0a3290b2
to help with change log generation. Next up, I will look into subproject
changelog automation using PR labels while waiting for more feedbacks on
the release process proposed in #771 (comment)
https://github.com/apache/arrow-datafusion/issues/771#issuecomment-888042993
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow-datafusion/issues/771#issuecomment-888497088,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHEBRAHVSSJQFZ3IGBLKMDT2A6TFANCNFSM5A4TC45A
.

I generally only have time at weekends though.

I am on a similar boat at the moment. I have went back and tagged most of the issues with proper labels for changelog automation. I should be able to get the automation completed tomorrow and send a PR for you all to review both the code and changelog.
@andygrove I think where you will be able to help the most would be writing the announcement blog post for the release ;) But I don't think this should be a blocker for the crates.io release. In fact, if we are concerned about reserving the crate names, we could grab the name now with a placeholder release similar to https://crates.io/crates/roapi/0.1.0.

I agree that writing the blog post concurrently / after the release is totally fine.
I am also happy to help draft the release blog

I went ahead and reserved the names "ballista-core", "ballista-scheduler",
"ballista-executor" on crates.io. I already had "ballista" from prior to
the donation.
I'll continue working on documentation for Ballista and we can re-use
and/or link to some of that in the blog post.
On Sat, Jul 31, 2021 at 5:23 AM Andrew Lamb @.***>
wrote:

I agree that writing the blog post concurrently / after the release is
totally fine.
I am also happy to help draft the release blog
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow-datafusion/issues/771#issuecomment-890340611,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHEBRBVOOO2LPX7DYQBDH3T2PTNDANCNFSM5A4TC45A
.

I have prepared a preview in https://github.com/apache/arrow-datafusion/pull/801. After the PR gets merged, the next step is to create a signed tarball for voting.
Here are the generated changelog for each subproject:

https://github.com/houqp/arrow-datafusion/blob/qp_release/datafusion/CHANGELOG.md
https://github.com/houqp/arrow-datafusion/blob/qp_release/python/CHANGELOG.md
https://github.com/houqp/arrow-datafusion/blob/qp_release/ballista/CHANGELOG.md

My proposed release process requires adding two new sets of tags: ballista-x.y.z and python-x.y.z. The x.y.z tag will be used to tag release versions for datafusion itself and is required for every voted datafusion release. ballista-x.y.z and python-x.y.z tags are optional and only needed if we want to release a new version of ballista or python binding together with the same datafusion release. In other words, releasing a new version of python binding or ballista requires a new release of datafusion as well, but not the other way around.
I am going to send the proposed release setup to the dev list to gather more feedbacks today.

I think I have all the apache release related automations completed for https://github.com/apache/arrow-datafusion/pull/801. @Jimexist let me know if you need any help with python documentation automation. I am aiming push a final update to #801 this weekend and mark it as ready for review/merge if everything goes well this week. We can also release the python doc as part of a fast follow release after the 5.0.0 release if we don't want to rush it. My current goal is to make the datafusion release as light weight as we could so we can release more often.

👍  -- @houqp  let me know if there is anything I can do to help

I will update #801 and mark it as ready for review tomorrow. @andygrove I am assuming we want to wait for https://github.com/apache/arrow-datafusion/pull/831 for the release?

Yes, I will have the documentation PR ready by for review in the next few
hours. It would be good to get this in before we release.
On Sat, Aug 7, 2021, 1:37 AM QP Hou @.***> wrote:

I will update #801 https://github.com/apache/arrow-datafusion/pull/801
and mark it as ready for review tomorrow. @andygrove
https://github.com/andygrove I am assuming we want to wait for #831
https://github.com/apache/arrow-datafusion/pull/831 for the release?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow-datafusion/issues/771#issuecomment-894619586,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHEBREOQGJJPVN5KDS62RLT3TPDXANCNFSM5A4TC45A
.

https://github.com/apache/arrow-datafusion/pull/831 is ready for review and I would also like to merge https://github.com/apache/arrow-datafusion/pull/833 before we release.

@alamb @andygrove @Dandandan @jorgecarleitao @nevi-me @Jimexist #801 is now ready for review. Once it's merged, we will be able to push tags, release tarball and send a rc voting email to the dev list.

Filed https://github.com/apache/arrow-datafusion/issues/837 to track the python doc release as a quick follow up so we don't have to block the current release.

OK, release has been approved on the dev list. I have pushed the final release tags on Github. However, I don't have write access to arrow's release directory in SVN, crates.io nor PyPI. So I will need anyone who has access to these resources help me finish up the release. The steps are documented at https://github.com/houqp/arrow-datafusion/blob/qp_release_doc/dev/release/README.md#finalize-the-release.
The remaining steps are:

run ./dev/release/release-tarball.sh 5.0.0 3 to upload to release SVN dir
from the released tarball, publish datafusion and ballista to crates.io: https://github.com/houqp/arrow-datafusion/blob/qp_release_doc/dev/release/README.md#publish-on-cratesio
publish python binding to PyPI

I will run the script to upload to SVN...

(@houqp  the credentials I use for svn is my apache username/password). I am not sure what, if any, additional permissions are needed to upload to the release svn directory

The release files can be found here: https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-5.0.0/
I do not have permissions on the datafusion or ballista projects on crates.io. I believe @andygrove will have to grant one or both of us those permissions
Thanks again for all this work @houqp  🚀

I have invited @alamb to become an owner on all of the crates (I think this is currently a PMC role) and I have published the crates. @houqp Could you call the vote on the mailing list?
https://crates.io/crates/datafusion/
https://crates.io/crates/ballista/
https://crates.io/crates/ballista-core/
https://crates.io/crates/ballista-executor/
https://crates.io/crates/ballista-scheduler/
This whole process went very smoothly. Thank you @houqp and @alamb.

@alamb I am double checking with asf infra on the permission, but it looks like only PMC member has write access to the release folder. There is one thing that I noticed is not right, the release tarball is signed with my signing key, I have added it to the end of https://dist.apache.org/repos/dist/dev/arrow/KEYS, but I also don't have permission to add it to release/arrow/KEYS. I believe my key should be added to the release KEYS file too so people can use it to verify the signature.

@houqp Could you call the vote on the mailing list?

@andygrove do we still need to call a vote for the crates.io release? I thought the vote on the source tarball should have alredy covered the creates.io source release?

No need to vote on the crates publishing. The vote was just for the source
release.
On Sat, Aug 14, 2021, 11:20 AM QP Hou @.***> wrote:

@alamb https://github.com/alamb I am double checking with asf infra on
the permission, but it looks like only PMC member has write access to the
release folder. There is one thing that I noticed is not right, the release
tarball is signed with my signing key, I have added it to the end of
https://dist.apache.org/repos/dist/dev/arrow/KEYS, but I also don't have
permission to add it to release/arrow/KEYS. I believe my key should be
added to the release KEYS file too so people can use it to verify the
signature.
@houqp https://github.com/houqp Could you call the vote on the mailing
list?
@andygrove https://github.com/andygrove do we still need to call a vote
for the crates.io release? I thought the vote on the source tarball
should have alredy covered the creates.io source release?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/arrow-datafusion/issues/771#issuecomment-898932978,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHEBRDEQSKIOY3SYY5WNOTT42QVLANCNFSM5A4TC45A
.

OK, I have called the vote on the dev list, will add this step to the release document as well.

Filed https://github.com/apache/arrow-datafusion/issues/887 to track Python PyPI release as a follow up.
I believe the only remaining item is to copy my code signing key from dev/arrow/KEYS to release/arrow/KEYS.

I believe the only remaining item is to copy my code signing key from dev/arrow/KEYS to release/arrow/KEYS.

@houqp  I have added your key here: https://dist.apache.org/repos/dist/release/arrow/KEYS

Thank you @alamb we are all good to close this issue then :)

source: gharchive/issue

authors: Jimexist alamb andygrove houqp jorgecarleitao

repo: apache/arrow-datafusion

license_type: permissive

license_source: github-api


text:
add csv registration and sql query

Which issue does this PR close?
Closes #2824

Codecov Report

Merging #2825 (240eef7) into master (88b88d4) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2825   +/-   ##
=======================================
  Coverage   85.26%   85.26%
=======================================
  Files         275      275
  Lines       48830    48830
=======================================
+ Hits        41633    41634    +1
+ Misses       7197     7196    -1

Impacted Files
Coverage Δ

datafusion/core/src/physical_plan/metrics/value.rs
86.93% <0.00%> (-0.51%)
:arrow_down:

datafusion/expr/src/logical_plan/plan.rs
74.50% <0.00%> (+0.39%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 88b88d4...240eef7. Read the comment docs.

source: gharchive/pull-request

authors: codecov-commenter waitingkuo


text:
Flight SQL Server sends incorrect response for DoPutUpdateResult

Describe the bug
The Flight SQL Server incorrectly does a as_any() and sends a protobuf enum instead of the message itself. This is incompatible with the JDBC implementation. (As usual, the spec itself is unclear). It results in the record_count being 0 instead of -1 which means a proper client implementation will poll forever.
To Reproduce
This one is a doozy to reproduce:

Run this fork of ballista
Build this fork of the JDBC driver
Install the fat jar driver into your favorite SQL tool
Run an update query, observe the tool hang as it follows the spec attempting to wait for the condition to be met:

There are no more results when the following is true:

     // stmt is a Statement object
     ((stmt.getMoreResults(current) == false) && (stmt.getUpdateCount() == -1))

Expected behavior
The query succeeds.

Tagging @tustvold and @andygrove , thank you both for your help!

repo: apache/arrow-rs


text:
Interval parsing interprets 5 day hour wrongly

Describe the bug
While working on https://github.com/apache/datafusion/pull/12448 I noticed that the string 5 day hour is not interpreted correctly by parse_interval_month_day_nano_config, it should be considered the same as 5 day 0 hour, instead it return an error.
To Reproduce
run that function, or look at https://github.com/apache/datafusion/pull/12448
Expected behavior
parse_interval_month_day_nano_config should match postgres behaviour in general
Additional context
Related to #6211 where I improved interval parsing a lot, although apparently not enough.

I think 5 day hour is a pretty uncommon usecase . Thank you for filing this

The most likely scenario where it could come up is select interval '5 day' hour, but I agree even that seems like either allowing a common mistake, or a side effect of their very permissive parser.

While we're talking about weird things postgres supports, as per https://github.com/sqlparser-rs/sqlparser-rs/issues/1345#issuecomment-2334514705, the following  are also supported:
select interval '1 month + 2 day'
select interval '1 minute + 2 s'  -- makes sense, all units are supported in arithmetic expressions
select interval '1 minute - 2 s'
select interval '1 minute + 2'  -- makes sense since raw numbers are interpreted as seconds
select interval '1 minute * 2s'  -- this is crazy! the "*" operator is interpretted as "+" so this evaluates as 0:01:02
select interval '1 minute minute' -- repeated units are ignored
select interval '1 month + 2 day' day -- same thing with the units just concatenated

@alamb do we have a plan on integrating this inside Arrow?

@alamb do we have a plan on integrating this inside Arrow?

I am not sure what you mean by this question @ByteBaker

I meant to ask if we plan to handle more cases in our parsing logic? To be closer to parsing abilities of Postgres? As @samuelcolvin pointed out above.

I meant to ask if we plan to handle more cases in our parsing logic? To be closer to parsing abilities of Postgres? As @samuelcolvin pointed out above.

I guess in general I think it would be driven by contributor needs -- if someone wants needs more exotic parsing support, we can definitely consider it, but if it gets too specific, it probably makes sense to keep the parsing in the application

In that case we need to decide if we wanna keep this one open or not.

Let's leave it open so it is easier to find if someone else hits the problem

authors: ByteBaker alamb samuelcolvin


text:
Archery Integration Test with c# failing on main

Describe the bug
archery integration test is failing on main : https://github.com/apache/arrow-rs/actions/runs/11363774836/job/31608436255
 ################# FAILURES #################
  FAILED TEST: primitive Rust producing,  C# consuming
  <class 'RuntimeError'>: Command failed: /__w/arrow-rs/arrow-rs/csharp/artifacts/Apache.Arrow.Flight.IntegrationTest/Debug/net8.0/Apache.Arrow.Flight.IntegrationTest client --port 44909 --path /tmp/arrow-integration-hk4dbqi4/generated_primitive.json
  With output:
  --------------
  Unhandled exception: Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')")
   ---> System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
     at System.Net.Dns.GetHostEntryOrAddressesCoreAsync(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, Boolean justAddresses, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Dns.GetHostAddressesAsync(String hostNameOrAddress, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Sockets.SocketAsyncEventArgs.DnsConnectAsync(DnsEndPoint endPoint, SocketType socketType, ProtocolType protocolType)
     at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
     at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.OnConnect(Socket socket, DnsEndPoint endpoint, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context, Int32 attempt)
     --- End of inner exception stack trace ---
     at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadSchemaAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 74
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadNextRecordBatchAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 118
     at Apache.Arrow.Flight.FlightRecordBatchStreamReader.MoveNext(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamReader.cs:line 64
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.ConsumeFlightLocation(FlightClient client, FlightTicket ticket, RecordBatch[] batches) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 160
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.RunClient(Int32 serverPort) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 109
     at Apache.Arrow.Flight.IntegrationTest.FlightClientCommand.Execute() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs:line 45
     at Apache.Arrow.Flight.IntegrationTest.Program.<>c.<<Main>b__0_0>d.MoveNext() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs:line 50

To Reproduce

Expected behavior
Should pass
Additional context
Not sure if this is related to https://github.com/apache/arrow-rs/issues/6568 or not

I think #6585 might fix this as well

I think #6585 might fix this as well

Thanks @itsjunetime  -- unfortunately it seems like the tests are still failing even with #6585 . I'll see what I can do

I poked around in the arrow repo. There are two PRs that went in around the same time this test started failing that could be related

https://github.com/apache/arrow/pull/44424
https://github.com/apache/arrow/pull/44377

https://github.com/apache/arrow/pull/44377 specifically looks related as this test appears to be new. I will ask some questions there and make a PR to disable this test in arrow-rs while we investigate more

Here is another more specific message from the logs https://github.com/apache/arrow-rs/actions/runs/11409436304/job/31749661931
 ======================================================================
  Testing file null
  Traceback (most recent call last):
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/util.py", line 136, in run_cmd
      output = subprocess.check_output(cmd, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/envs/arrow/lib/python3.11/subprocess.py", line 466, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/envs/arrow/lib/python3.11/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/__w/arrow-rs/arrow-rs/csharp/artifacts/Apache.Arrow.Flight.IntegrationTest/Debug/net8.0/Apache.Arrow.Flight.IntegrationTest', 'client', '--port', '46599', '--path', '/tmp/arrow-integration-2gthhuy4/generated_null.json']' returned non-zero exit status 1.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/runner.py", line 435, in _run_flight_test_case
      consumer.flight_request(port, **client_args)
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/tester_csharp.py", line 218, in flight_request
      run_cmd(cmd)
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/util.py", line 145, in run_cmd
      raise RuntimeError(sio.getvalue())
  RuntimeError: Command failed: /__w/arrow-rs/arrow-rs/csharp/artifacts/Apache.Arrow.Flight.IntegrationTest/Debug/net8.0/Apache.Arrow.Flight.IntegrationTest client --port 46599 --path /tmp/arrow-integration-2gthhuy4/generated_null.json
  With output:
  --------------
  Unhandled exception: Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')")
   ---> System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
     at System.Net.Dns.GetHostEntryOrAddressesCoreAsync(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, Boolean justAddresses, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Dns.GetHostAddressesAsync(String hostNameOrAddress, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Sockets.SocketAsyncEventArgs.DnsConnectAsync(DnsEndPoint endPoint, SocketType socketType, ProtocolType protocolType)
     at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
     at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.OnConnect(Socket socket, DnsEndPoint endpoint, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context, Int32 attempt)
     --- End of inner exception stack trace ---
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadSchemaAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 74
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadNextRecordBatchAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 118
     at Apache.Arrow.Flight.FlightRecordBatchStreamReader.MoveNext(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamReader.cs:line 64
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.ConsumeFlightLocation(FlightClient client, FlightTicket ticket, RecordBatch[] batches) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 160
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.RunClient(Int32 serverPort) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 109
     at Apache.Arrow.Flight.IntegrationTest.FlightClientCommand.Execute() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs:line 45
     at Apache.Arrow.Flight.IntegrationTest.Program.<>c.<<Main>b__0_0>d.MoveNext() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs:line 50
  --- End of stack trace from previous location ---
     at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
     at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

  --------------

Here is the job definition: https://github.com/apache/arrow-rs/blob/1666a4d7b237370c167e2b90b45d1354e28f500a/.github/workflows/integration.yml#L50-L114

Posted upstream on https://github.com/apache/arrow/pull/44377#issuecomment-2424869128 to see if we can get any hints on how to fix this

Here is a fix: https://github.com/apache/arrow-rs/pull/6611 but it is not clear if it is the right one

label_issue.py automatically added labels {'documentation'} from #6554

label_issue.py automatically added labels {'arrow'} from #6554

authors: alamb itsjunetime


text:
GH-36045:[Python] Improve usability of pc.map_lookup / MapLookupOptions

Rationale for this change
This PR is for #36045, which aims to improve usability of pc.map_lookup / MapLookupOptions
What changes are included in this PR?
For query_key which is not a subclass of pyarrow.lib.Scalar, we convert it to scalar
Are these changes tested?
Yes, add one test in tests/test_compute.py::test_map_lookup

Closes: #36045

Thanks for suggestions. Have merged.

repo: apache/arrow


text:
GH-41310: [C++] S3FS Read file using the version id obtained with HEAD call

Rationale for this change
GET calls to s3 is not made using the version id (if available) from the HEAD call (which is used to get the content-length). If the version is available, using that makes the GET call more reliably return the file data without running in to race condition, if a write was intercepted between the HEAD and GET.
What changes are included in this PR?
s3fs.cc
The changes include capturing the version if available from the HEAD call and using it subsequently in the GET requests.
This doesnt need any specific tests. The existing tests should cover for regression.

GitHub Issue: #41310

Would it be possible to write a test that reproduces the issue? Possibly by writing a first version, then spawn a thread that writes a second version while reading the first version from the main thread.

@gitfy Would you like to answer the question above? For the record, I think this is a good improvement.

authors: gitfy pitrou


text:
ARROW-7277: [Java] [Doc] Add discussion about vector lifecycle

As discussed in https://issues.apache.org/jira/browse/ARROW-7254?focusedCommentId=16983284&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16983284, we need a discussion about the lifecycle of a vector.
Each vector has a lifecycle, and different operations should be performed in particular phases of the lifecycle. If we violate this, some unexpected results may be produced. This may cause some confusion for Arrow users. So we want to add a new section to the prose document, to make it clear and explicit.

@liyafan82 sorry, I've been backlogged will try to review your open PRs over the next week or so (unless others get to them sooner CC @praveenbingo @pravindra @BryanCutler)

@liyafan82 sorry, I've been backlogged will try to review your open PRs over the next week or so (unless others get to them sooner CC @praveenbingo @pravindra @BryanCutler)

@emkornfield Please do not be sorry. You have already given me so much help and guidance. I really appreciate all your effort and professional spirit. So please don't worry, just take your time.
I think it is late in your location. I also notice that you often work late into night reviewing our PR. So I also hope you can take good care of your health and have some good rest :-)

Going to merge this.  Thanks.


text:
Added a test to try and reproduce arrow-11067.  Since it can only fai…

…l on mac I'm going to commit just the test first to see if it fails properly and I can avoid setting up a mac devenv.

I restarted the MacOS jobs because they did fail, but not for the right reason :-(

I restarted the MacOS jobs because they did fail, but not for the right reason :-(

@westonpace is Github Actions enabled on your fork?

@westonpace is Github Actions enabled on your fork?

@pitrou I'm pretty sure it is.  Is there something I need to check?

@pitrou I'm pretty sure it is.  Is there something I need to check?

@westonpace MinGW Windows builds I think.

@westonpace MinGW Windows builds I think.

authors: pitrou westonpace


text:
Remove useless pass statement

This patch will remove a useless pass statement in atlas_config.py file.

@nixonrodrigues PTAL

repo: apache/atlas


text:
AVRO-4055: Validate records properly when parsing a schema

See https://issues.apache.org/jira/browse/AVRO-4055

Updated based on the feedback!

Thank you, @woile !

authors: martin-g woile

repo: apache/avro-rs


text:
[Bug]: DirectRunner fails after grouping into batches, and applying flat map with assertion error

What happened?
I am running the following code:
import argparse

import apache_beam as beam
import structlog
from apache_beam.options.pipeline_options import PipelineOptions

logger = structlog.getLogger()

def run(argv=None, save_main_session=True):
    parser = argparse.ArgumentParser()

    # Parse the arguments from the command line as defined in the options class
    known_args, pipeline_args = parser.parse_known_args(argv)

    pipeline_options = PipelineOptions(pipeline_args)

    with (beam.Pipeline(options=pipeline_options) as p):
        (
            p
            | "Read Vertex embedder output" >> beam.Create(["a", "b", "c"])
            | "Assign dummy keys" >> beam.Map(lambda x: (None, x))
            | "Batch up to 30 elements" >> beam.GroupIntoBatches(30)
            | "Flat map" >> beam.FlatMap(lambda x: x)
            | "Write to GCS"
            >> beam.io.WriteToText(
                "gs://my_bucket/apache_beam_test/output",
                file_name_suffix=".jsonl",
            )
        )

if __name__ == "__main__":
    run()

As you can see this code generates 3 dummy values. Let's say I want to take my elements in batches of 30, so I assign the same dummy key to all of them. After working with my batches, I want to go back to single elements, and then publish the results to Google Cloud Storage.
Running the code above fails with the following error:
ERROR:apache_beam.runners.direct.executor:Giving up after 4 attempts.
WARNING:apache_beam.runners.direct.executor:A task failed with exception:
Traceback (most recent call last):
  File "/Users/robert-andreidamian/Workspace/panprices/product_image_hasher/after_embeddings.py", line 34, in <module>
    run()
  File "/Users/robert-andreidamian/Workspace/panprices/product_image_hasher/after_embeddings.py", line 18, in run
    with (beam.Pipeline(options=pipeline_options) as p):
  File "/Users/robert-andreidamian/.local/share/virtualenvs/product_image_hasher-uOTOvNp0/lib/python3.11/site-packages/apache_beam/pipeline.py", line 601, in __exit__
    self.result.wait_until_finish()
  File "/Users/robert-andreidamian/.local/share/virtualenvs/product_image_hasher-uOTOvNp0/lib/python3.11/site-packages/apache_beam/runners/direct/direct_runner.py", line 585, in wait_until_finish
    self._executor.await_completion()
  File "/Users/robert-andreidamian/.local/share/virtualenvs/product_image_hasher-uOTOvNp0/lib/python3.11/site-packages/apache_beam/runners/direct/executor.py", line 432, in await_completion
    self._executor.await_completion()
  File "/Users/robert-andreidamian/.local/share/virtualenvs/product_image_hasher-uOTOvNp0/lib/python3.11/site-packages/apache_beam/runners/direct/executor.py", line 480, in await_completion
    raise update.exception
  File "/Users/robert-andreidamian/.local/share/virtualenvs/product_image_hasher-uOTOvNp0/lib/python3.11/site-packages/apache_beam/runners/direct/executor.py", line 370, in call
    self.attempt_call(
  File "/Users/robert-andreidamian/.local/share/virtualenvs/product_image_hasher-uOTOvNp0/lib/python3.11/site-packages/apache_beam/runners/direct/executor.py", line 413, in attempt_call
    evaluator.process_element(value)
  File "/Users/robert-andreidamian/.local/share/virtualenvs/product_image_hasher-uOTOvNp0/lib/python3.11/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 939, in process_element
    assert not self.global_state.get_state(
AssertionError

I am running the code in a pipenv environent, with python version 3.11. I attached the requirements.
requirements.txt
Here is my Pipfile:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
google-cloud-bigquery-storage = "*"
google-cloud-bigquery = {extras = ["bqstorage", "pandas"], version = "*"}
google-cloud-tasks = "*"
firebase-admin = "*"
Pillow = "*"
imagehash = "*"
structlog = "*"
functions-framework = "*"
opencv-python = "*"
numpy = "*"
asyncio = "*"
pydantic = {extras = ["dotenv"], version="~=1.10"}
psycopg2-binary = "*"
google-cloud-aiplatform = "*"
gcloud-aio-storage = "*"
gcloud-rest-storage = "*"
validators = "*"
avif = {extras = ["pillow"], version="*"}
pillow-avif-plugin = "*"
wheel = "*"
apache-beam = {extras = ["gcp"], version = "*"}
cloud-sql-python-connector = {extras = ["pg8000"], version = "*"}

[dev-packages]
flask = {extras = ["async"], version="*"}

[requires]
python_version = "3.11"

Issue Priority
Priority: 3 (minor)
Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

Okay so I can repro the issue, specifically when the batch size specified is > the number of elements in the input PCollection and the another GBK happens in the pipeline after the GroupIntoBatches call. (GBK is marked as complete here and then re-invoked here.)
This would happen if _is_final_bundle() returns true, AKA the watermark was advanced to infinity prematurely for the next invocation of a GBK. This seems to be the case if batch size > num elements, with the watermark still being at positive infinity on the next invocation instead of being reset to negative infinity for the next DoFn.
I can demonstrate this with this pipeline + some extra logging within the GBK transform evaluator:
def run(argv=None, save_main_session=True):
    parser = argparse.ArgumentParser()

    # Parse the arguments from the command line as defined in the options class
    known_args, pipeline_args = parser.parse_known_args(argv)

    pipeline_options = PipelineOptions(pipeline_args)

    with (beam.Pipeline(options=pipeline_options) as p):
        (
            p
            | "Read Vertex embedder output" >> beam.Create(["a", "b", "c"])
            | "Assign dummy keys" >> beam.Map(lambda x: ("key", x))
            | "Batch up to 30 elements" >> beam.GroupIntoBatches(3)
            | "Flat map" >> beam.FlatMap(lambda x: x[1])
            | "re-key" >> beam.Map(lambda x: ("key2", x))
            | "gbk" >> beam.GroupByKey()
        )

if __name__ == "__main__":
    run()

With batch size = 3 we get each element through both GBKs, but with batch size > 3 the second GBK invocation fails on the first element because the Completion Tag and watermark are not reset. I haven't quite found where the reset between DoFn invocations happens yet, but this is where the problem is.

Somehow this pipeline is using a wrong direct runner (apache_beam/runners/direct/direct_runner.py).
the old bundlebased direct runner is no longer maintained, and it is not expected that it is still being used for this batch pipeline.
The relevant code is in https://github.com/apache/beam/blob/223dded769df48270df317868dc32144ec2fb353/sdks/python/apache_beam/runners/direct/direct_runner.py#L115-L126

@tvalentyn
Thanks for your reply. Based on what you said I added some breakpoints and I discovered that this is where the decision to not use FnApi runner is taken:
https://github.com/apache/beam/blob/05861613f484a6159c21796006c52d8da5e10b2a/sdks/python/apache_beam/runners/direct/direct_runner.py#L111C18-L111C18
This happens while it checks _GroupIntoBatchesDoFn.
Here is a screenshot of the timer data I took in my IDE:

I could investigate further and come up with a PR maybe, but I will need a few pointers:

Why is the check for "REAL_TIME" here in the first place?
What does it mean for a ParDo to be "REAL_TIME"?

Thanks!
The implementation of GroupIntoBatches includes a processing-time timer: https://github.com/apache/beam/blob/3a45ecf4b271997b5ce03e1181676356eaa351e1/sdks/python/apache_beam/transforms/util.py#L1067 , you can find more information about timers in https://beam.apache.org/documentation/programming-guide/#processing-time-timers.
This logic was added in https://github.com/apache/beam/pull/13144
@robertwb do you know if this restriction still applies:  https://github.com/apache/beam/blob/223dded769df48270df317868dc32144ec2fb353/sdks/python/apache_beam/runners/direct/direct_runner.py#L110-L111 ?
We could also try to lift it, run the tests and see what fails.

I was trying to set up my environment using the following guide: https://cwiki.apache.org/confluence/display/BEAM/Python+Tips
I noticed that the build-requirements.txt file is missing. I am guessing that one can get the dependencies using the gradle build file.
Should I open another issue that is about updating the documentation?

@damianr13 I am working in the updating the wiki page. Thanks.
You don't need to use build-requirements.txt anymore if you rebase/merge onto master branch.

GroupIntoBatches seems to still cause this issue with beam 2.58.0 when using DirectRunner and even TestPipeline.

authors: AnandInguva damianr13 jrmccluskey keunhong tvalentyn

repo: apache/beam


text:
[Bug]: GBK with global window must fit all values for a key into a memory

What happened?
With optimization global window grouping in Spark
https://issues.apache.org/jira/browse/BEAM-12646
https://github.com/apache/beam/commit/8c3af01cdf404891e82f2f4c9b6d8e369ebdad4f#diff-c13404655c9bf261fcbcc72feb949e0ffcf428802897d2f39097f34d7a3d995aL185
it reintroduced issue for jobs with global windows. Now it is using spark's groupByKey = all values for a single key need to fit in-memory at once. It was already solved in:
https://issues.apache.org/jira/browse/BEAM-5392
Issue Priority
Priority: 3 (minor)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[X] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

Hello, has this issue been resolved? Was it addressed in PR #33322 by any chance? Thank you.

authors: mareksimunek twosom


text:
[BEAM-8847] Handle the registration request synchronously in the Java…

Currently the registration request is handled asynchronously in the Java SDK harness. As discussed in #10208, this JIRA. tries to change it to synchronous.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run Java PreCommit

I'm still interested in having this change go in, @sunjincheng121 any updates?

Thanks for the reminder @lukecwik :)
I would like to have another debug...
Run Java PreCommit

Run Java PreCommit

Run Java PreCommit

Going with a different approach where the SDK can request any bundle descriptors it doesn't have, see BEAM-9618


text:
[BEAM-8575] Added a unit test to test that Combine works with FixedWi…

…ndows.
[BEAM-8575] Added a unit test to test that Combine works with FixedWindows.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @chamikaramj

Run Python PreCommit

Don't review this PR. It's polluted. Please review https://github.com/apache/beam/pull/10383 instead.

Closing this, the other referenced PR (#10383) is merged already.

authors: aaltay bumblebee-coming


text:
[release-2.36.0][BEAM-13541] More intelligent caching of CoGBK values. (#16354)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

Java

Python
---

---

XLang

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @emilymye

You may also need to cherry pick https://github.com/apache/beam/pull/16407 as well.

authors: aaltay robertwb


text:
[BEAM-9324] Fix incompatibility of direct runner with cython

This is due to Cython.Shadow.cast(Receiver, value) attempting to run Receiver(value) (probably very old code in Python 2 age where "old" class did not have __call__)
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Run Python PreCommit

R: @TheNeuralBit
R: @robertwb

I've seen the errors from BEAM-9324 TypeError: Receiver() takes no argument before when I've got my build into an inconsistent state (some files Cythonized, some not). I wonder if there's a build system change we can make to help avoid this situation rather than branching throughout the code?

Ah ok, one last question from me: should we remove this when/if cython 3 is out and we upgrade it? If so let's file a jira and maybe add TODOs.

Ah ok, one last question from me: should we remove this when/if cython 3 is out and we upgrade it? If so let's file a jira and maybe add TODOs.

Added TODO. used this jira for reference since Cython3 may still be pending in nearly future and thus not filing a ticket at this moment.

Run Python PreCommit

Run PythonDocker PreCommit

authors: Abacn TheNeuralBit asf-ci


text:
Add beam_PreCommit_RAT job

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run RAT PreCommit


text:
[#29803] Fix nil pointer access in logRuntimeDependencies

Corrects a nil pointer usage when Python containers are built through processArtifactsInSetupOnlyMode()
Fixes #29803

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @damccorm @riteshghorse
CC: @tvalentyn

It is an odd pattern that at present:

setup passes down a nil logger,
then we manually re-build/wrap with buffered loggers wherever we actually do log something.

I'd expect us to pass around the buffered logger type from the top, instead of arbitrarily trying to re-construct it.
Stuff to clean up when it's not a release blocker.

authors: jrmccluskey lostluck


text:
[BEAM-1345] Mark windowed value as experimental

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify.
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @aaltay PTAL

LGTM


text:
[BEAM-6697] Upgrades gcsio to 1.9.16

gcsio 1.9.15 and prior has a bug which will result in a failure if a user invoke a metadata request before reading data. ParquetIO fails due to this.
Please see https://issues.apache.org/jira/browse/BEAM-6697 for more info.

Run Java PostCommit

Run Java ParquetIO Performance Test

Run Java PreCommit

Run Java PostCommit

Run Java ParquetIO Performance Test

Run Website PreCommit


text:
[BEAM-7574] fix Combine performance for SparkRunner

Fixes [BEAM-7574]

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @dmvk @iemejia @robertwb

Performance testing of this fix is still ongoing.

First run of test job results in significant performance boost compared to previous version (minutes opposed to hours). My test job is a little specific, though. It has many little time windows. @dmvk can you test this on some job you have?

SUCCESS
--none--

Run Spark ValidatesRunner

Run Spark ValidatesRunner

Run Nexmark on the current PR and saw no regressions so far and clear improvements in Combine heavy queries. So positive about this, even if not checked the code yet.
Command:
./gradlew :sdks:java:testing:nexmark:run \
    -Pnexmark.runner=":beam-runners-spark" \
    -Pnexmark.args="
        --runner=SparkRunner
        --streaming=false
        --suite=SMOKE
        --manageResources=false
        --monitorJobs=true
        --enforceEncodability=true
        --enforceImmutability=true"

Results:
Master
  Conf  Runtime(sec)    (Baseline)  Events(/sec)    (Baseline)       Results    (Baseline)
  0000           1.5                     68870.5                      100000
  0001           0.6                    172711.6                       92000
  0002           0.4                    284900.3                         351
  0003           1.7                     58685.4                         580
  0004           0.9                     10893.2                          40
  0005           2.1                     46992.5                          12
  0006           0.6                     17006.8                         103
  0007           2.7                     37243.9                           1
  0008           1.2                     85324.2                        6000
  0009           0.5                     20080.3                         298
  0010           0.6                    161550.9                           1
  0011           1.0                     98328.4                        1919
  0012           0.8                    118063.8                        1919
  0013           0.4                    267379.7                       92000
  0014           2.9                     69710.7                      184000
==========================================================================================

This PR
  Conf  Runtime(sec)    (Baseline)  Events(/sec)    (Baseline)       Results    (Baseline)
  0000           1.3                     74906.4                      100000
  0001           0.6                    166666.7                       92000
  0002           0.3                    392156.9                         351
  0003           1.6                     61050.1                         580
  0004           0.8                     12610.3                          40
  0005           2.1                     48709.2                          12
  0006           0.6                     17574.7                         103
  0007           2.4                     42247.6                           1
  0008           1.0                     98814.2                        6000
  0009           0.4                     22421.5                         298
  0010           0.6                    171821.3                           1
  0011           1.4                     70028.0                        1919
  0012           0.8                    130378.1                        1919
  0013           0.4                    253164.6                       92000
==========================================================================================

Note that the execution is breaking on query 14 but that seems unrelated to the PR.

There seems to be performance regression on tests 11 and 12, can you post a link to there tests?

Of course, not sure if these differences are significant, does the test suite run multiple times?

I run it multiple times and this is an example of an average run. Of course in this case it uses a really small synthetic dataset but Nexmark has helped us multiple times to catch regressions or confirm improvements.
There is always a skew between runs of +/- 3%, it used to be more stable between runs in the pre gradle and pre meltdown days. I really would like to find a way to execute these tests in a more reliable way so any hint on that would be greatly apprecaited. In any case the average of runs is positive.

@iemejia cool, that sounds promising, thanks!

Commit 4a7531b had roughly 25% performance gain on my test job, which uses massive counts of windows per key.

Run Spark ValidatesRunner

Run Java Spark PortableValidatesRunner Batch

R: @RyanSkraba Can you help me with this review. Thanks!

Run Spark ValidatesRunner

@je-ik can you please check if there is some issue with the implementation?. From the jenkins logs it looks like at some point the timestamps are becoming null.

@iemejia sure, thanks. I  didn't check these changes against streaming validates runner, because I thought that these changes were related to batch only, but I might had been wrong.
@RyanSkraba thanks for review and for some comments. I will walk through it as soon as possible (during this week).

Run Spark ValidatesRunner

Run Spark ValidatesRunner
Run JavaPortabilityApi PreCommit

Run Spark ValidatesRunner

Run Spark ValidatesRunner

Run JavaPortabilityApi PreCommit

@iemejia @RyanSkraba all tests green, there was bug in Combine.globally, which is fixed now.

Run Java PreCommit

Run JavaPortabilityApi PreCommit

:+1: For the extra @SuppressWarnings("unchecked") -- I wouldn't let that block this PR, since they aren't a checkstyle error.  Thanks for this opportunity to dive quite a bit deeper into the SparkRunner!

Run Spark ValidatesRunner

I also plan to remove the AbstractSparkCombineFn as you suggested, just didn't have time to do it yet.

Should we wait for this fix or maybe we can do it in a subsequent PR @je-ik @RyanSkraba?
I would like to get this issue in 2.15.0 if possible (branch cut is in 2 days).

I'd like to do it in this PR. Will push ASAP. This PR then needs rebase and squash commits.

Ok, Thanks a lot @je-ik Just ping me when done for the final check + merge.

Retest this please

Run Spark ValidatesRunner

Run Java Spark PortableValidatesRunner Batch

Run Java PreCommit

Thanks for the clean-up!  In my opinion, this looks good to merge for 2.15.0!

@iemejia Tests pass, LGTM

@dmvk you can be interested in testing this (probably via the SNAPSHOTS after tomorrow). Any info on regressions or improvements for your use cases will be definitely appreciated.

authors: RyanSkraba asfgit dmvk iemejia je-ik


text:
BIGTOP-4001. Bump HBase to 2.4.17.

https://issues.apache.org/jira/browse/BIGTOP-4001
While this is mainly for Bigtop 3.2.2, this could be useful for master branch too as intermediate step to bump HBase to 2.5.

patch1-relocate-okhttp.diff is no longer needed since it is contained in 2.4.17 as HBASE-27292.

I built HBase and Phoenix then ran smoke-tests on Ubuntu 22.04 x86_64:
$ ./docker-hadoop.sh \
    --create 3 \
    --image bigtop/puppet:trunk-ubuntu-22.04 \
    --docker-compose-yml docker-compose-cgroupv2.yml \
    --docker-compose-plugin \
    --memory 8g \
    --repo file:///bigtop-home/output/apt \
    --disable-gpg-check \
    --stack hdfs,yarn,mapreduce,hbase,phoenix \
    --smoke-tests hbase,phoenix

Verified the PR on Fedora-36 Arm64.
+1
./docker-hadoop.sh --create 3 --image bigtop/puppet:3.2.1-fedora-36 --memory 16g \
    --repo file:///bigtop-home/output --disable-gpg-check --enable-local-repo \
    --stack hdfs,yarn,mapreduce,hbase,phoenix \
    --smoke-tests hbase,phoenix

Thanks, @guyuqi. I cherry-picked this to branch-3.2.

authors: guyuqi iwasakims

repo: apache/bigtop

license_source: bigquery


text:
Fix creation script

An issue was created by https://github.com/apache/brooklyn-server/pull/812.
That PR removed the password needed to create the user brooklyn'@'localhost

I have tested this on a MySQLNode and it works

@Graeme-Miller @drigodwin odd - local seemed to work for me but sometimes mysql is weird which NIC/socket it uses for localhost, and different versions are different.  good fix.
two things--

is the same update made to brooklyn-library ?

is there a reason we're not consistent what perms we give to the two users?  (this PR gives usage to the users @ % but all to @ localhost -- previously was all on visitors and usage elsewhere, to both users) ... in other words should the file read

grant usage on *.* to 'brooklyn'@'%' identified by '${config["creation.script.password"]!"br00k11n"}';
grant all privileges on visitors.* to 'brooklyn'@'%';
# useful if sockets work also
grant all privileges on *.* to 'brooklyn'@'localhost' identified by '${config["creation.script.password"]!"br00k11n"}';
grant all privileges on visitors.* to 'brooklyn'@'localhost';

authors: Graeme-Miller ahgittin

repo: apache/brooklyn-server


text:
支持bRPC网关转发baidu协议请求的功能

Is your feature request related to a problem? (你需要的功能是否与某个问题有关?)
目前brpc框架不支持bRPC网关服务高效地转发baidu协议的请求——只反序列化meta部分，直接转发payload，无需反序列化payload。
Describe the solution you'd like (描述你期望的解决方法)

baidu协议实现类似http_master_service的功能，接管所有baidu协议流量。同时，http_master_service可用于多语言接入bRPC服务场景。
baidu协议meta中支持用户自定义字段（kv），用于传递upstream信息、路由信息等。
支持baidu协议泛化调用，payload支持pb序列化数据、json字符串。网关服务可以不受upstream pb的限制。（#1385 中有相关咨询）

Describe alternatives you've considered (描述你想到的折衷方案)
Additional context/screenshots (更多上下文/截图)

#2406 实现baidu协议meta中传递用户自定义字段的特性

related issue: #1444

功能3的意思是用户可以不用pb协议，使用其它类型协议也可以是吗？c++客户端到服务端直接这样可以吗？中间不用加一层代理。

用户可以不用pb协议，使用其它类型协议也可以是吗？

要求能跟pb互转，例如json。

c++客户端到服务端直接这样可以吗？

可以的。HTTP body类似。

为啥需要和pb互转呢？我理解这个过程bRPC完全透传了用户的消息，不用做任何处理，就类似attachment。
bRPC只处理baidu_std这层协议使用的是pb。

嗯嗯，网关只转发baidu协议请求的话，是不需要json、pb互转的。
扩展一下，网关要支持多语言（php、nodejs、python等）接入baidu协议的brpc服务的话，client的payload一般使用json，网关负责服务发现和转发payload，brpc服务需要支持json、pb互转的能力。

稍微修改了一下原来的描述。

authors: chenBright wwbmmm yanglimingcn

repo: apache/brpc


text:
Remote Execution should maximize parallel builders

See original issue on GitLab
In GitLab by [Gitlab user @sstriker] on Sep 7, 2018, 12:00
When doing remote execution, the number of parallel build jobs should be maximized, allowing the remote Execution service to take care of scheduling and resource management of remote workers.
Reference: Context.sched_builders, Resources.

In GitLab by [Gitlab user @tristanvb] on Jun 1, 2020, 08:43
marked this issue as related to #1333

repo: apache/buildstream


text:
Retrying failed builds is important, but difficult for new users

See original issue on GitLab
In GitLab by [Gitlab user @aevri] on Mar 20, 2019, 16:20
Summary
In theory, users should have fully reproducible builds, and it is not necessary to retry builds.
In practice there are numerous issues that lead to a need for retrying:

Race conditions in the build.
Limited memory.
Limited disk space.
etc.

Here are the difficulties with retrying as I see them, will create separate tickets depending on how this discussion goes:

Retrying a build with 'r' does not try to rebuild, it uses the cached failed build. (It retries the 'build job' not the build).
A subsequent 'bst build' fails immediately, perhaps it could offer 'retry' and 'logs' options as with first run.
'bst build' failure doesn't signpost commands that can help from there, e.g. 'artifact logs' and 'artifact delete'.

How to repro:
git clone https://gitlab.com/buildstream/buildstream.git
cd buildstream/doc/examples/running-commands
bst build hello.bst
# -- Should be ok up to here --

echo "  - sleep 10" >> elements/hello.bst
echo "  - 'false'" >> elements/hello.bst

bst build hello.bst
# .. wait 10 seconds for it to fail ..

# Choose:
#  (r)etry     - Retry this job
# .. instantly fail again, no 10 second wait ..

bst build hello.bst
# .. instantly fail again, with only the failed command mentioned, logs would be nice ..

bst artifact log hello.bst
# Ok, now I can see the log, but I didn't get a signpost from the 'bst build' instafail.

bst artifact delete hello.bst
bst build hello.bst
# Hurrah, I retried :)

In GitLab by [Gitlab user @aevri] on Mar 20, 2019, 16:27
[Gitlab user @cs]-shadow has expressed some interested in this

In GitLab by [Gitlab user @aevri] on Oct 28, 2019, 13:34
unassigned [Gitlab user @aevri]

In GitLab by [Gitlab user @abderrahimk] on Jun 23, 2020, 12:13
mentioned in merge request !1970

In GitLab by [Gitlab user @marge-bot123] on Jul 29, 2020, 16:48
mentioned in commit 7f0e0205c066f6ad3e4cb2cb8d719c8022d68ab9


text:
Fixup e72113e send optaplanner test messages asynchronously

Minor amendment to the optaplanner rework in e72113e8e67347d057b23dd918ef3a2f9731527e. Since we're polling for results, the sending of message payloads can be async 'fire and forget'.

Test failure is unrelated to this change.

repo: apache/camel-quarkus


text:
Quartz: solve autowired scheduler properly and remove quick workaroun…

fixes https://github.com/apache/camel-quarkus/issues/4076
change requires https://github.com/quarkusio/quarkus/pull/28218, therefore based on quarkus-main
supersedes https://github.com/apache/camel-quarkus/pull/4119
Problem is caused by the fact, that Quarkus may @Produce null as aorg.quartz.Scheduler. Unfortunately feature can not be changed on quarkus side (see ticket)
This PR proposes a fix in camel-qurkus. It is possible to override autowiring mechanism thanks to @Recorder .
Arc is used directly during autowirng, therefore null occurrence can be ignored.
I injected the custom code into doStart() method.
JUnit tests covering different cases are present.

@jamesnetherton here is a better fix of quartz (for camel 3.19.0), it requires new Quarkus. This PR replaces https://github.com/apache/camel-quarkus/pull/4119

I manually merged this to quarkus-main with some minor tweaks.
Seems that QuartzComponent.doStartScheduler is invoked before QuartzComponent.doStart, so I moved the custom logic to that method. Otherwise Camel proceeds to create a scheduler even if the Quarkus one was provided.

authors: JiriOndrusek jamesnetherton


text:
fix links in contibutor docs.

I have some doubts, even on the other PR, I'm not sure the website will build in this way.

Yeah I was curious about it. The website is working fine as things are. Wondering why these changes are needed?

i am not sure if this have an impact on how the site is built. i just followed github links and it was giving error as

Indeed, now your PR makes much more sense to us! As it happens, these adoc files are not primarily intended for github browsing. They serve as an input for Apache Camel site build whose resuts can be visible at https://camel.apache.org/camel-quarkus/latest/index.html Hence links work as designed. Thanks for taking care anyway!

authors: jamesnetherton onderson ppalaga


text:
CAMEL-20113: camel/tests consolidating

Description
org.apache.camel.itest.jms:
-AdviceWithTransactionIssueTest - moved to camel-jms;
-DynamicRouteTest - moved to camel-jms;
-FileToJmsTest - moved to camel-jms;
-JmsConsumerShutdownTest - moved to camel-jms;
-JmsIntegrationTest - jms sending to queue and processing with custom bean is checked in others tests, so it is deleted;
-JmsMediumQueuePerformanceTest - tests to play with the performance tuning are org.apache.camel.component.jms.tuning.PerformanceRouteTest and PerformanceRoutePojoTest, so this test is deleted;
-JmsPerformanceTest - tests to play with the performance tuning are org.apache.camel.component.jms.tuning.PerformanceRouteTest and PerformanceRoutePojoTest, so this test is deleted;
-JmsPollingConsumerTest - moved to camel-jms;
-JmsPollEnrichTest - moved to camel-jms;
-JMSNestedTransactionRollbackTest - moved to camel-jms;
-JmsResequencerTest - resequence is checked in other camel-jms tests, so this test is deleted;
-JmsValidatorTest - moved to camel-jms;
-SpringJmsValidatorTest - moved to camel-jms;
Tracking
https://issues.apache.org/jira/browse/CAMEL-20113

/component-test camel-jms

/component-test camel-jms

I wonder if this test fails for you also

org.apache.camel.component.jms.JmsDeliveryDelayTest.testInOnlyWithDelay
10.21s
FAILURE

I wonder if this test fails for you also
org.apache.camel.component.jms.JmsDeliveryDelayTest.testInOnlyWithDelay	10.21s	FAILURE

Yes, it have failed in the build job, but it passes in /component-test camel-jms and local runs

/component-test camel-jms

I wonder if this test fails for you also
org.apache.camel.component.jms.JmsDeliveryDelayTest.testInOnlyWithDelay	10.21s	FAILURE

Yes, it have failed in the build job, but it passes in /component-test camel-jms and local runs EDIT: it seems that sometimes it fails in /component-test camel-jms also. Should I look into why it fails?

Hi, yes. Ideally we want to reduce the flakiness of our tests, so we can trust the results from our CI.

I wonder if this test fails for you also
org.apache.camel.component.jms.JmsDeliveryDelayTest.testInOnlyWithDelay	10.21s	FAILURE

Yes, it have failed in the build job, but it passes in /component-test camel-jms and local runs EDIT: it seems that sometimes it fails in /component-test camel-jms also. Should I look into why it fails?

Hi, yes. Ideally we want to reduce the flakiness of our tests, so we can trust the results from our CI.

Hello, thanks for the reply! Ok, I'll look into it.
Do I understand correctly that it is failing due to assertion error?
Also, should I work on it into a separate pr? It seems nothing to do with the tests moved in this pr.

Do I understand correctly that it is failing due to assertion error?

I think so. It seems to be flaky, but unfortunately GH doesn't keep logs from multiple runs so I cannot double check (and add it here to make it easier for you to debug).

Also, should I work on it into a separate pr? It seems nothing to do with the tests moved in this pr.

Oh, my bad. I am sorry. I thought this test was one of the tests modified as part of the PR ... but after a second review, that's not the case.
In this case, it's something to be fixed later (feel free to contribute it if you want, otherwise we will fix it eventually).
Based on that, I think this one is ready to go.

authors: davsclaus kulagaIA orpiske

repo: apache/camel


text:
remove duplicate "Available as of Camel version" entries from compone…

…nt asciidoc files

Thanks for the PR it has been merged. Do you mind closing this?

authors: PascalSchumacher davsclaus


text:
CAMEL-17012: Fix listAliases operation in camel-aws2-lambda

From what I have grabbed, I think the functionVersion parameter was never required. If that's not the case, maybe relaxing the constraint could be discussed.


text:
[CARBONDATA-2771]block update and delete on table if the compaction is in progress

Problem:
if update is triggered when comapction is in progress, in some cases there will be data miss match and sometimes update fails as the segmentId mapping which is used in getPartitions may differ, during update operation.
Solution
Block update and delete if compaction is in progress for table
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[x] Any interfaces changed?
NA

[x] Any backward compatibility impacted?
NA

[x] Document update required?
NA

[x] Testing done
UTs are added
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7410/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6174/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7433/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6191/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7447/

Build Failed  with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7453/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5980/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6212/

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7465/

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6221/

retest this please

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7477/

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6234/

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5992/

LGTM

repo: apache/carbondata


text:
[CELEBORN-1572][FOLLOWUP] master --send-worker-event should support RECOMMISSION and NONE event type

What changes were proposed in this pull request?
master --send-worker-event should support RECOMMISSION and NONE event type.
Why are the changes needed?
Legal event types of /api/v1/workers/events are IMMEDIATELY, DECOMMISSION, DECOMMISSION_THEN_IDLE, GRACEFUL, RECOMMISSION, NONE. Therefore, master --send-worker-event should also support RECOMMISSION and NONE event type.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
TestCelebornCliCommands#master --send-worker-event

Ping @akpatnam25, @mridulm.

repo: apache/celeborn