text:
[FLINK-14042] [flink-table-planner] Fix TemporalTable row schema always inferred as nullable

What is the purpose of the change
This fixes FLINK-14042.
Fix a bug in the table planner where FlinkTableFunctionImpl.getRow would generate an incorrect type for schemas created for Temporal Tables
Verifying this change
This change is already covered by existing tests, such as (please describe tests).
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)

Anyone?

Hi @YuvalItzchakov is there any progress on the unit test? Do you need any help?

@aljoscha Hi, I just noticed I got the reply. Working on it now :)

@twalthr A general question: what are CorrelateTest and CorrelateITTest used for?

Those tests test table functions. But there should also be tests for temporal table functions. Test indicates that there is not Flink cluster used for testing. ITCase means we run a full program and validate the output results.

@twalthr I see: TemporalTableJoinTest and TemporalTableJoinValidationTest. I'm not sure this exactly fits in either but I could just put it in one of them.

We can also discuss the location once you wrote the test. Maybe TimeAttributesITCase could also be ok.

@twalthr OK, once I wrote the test I figured out my fix here won't solve the problem. After doing some more debugging, I've come to the following findings:
Given the following test:
val util = scalaStreamTestUtil()

val sourceTable = util.addTableSource[(Timestamp, Long, String)]("MyTable", 'f1, 'f2, 'f3)
val temporal =
util.tableEnv.sqlQuery("SELECT f1, f2, COLLECT(DISTINCT f3) f3s FROM MyTable GROUP BY f1, f2")
val temporalFunc = temporal.createTemporalTableFunction('f1, 'f2)

util.tableEnv.registerFunction("f", temporalFunc)
val queryTable =
  util.tableEnv.sqlQuery("select f1, f2, b from MyTable, LATERAL TABLE(f(f1)) AS T(a, b, cs)")

util.verifyPlan(queryTable)

If we first look at the generated table schema for the underlying table by the SQL query for the temporal table, we see:
root
 |-- f1: TIMESTAMP(3)
 |-- f2: BIGINT
 |-- f3s: MULTISET<STRING> NOT NULL

When FlinkPlanner validates the SQL query it reaches the part where it needs to validate the TemporalTableFunction I've defined called f. It calls FunctionCatalogOperatorTable.lookupOperatorOverloads to search for the method and once it finds it, it needs to convert it to a standard SqlFunction. In order to do that, it needs to convert the TypeInformation[T] of the TemporalTableFunction's result type to the new DataType API.
Problem is, TypeInformation[T] doesn't carry information about the nullability of the field, thus when the conversion to Multiset[T] happens, it ends up calling the default constructor which sets nullable = true by default, which ends up blowing up at runtime because the TableSchema expected a NOT NULL field.
I'm not entirely sure how we can get around this issue.

Thanks for looking into this @YuvalItzchakov and the detailed explanation. I will also take a look at this problem with the test case you've provided. In general, the nullability topic is difficult in the old planner because TypeInformation was unable to store it. That's why we introduced DataType recently, however, the legacy planner should not use it internally. The Blink planner will fully support this type. I will come back to you shortly.

Your findings are correct @YuvalItzchakov. It is another issue that shows that FLIP-37 and FLIP-65 are important. Luckily, the new Blink planner will not have these problems anymore as it deeply integrated the new logical types with proper nullability. I have a working solution that could solve this use case in particular:
https://github.com/apache/flink/compare/master...twalthr:FLINK-14042?expand=1.
We could merge this as a temporary solution. Of course it does not solve further use cases with other types. But FLIP-65 might solve this issue on the way for the next release.

@twalthr The problem doesn't only occur for Multiset. It's engrained in every logical type which defaults to NULL (I think that's all of them?) and is being used in conjunction with a temporal table.
Does Blink not currently suffer from the same problem?

That's why I mentioned Of course it does not solve further use cases with other types.. The Blink planner is not based on type information anymore, so the problem should be solved. With FLIP-65 we will get proper logical types out of functions which could also help for this particular issue.

@twalthr Ah, sorry I missed that :)
What do you think would be the best way to move forward with this issue, given these FLIP's are still being finalized?

I don't see a good solution without touching a lot of code. I think it is not worth it at the moment given that FLIP-65 is already in the process of voting. If you come up with a good short-term solution, let me know :/

@twalthr This issue doesn't seem to be fixed by Blink. I enabled it and re-ran the test and I still get the same error.

@twalthr @aljoscha Did anything change with respect to Flink 1.10 on this issue?

Anyone?

@YuvalItzchakov sorry for now responding earlier. Let's come to a conclusion here. FLIP-65 has been implemented recently for table functions: https://issues.apache.org/jira/browse/FLINK-15802
So we can finally have a long-term fix for your issue in 1.11.

I will run your test on the current master to verify FLINK-15802 as well.

authors: YuvalItzchakov aljoscha twalthr


text:
GEODE-9360: Initial checkin of .net core support

Still a long way to go with this, but basic put/get/remove of strings works, along with infrastructure to get there, and AuthInitialize.
To build, after installing .net core:
* dotnet build from /netcore in geode-native tree
To run tests:
* Add geode-native library directory to $PATH (Windows) or $LD_LIBRARY_PATH (Linux), or
* Put geode-native library and apache-geode-c library in standard library location (Linux, MacOS), or
* Put geode-native library and apache-geode-c library in app working directory (Windows, MacOS), then
* dotnet test from /netcore in geode-native tree

any public package available to use the .net core client of Geode?

No, sorry.  There is a ton of work left to get to a full client, and this particular attempt at a first cut may not even make it into the final product.  You can build it yourself and use it's (extremely limited) functionality, but it's not up on Nuget and we don't post binaries for geode-native builds.

source: gharchive/pull-request

authors: pdxcodemonkey prashant-upadhyay

repo: apache/geode-native

license_type: permissive

license_source: github-api


text:
GEODE-4913: gfsh start server cmd is not recognizing local properties…

… and cache.xml, instead using default

Reverted having a default cluster xml, having a default xml
in the cluster config caused a default cache server to be
launched, which caused the sighted issue.

Instead cluster config is initialized only when needed.

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

pre-checkin is green except the failure regarding GEODE-4963

repo: apache/geode


text:
GEODE-7670: Add Tests for PR clear

Added distributed tests to verify that the clear operation on
Partitioned Regions works as expected when there are other
concurrent operations happening on the cache (puts, gets, members
added and members removed).
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[X] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[X] Is your initial contribution a single, squashed commit?

[X] Does gradlew build run cleanly?

[X] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

The :geode-redis:distributedTest task has failed twice while running CI for this PR:
> Task :geode-redis:distributedTest

org.apache.geode.redis.RedisDistDUnitTest > classMethod FAILED
    java.lang.AssertionError: Suspicious strings were written to the log during this run.
    Fix the strings or use IgnoredException.addIgnoredException to ignore.
    -----------------------------------------------------------------------
    Found suspect string in log4j at line 5300

    [error 2020/03/26 13:30:55.981 GMT <GeodeRedisServer-WorkerThread-3> tid=98] org.apache.geode.ToDataException: toData failed on DataSerializer with id=0 for class class java.util.HashMap

The above can be safely ignored while reviewing the changes as this PR only adds new DistributedTests for the new clear() feature on Partition Regions.

@gesterzhou
I've rebased the PR against feature/GEODE-7665 to include the latest changes from the clear feature. There are still some failures while running the tests, concurrent putAll with clear (test method clearWithConcurrentPutAllShouldWorkCorrectly) always reproduces a discrepancy between primary buckets a secondary buckets for PARTITION_REDUNDANT regions.
Other tests also sporadically fail when I uncomment the following lines in the assertRegionBucketsConsistency method too:
assertThat(otherDump.getRvv())
    .as("RegionVersionVector for bucket " + bucketId + " on member "
        + otherDump.getMember() + " is not consistent with member " + firstDump.getMember())
    .isEqualTo(firstDump.getRvv());

@agingade
According to what we discussed offline about the clear operation:

The clear may be successful or partial.
In case of partial, it will throw PartitionedRegionPartialClearException.
In both cases, the secondary bucket data should be same as primary (even with member departing).
The RegionVersionVector between primary and secondary should be same, no matter what.

Having that in mind, I've re-based the PR against the latest changes from the feature/GEODE-7665 branch. There are still some test failures, below are the details:

Test clearWithConcurrentPutsShouldWorkCorrectly always fails for PARTITION_REDUNDANT regions, doesn't matter whether the clear coordinator is the actual data host or the accessor node. After the clear finishes, the RegionVersionVector is not the same on the primary and secondary copies for a particular bucket.
Test clearShouldFailWhenCoordinatorMemberIsBounced always fails for PARTITION_REDUNDANT regions. After the clear finishes, the RegionVersionVector is not the same on the primary and secondary copies for a particular bucket.
Test clearWithConcurrentPutAllShouldWorkCorrectly always fails for PARTITION_REDUNDANT regions, doesn't matter whether the clear coordinator is the actual data host or the accessor node. After the clear finishes, the RegionVersionVector is not the same on the primary and secondary copies for a particular bucket.
Test clearWithConcurrentPutsAndGetsShouldWorkCorrectly always fails for PARTITION_REDUNDANT regions, doesn't matter whether the clear coordinator is the actual data host or the accessor node. After the clear finishes, the RegionVersionVector is not the same on the primary and secondary copies for a particular bucket.
Test clearWithConcurrentPutsShouldWorkCorrectlyWhenNonCoordinatorMembersAreBounced randomly fails (for PARTITION_REDUNDANT regions) with PartitionedRegionPartialClearException or inconsistent RegionVersionVector across buckets copies, I believe I'm hitting the same scenario as the one @DonalEvans described here.

Thoughts?.

@agingade
I've updated the PR with the changes you suggested (add bucketRegion.lockRVVForBulkOp() and bucketRegion.unlockRVVForBulkOp() to PutAllPRMessage.doLocalPutAll). I've also modified the test to correctly compare the RVVs from the buckets.
I'm still seeing some random failures when comparing (primary vs secondary) the values and versions stored within the Bucket, specially whenever the clear is executed concurrently with putAll.
That said, I'd suggest to move your suggestions (bucketRegion.lockRVVForBulkOp() and bucketRegion.unlockRVVForBulkOp()) to a new ticket and only use this PR for the actual distributed test.
Please let me know what you think.

@agingade @gesterzhou
After changing the approach used to shutdown the members during the tests (DistributionMessageObserver instead of CacheWriter) there are no more failures related to the Timeout during netsearch/netload/netwrite I was seeing before, CI is fully green so this PR is ready to merge once you approve it.


text:
[GOBBLIN-1708] Improve TimeAwareRecursiveCopyableDataset to lookback only into datefolders that match range

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-XXX

Description

[ ] Here are some details about my PR, including screenshots (if applicable):

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3563 (1c65c37) into master (d0e530a) will increase coverage by 1.57%.
The diff coverage is 93.75%.

:exclamation: Current head 1c65c37 differs from pull request most recent head 19f89bb. Consider uploading reports for the commit 19f89bb to get more accurate results

@@             Coverage Diff              @@
##             master    #3563      +/-   ##
============================================
+ Coverage     46.81%   48.38%   +1.57%
+ Complexity    10514     7764    -2750
============================================
  Files          2095     1459     -636
  Lines         81945    57458   -24487
  Branches       9129     6627    -2502
============================================
- Hits          38361    27801   -10560
+ Misses        40040    27044   -12996
+ Partials       3544     2613     -931

Impacted Files
Coverage Δ

...gement/copy/TimeAwareRecursiveCopyableDataset.java
83.83% <93.75%> (-0.50%)
:arrow_down:

...a/org/apache/gobblin/cluster/GobblinHelixTask.java
64.51% <0.00%> (-19.36%)
:arrow_down:

.../gobblin/cluster/GobblinHelixTaskStateTracker.java
62.50% <0.00%> (-6.25%)
:arrow_down:

...main/java/org/apache/gobblin/util/HadoopUtils.java
29.54% <0.00%> (-0.33%)
:arrow_down:

...odules/orchestration/AzkabanExecuteFlowStatus.java

...iance/retention/CleanableHivePartitionDataset.java

...a/org/apache/gobblin/writer/JdbcWriterBuilder.java

...apache/gobblin/fork/CopyNotSupportedException.java

...in/kafka/schemareg/CachingKafkaSchemaRegistry.java

... and 640 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Please update the details in PR template

repo: apache/gobblin


text:
GROOVY-9594: Rename whitelist/blacklist in SecureASTCustomizer to mor…

…e meaningful names

Merged, thanks everyone for the feedback.

repo: apache/groovy


text:
ci: Fix problem locating build component

The transitive build script dependency org.jfrog.buildinfo:build-info-extractor:2.3.2 cannot be found.
This commit adds Groovy's artifactory repo to the build script repositories, to make the build find the missing dependency.
The problem started occurring in Grails groovy-joint-workflow Github action.
Error message is:
:buildSrc:compileJava NO-SOURCE
:buildSrc:compileGroovy FROM-CACHE
:buildSrc:processResources NO-SOURCE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar
:buildSrc:assemble
:buildSrc:compileTestJava NO-SOURCE
:buildSrc:compileTestGroovy NO-SOURCE
:buildSrc:processTestResources NO-SOURCE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test NO-SOURCE
:buildSrc:check UP-TO-DATE
:buildSrc:build

FAILURE: Build failed with an exception.

* What went wrong:
A problem occurred configuring root project 'groovy'.
> Could not resolve all artifacts for configuration ':classpath'.
   > Could not find org.jfrog.buildinfo:build-info-extractor:2.3.2.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/org/jfrog/buildinfo/build-info-extractor/2.3.2/build-info-extractor-2.3.2.pom
       - https://plugins.gradle.org/m2/org/jfrog/buildinfo/build-info-extractor/2.3.2/build-info-extractor-2.3.2.pom
       - https://oss.sonatype.org/content/repositories/snapshots/org/jfrog/buildinfo/build-info-extractor/2.3.2/build-info-extractor-2.3.2.pom
       - https://jitpack.io/org/jfrog/buildinfo/build-info-extractor/2.3.2/build-info-extractor-2.3.2.pom
     Required by:
         project : > org.jfrog.buildinfo:build-info-extractor-gradle:3.0.3

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.9.4/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 20s

Related: grails/grails-core#13573

I see now, that this caused another build dependency problem instead, in the "Distribute Groovy SNAPSHOT" workflow:
(should this workflow really be triggered on pull_request by the way?)
* What went wrong:
Execution failed for task ':groovy-ant:asciidoctor'.
> Could not resolve all files for configuration ':groovy-ant:asciidoctor'.
   > Could not find asciidoctorj-groovy-dsl-1.0.0.Alpha2.jar (org.asciidoctor:asciidoctorj-groovy-dsl:1.0.0.Alpha2).
     Searched in the following locations:
         https://groovy.jfrog.io/artifactory/libs-release/org/asciidoctor/asciidoctorj-groovy-dsl/1.0.0.Alpha2/asciidoctorj-groovy-dsl-1.0.0.Alpha2.jar

Merged, Thanks!
The trigger on pull_request is intentional. It doesn't load the generated artifacts into the normal snapshot repo but rather builds artifacts which are available at the bottom of the action run log, e.g. scroll to the bottom of:
https://github.com/apache/groovy/actions/runs/10164041226
This makes it easier for us to test someone's PR when needed. Maybe "Generate Groovy SNAPSHOT" would be a better name than "Distribute Groovy SNAPSHOT".

authors: matrei paulk-asert


text:
HADOOP-17230 - Fix for spaces in %JAVA_HOME%

NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

Accidental request, wrong files pulled. Apologies.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
4m  5s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
36m 40s
trunk passed

+1 :green_heart:
mvnsite
1m 42s
trunk passed

+1 :green_heart:
shadedclient
55m 38s
branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  2s
the patch passed

+1 :green_heart:
mvnsite
1m 27s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
shadedclient
16m 48s
patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 11s
hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 34s
The patch does not generate ASF License warnings.

82m 34s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2250/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2250

Optional Tests
dupname asflicense mvnsite unit

uname
Linux ef15783c21f8 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 5e52955112a

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2250/1/testReport/

Max. process+thread count
308 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2250/1/console

versions
git=2.17.1 maven=3.6.0

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

authors: hadoop-yetus wayneseguin

repo: apache/hadoop


text:
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown

Description of PR
As mentioned on HADOOP-17668, we can't upgrade hbase2 profile version beyond 2.2.4 until we either have hbase 2 artifacts available that are built with hadoop 3 profile by default or hbase 3 is rolled out (hbase 3 is compatible with hadoop 3 versions only).
This PR is to upgrade hbase2 profile version to 2.2.4 and also fix TestTimelineWriterHBaseDown to create connection only after mini cluster is up.
How was this patch tested?
Local testing.
For unit testing, here is the test result with hbase2 profile:
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.996 s - in org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.288 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.485 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 201.434 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.86 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.672 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.734 s - in org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.285 s - in org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.799 s - in org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
[INFO] Running org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.664 s - in org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 103, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  09:33 min
[INFO] Finished at: 2021-12-08T19:32:42+05:30
[INFO] ------------------------------------------------------------------------

For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 39s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
12m 49s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
21m 39s

trunk passed

+1 :green_heart:
compile
21m 50s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
19m  9s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
3m 37s

trunk passed

+1 :green_heart:
mvnsite
1m 26s

trunk passed

+1 :green_heart:
javadoc
1m 21s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 17s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+0 :ok:
spotbugs
0m 43s

branch/hadoop-project no spotbugs output file (spotbugsXml.xml)

+1 :green_heart:
shadedclient
20m 36s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 29s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 34s

the patch passed

+1 :green_heart:
compile
21m 14s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
21m 14s

the patch passed

+1 :green_heart:
compile
19m  9s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
19m  9s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
3m 35s

the patch passed

+1 :green_heart:
mvnsite
1m 25s

the patch passed

+1 :green_heart:
xml
0m  3s

The patch has no ill-formed XML file.

+1 :green_heart:
javadoc
1m 16s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 16s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+0 :ok:
spotbugs
0m 38s

hadoop-project has no data from spotbugs

+1 :green_heart:
shadedclient
20m 52s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 37s

hadoop-project in the patch passed.

+1 :green_heart:
unit
14m 29s

hadoop-yarn-server-timelineservice-hbase-tests in the patch passed.

+1 :green_heart:
asflicense
0m 59s

The patch does not generate ASF License warnings.

198m  2s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3768/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3768

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell xml spotbugs checkstyle

uname
Linux 6b089725be82 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 38fbe6959c6453d0d08b58a58ea29c1d862cb8db

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3768/1/testReport/

Max. process+thread count
820 (vs. ulimit of 5500)

modules
C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3768/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@aajisaka @ayushtkn Could you please take a look?

Merged. Thank you @virajjasani and @ayushtkn.

authors: aajisaka hadoop-yetus virajjasani


text:
HADOOP-17650. Bump solr to unblock build failure with Maven 3.8.1 (#2939)

Reviewed-by: Siyao Meng siyao@apache.org
Description of PR
patch 538ce9c / #2939 on branch-3.3; if good will add to 3.3.3
without this you can only build hadoop with a proxy nexus server
How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

just having nexus review this; will merge if it is happy


text:
YARN-11321. Add RMWebServices#getSchedulerOverview.

JIRA: YARN-11321. Add RMWebServices#getSchedulerOverview.
In the pr of completing the Yarn Federation Router About Page, it was found that some Scheduler indicators are non-accumulative indicators. It is necessary to redefine a new method to obtain related indicators. Add the getSchedulerOverview method in RMWebServices to obtain Scheduler overview information, including the following

Cols

Scheduler Type

Scheduling Resource Type

Minimum Allocation

Maximum Allocation

Maximum Cluster Application Priority

Scheduler Busy %

RM Dispatcher EventQueue Size %

The new method will be tested, covering 3 Scheduler, Fifo Scheduler, Fair Scheduler , Capacity Scheduler.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
39m  4s

trunk passed

+1 :green_heart:
compile
1m 13s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m  6s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  9s

trunk passed

+1 :green_heart:
mvnsite
1m 12s

trunk passed

+1 :green_heart:
javadoc
1m  6s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 52s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 13s

trunk passed

+1 :green_heart:
shadedclient
21m 48s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 57s

the patch passed

+1 :green_heart:
compile
1m  1s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m  1s

the patch passed

+1 :green_heart:
compile
0m 57s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 57s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 44s

the patch passed

+1 :green_heart:
mvnsite
1m  0s

the patch passed

+1 :green_heart:
javadoc
0m 44s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 44s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m  0s

the patch passed

+1 :green_heart:
shadedclient
21m 34s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
104m  1s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 42s

The patch does not generate ASF License warnings.

205m  3s

Reason
Tests

Failed junit tests
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4947

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 3f98a1816710 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ddeaaea91bfe5b49a403ef96fe41521cf06ac346

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/1/testReport/

Max. process+thread count
942 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
43m 44s

trunk passed

+1 :green_heart:
compile
1m 22s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
1m  8s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  3s

trunk passed

+1 :green_heart:
mvnsite
1m 14s

trunk passed

+1 :green_heart:
javadoc
1m 11s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 53s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 29s

trunk passed

+1 :green_heart:
shadedclient
25m  4s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 57s

the patch passed

+1 :green_heart:
compile
1m  3s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
1m  3s

the patch passed

+1 :green_heart:
compile
0m 54s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 54s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 45s

the patch passed

+1 :green_heart:
mvnsite
0m 58s

the patch passed

+1 :green_heart:
javadoc
0m 44s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 41s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m  7s

the patch passed

+1 :green_heart:
shadedclient
24m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
102m 31s

hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 41s

The patch does not generate ASF License warnings.

214m  5s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4947

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 2b241c5b55ef 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ded5def62fea594af2f3db6f92df09316645be0f

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/2/testReport/

Max. process+thread count
932 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 12s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m 19s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
28m  0s

trunk passed

+1 :green_heart:
compile
10m 50s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
9m 33s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
2m  4s

trunk passed

+1 :green_heart:
mvnsite
2m 59s

trunk passed

+1 :green_heart:
javadoc
2m 50s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m 33s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 45s

trunk passed

+1 :green_heart:
shadedclient
22m 45s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 32s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 48s

the patch passed

+1 :green_heart:
compile
9m 21s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
9m 21s

the patch passed

+1 :green_heart:
compile
8m 38s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

-1 :x:
javac
8m 38s
/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 3 new + 649 unchanged - 2 fixed = 652 total (was 651)

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m 58s
/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11)

+1 :green_heart:
mvnsite
2m 35s

the patch passed

+1 :green_heart:
javadoc
2m 11s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m 34s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 34s

the patch passed

+1 :green_heart:
shadedclient
23m  5s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
5m 14s

hadoop-yarn-common in the patch passed.

+1 :green_heart:
unit
103m 18s

hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
1m 13s

The patch does not generate ASF License warnings.

272m 14s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4947

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 81c179f0e9ba 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d4c3813f0898e1dcf498123bc079fc469c7410ff

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/3/testReport/

Max. process+thread count
939 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 57s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
16m  3s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
26m  9s

trunk passed

+1 :green_heart:
compile
9m 50s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
8m 51s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
2m  1s

trunk passed

+1 :green_heart:
mvnsite
2m 42s

trunk passed

+1 :green_heart:
javadoc
2m 45s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m 50s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 44s

trunk passed

+1 :green_heart:
shadedclient
22m 35s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 27s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 38s

the patch passed

+1 :green_heart:
compile
9m 13s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
9m 13s

the patch passed

+1 :green_heart:
compile
8m 52s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

-1 :x:
javac
8m 52s
/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 3 new + 649 unchanged - 2 fixed = 652 total (was 651)

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m 47s
/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)

+1 :green_heart:
mvnsite
2m 26s

the patch passed

+1 :green_heart:
javadoc
2m 13s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m 15s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 33s

the patch passed

+1 :green_heart:
shadedclient
22m 54s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
5m 13s

hadoop-yarn-common in the patch passed.

-1 :x:
unit
106m 38s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
1m 21s

The patch does not generate ASF License warnings.

270m 30s

Reason
Tests

Failed junit tests
hadoop.yarn.server.resourcemanager.TestRMAdminService

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4947

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 1ef4a931f558 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ff51c8a2fec864b6f17bac2a354df50d7d53fd3e

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/4/testReport/

Max. process+thread count
929 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 58s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m 13s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
25m 43s

trunk passed

+1 :green_heart:
compile
9m 55s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
8m 44s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
2m  3s

trunk passed

+1 :green_heart:
mvnsite
2m 44s

trunk passed

+1 :green_heart:
javadoc
2m 36s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m 34s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 34s

trunk passed

+1 :green_heart:
shadedclient
22m 41s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 30s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 39s

the patch passed

+1 :green_heart:
compile
9m 14s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
9m 14s

the patch passed

+1 :green_heart:
compile
8m 50s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
8m 50s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 49s

the patch passed

+1 :green_heart:
mvnsite
2m 40s

the patch passed

+1 :green_heart:
javadoc
2m 14s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m 18s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 33s

the patch passed

+1 :green_heart:
shadedclient
23m 53s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
5m 49s

hadoop-yarn-common in the patch passed.

+1 :green_heart:
unit
104m 23s

hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
1m 16s

The patch does not generate ASF License warnings.

268m 24s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4947

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 8d5ca48614d2 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d828a3b5f2d199d2b1036ae0e4c3296997b184f8

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/5/testReport/

Max. process+thread count
961 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  5s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m 17s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
25m 56s

trunk passed

+1 :green_heart:
compile
9m 56s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
8m 47s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
2m 14s

trunk passed

+1 :green_heart:
mvnsite
2m 40s

trunk passed

+1 :green_heart:
javadoc
2m 33s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m 53s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 56s

trunk passed

+1 :green_heart:
shadedclient
24m 33s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 31s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 43s

the patch passed

+1 :green_heart:
compile
10m 18s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
10m 18s

the patch passed

+1 :green_heart:
compile
9m  7s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
9m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m 47s

the patch passed

+1 :green_heart:
mvnsite
2m 29s

the patch passed

+1 :green_heart:
javadoc
2m 19s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
2m  8s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
4m 44s

the patch passed

+1 :green_heart:
shadedclient
22m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
5m 24s

hadoop-yarn-common in the patch passed.

+1 :green_heart:
unit
103m 35s

hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
1m 10s

The patch does not generate ASF License warnings.

270m 34s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4947

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux a4e135c90ff0 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / db3bc78bbf27df22951bc8c3521b46b4185110a4

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/6/testReport/

Max. process+thread count
951 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4947/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


text:
HBASE-26277: Fix InterfaceAudience for BalanceRequest$Builder and BalanceResponse$Builder

Follow-up from #3663
Apologies for missing this and creating extra work for you all, @Apache9 @ndimiduk @joshelser

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 27s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 59s
master passed

+1 :green_heart:
compile
0m 27s
master passed

+1 :green_heart:
shadedjars
8m 18s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 25s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 40s
the patch passed

+1 :green_heart:
compile
0m 26s
the patch passed

+1 :green_heart:
javac
0m 26s
the patch passed

+1 :green_heart:
shadedjars
8m  9s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 23s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
1m 16s
hbase-client in the patch passed.

28m 49s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3674

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 0ad38f23f9b5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 8679e08dab

Default Java
AdoptOpenJDK-1.8.0_282-b08

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/testReport/

Max. process+thread count
345 (vs. ulimit of 30000)

modules
C: hbase-client U: hbase-client

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 28s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 33s
master passed

+1 :green_heart:
compile
0m 29s
master passed

+1 :green_heart:
shadedjars
8m 15s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 29s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 15s
the patch passed

+1 :green_heart:
compile
0m 30s
the patch passed

+1 :green_heart:
javac
0m 30s
the patch passed

+1 :green_heart:
shadedjars
8m  4s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 26s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
1m 22s
hbase-client in the patch passed.

30m 10s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3674

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 6e26ed7893ed 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 8679e08dab

Default Java
AdoptOpenJDK-11.0.10+9

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/testReport/

Max. process+thread count
299 (vs. ulimit of 30000)

modules
C: hbase-client U: hbase-client

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 30s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 25s
master passed

+1 :green_heart:
compile
0m 59s
master passed

+1 :green_heart:
checkstyle
0m 29s
master passed

+1 :green_heart:
spotbugs
1m  7s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 16s
the patch passed

+1 :green_heart:
compile
0m 57s
the patch passed

+1 :green_heart:
javac
0m 57s
the patch passed

-0 :warning:
checkstyle
0m 28s
hbase-client: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
20m 58s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
spotbugs
1m 16s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 13s
The patch does not generate ASF License warnings.

45m 11s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3674

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux c5cf08853c18 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 8679e08dab

Default Java
AdoptOpenJDK-1.8.0_282-b08

checkstyle
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt

Max. process+thread count
86 (vs. ulimit of 30000)

modules
C: hbase-client U: hbase-client

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3674/1/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Submitted to branch-2 and master

Apologies for missing this and creating extra work for you all

No need to apologize. Thanks for your diligence and care.

authors: Apache-HBase bbeaudreault joshelser

repo: apache/hbase


text:
HBASE-28876 Should call ProcedureSchduler.completionCleanup for non-r…

…oot procedure too
(cherry picked from commit 8cdc1444be22abbc60c191b0b7e7b9cfeeb72c18)

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 37s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
hbaseanti
0m  0s

Patch does not have any anti-patterns.

_ branch-2 Compile Tests _

+0 :ok:
mvndep
0m 11s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 24s

branch-2 passed

+1 :green_heart:
compile
3m 27s

branch-2 passed

+1 :green_heart:
checkstyle
0m 48s

branch-2 passed

+1 :green_heart:
spotbugs
2m  2s

branch-2 passed

+1 :green_heart:
spotless
0m 49s

branch has no errors when running spotless:check.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 17s

the patch passed

+1 :green_heart:
compile
3m 25s

the patch passed

+1 :green_heart:
javac
3m 25s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 44s

the patch passed

+1 :green_heart:
spotbugs
2m 17s

the patch passed

+1 :green_heart:
hadoopcheck
17m 16s

Patch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.

+1 :green_heart:
spotless
0m 48s

patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 19s

The patch does not generate ASF License warnings.

41m 42s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6363

Optional Tests
dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless

uname
Linux 2048f837fae0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / c046d7c2216ae655a6a5d50114abf6cd89ea6f36

Default Java
Eclipse Adoptium-11.0.23+9

Max. process+thread count
79 (vs. ulimit of 30000)

modules
C: hbase-procedure hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/console

versions
git=2.34.1 maven=3.9.8 spotbugs=4.7.3

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 20s

Docker mode activated.

-0 :warning:
yetus
0m  5s

Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+0 :ok:
mvndep
0m 10s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m  7s

branch-2 passed

+1 :green_heart:
compile
1m 16s

branch-2 passed

+1 :green_heart:
javadoc
0m 45s

branch-2 passed

+1 :green_heart:
shadedjars
5m 32s

branch has no errors when building our shaded downstream artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 17s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m  0s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
javadoc
0m 43s

the patch passed

+1 :green_heart:
shadedjars
5m 28s

patch has no errors when building our shaded downstream artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 32s

hbase-procedure in the patch passed.

+1 :green_heart:
unit
212m 54s

hbase-server in the patch passed.

241m 13s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6363

Optional Tests
javac javadoc unit compile shadedjars

uname
Linux 64f2d3263d84 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / c046d7c2216ae655a6a5d50114abf6cd89ea6f36

Default Java
Eclipse Adoptium-17.0.11+9

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/testReport/

Max. process+thread count
4899 (vs. ulimit of 30000)

modules
C: hbase-procedure hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/console

versions
git=2.34.1 maven=3.9.8

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 39s

Docker mode activated.

-0 :warning:
yetus
0m  5s

Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+0 :ok:
mvndep
0m 10s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
2m 30s

branch-2 passed

+1 :green_heart:
compile
0m 59s

branch-2 passed

+1 :green_heart:
javadoc
0m 40s

branch-2 passed

+1 :green_heart:
shadedjars
4m 58s

branch has no errors when building our shaded downstream artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 28s

the patch passed

+1 :green_heart:
compile
1m  1s

the patch passed

+1 :green_heart:
javac
1m  1s

the patch passed

+1 :green_heart:
javadoc
0m 36s

the patch passed

+1 :green_heart:
shadedjars
4m 56s

patch has no errors when building our shaded downstream artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 28s

hbase-procedure in the patch passed.

-1 :x:
unit
234m 25s
/patch-unit-hbase-server.txt
hbase-server in the patch failed.

260m  0s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6363

Optional Tests
javac javadoc unit compile shadedjars

uname
Linux 0b6a47a84c4f 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / c046d7c2216ae655a6a5d50114abf6cd89ea6f36

Default Java
Temurin-1.8.0_412-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/testReport/

Max. process+thread count
4239 (vs. ulimit of 30000)

modules
C: hbase-procedure hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/console

versions
git=2.34.1 maven=3.9.8

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 44s

Docker mode activated.

-0 :warning:
yetus
0m  6s

Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+0 :ok:
mvndep
0m 10s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m  5s

branch-2 passed

+1 :green_heart:
compile
1m  2s

branch-2 passed

+1 :green_heart:
javadoc
0m 38s

branch-2 passed

+1 :green_heart:
shadedjars
5m 53s

branch has no errors when building our shaded downstream artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 17s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
2m 59s

the patch passed

+1 :green_heart:
compile
1m  5s

the patch passed

+1 :green_heart:
javac
1m  5s

the patch passed

+1 :green_heart:
javadoc
0m 37s

the patch passed

+1 :green_heart:
shadedjars
5m 51s

patch has no errors when building our shaded downstream artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 35s

hbase-procedure in the patch passed.

-1 :x:
unit
233m 47s
/patch-unit-hbase-server.txt
hbase-server in the patch failed.

262m 26s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/6363

Optional Tests
javac javadoc unit compile shadedjars

uname
Linux 1d92bc203750 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / c046d7c2216ae655a6a5d50114abf6cd89ea6f36

Default Java
Eclipse Adoptium-11.0.23+9

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/testReport/

Max. process+thread count
4351 (vs. ulimit of 30000)

modules
C: hbase-procedure hbase-server U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6363/1/console

versions
git=2.34.1 maven=3.9.8

Powered by
Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.


text:
HIVE-23998: Upgrade guava to 27 for Hive 2.3 branch

What changes were proposed in this pull request?

This PR proposes to upgrade Guava to 27 in Hive 2.3 branch.
Why are the changes needed?

When trying to upgrade Guava in Spark, found the following error. A Guava method became package-private since Guava version 20. So there is incompatibility with Guava versions > 19.0.
sbt.ForkMain$ForkError: sbt.ForkMain$ForkError: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator
	at org.apache.hadoop.hive.ql.exec.FetchOperator.<init>(FetchOperator.java:108)
	at org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:87)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:541)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)

Does this PR introduce any user-facing change?

Yes. This upgrades Guava to 27.
How was this patch tested?

Built Hive locally.

cc @sunchao Thanks.

@viirya I wonder if we can create another PR targeting branch-2. It seems there is minimum conflict between branch-2.3 and branch-2 for this, and the testing seems to be working for branch-2 (see this as example).

@sunchao OK, let me try it. But I think by the end the 2.3.8 should be from branch-2.3, right?

Yes that's right. If tests pass on branch-2 we can cherry-pick this to branch-2.3 and probably commit it directly, I think.

authors: sunchao viirya

repo: apache/hive


text:
HIVE-27695: Intermittent OOM when running TestMiniTezCliDriver

What changes were proposed in this pull request?
Increase max heap space for Tez Application master to 512MB.
Why are the changes needed?
java.lang.OutOfMemoryError: GC overhead limit exceeded is thrown by the Tez Application Master (AM) cause the current heap size (128MB) is not enough to accommodate the needs of multiple Tez containers running. Each running container requires roughly 10MB of memory in the AM. The AM accumulates/manipulates multiple configuration objects (some of them retaining as much as 1MB of heap) per container. The heap gradually becomes full and GC is spending a lot of CPU time to clean things up without really making much progress since containers are reused and heap cannot shrink.
Extended analysis and justification under HIVE-27695.
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
mvn test -Dtest=TestMiniTezCliDriver

Running all tests for TestMiniTezCliDriver without the changes here leads to OOM 95% of the time.

can we run the flaky job for TestMiniTezCliDriver as well?

Started a flaky check: http://ci.hive.apache.org/job/hive-flaky-check/743/
The TestMiniTezCliDriver needs 10 to 15 minutes to complete so multiplied by 100 runs is roughly 17H so the flaky check will be running for a while.

The http://ci.hive.apache.org/job/hive-flaky-check/743/ run didn't run property some qfiles cause there are some dependencies to test-serde that is not compiled as part of the flaky-check job.
A "hacked" run where the TestMiniTezCliDriver is running properly all tests can be found here: http://ci.hive.apache.org/job/hive-flaky-check/745/

authors: ayushtkn zabetak


text:
[Task]: Update ivy dependency to reduce CVE impact

What needs to happen?
Update ivy to 2.5.2
Issue Priority
Priority: 3
Issue Component
Component: Transforms

.take-issue

Hi,
First, thanks for the great work done on HOP !
We use HOP in docker images and we check their vulnerabilities with Trivy v0.44
This issue is reported as critical because of the cve-2022-37865 and high because of the cve-2022-37866 and cve-2022-46751.
Trivy suggests to move at least to 2.5.2
Our corp is quite fussy about security issues...
Would it be possible to make the update in the next release please ?
Regards

We are in the process of creating the next release, one of the final steps we do is a trivy scan and try to upgrade all we can. In some cases we are unable to do so because of 3rd party dependencies. Will give give an update here if this one is resolved

source: gharchive/issue

authors: hansva ricozen

repo: apache/hop


text:
[BUG] Hudi 0.11.x Spark SQL CTAS requires hoodie properties in addition to table properties

Describe the problem you faced
Hudi guide for Spark SQL Create table states that record key, precombine key and partition keys should be specified with special arguments primaryKey, preCombineField and partition by ().
To Reproduce
Steps to reproduce the behavior:
Failure scenario:

Run SQL as per docs:

CREATE TABLE {target_db}.{target_table_name} using hudi
location 's3://{target_bucket_name}/{target_table_name}'
options (
    type = 'cow', primaryKey='{pk}', preCombineField='{precombine_field}',

    hoodie.table.name='{target_table_name}',
    hoodie.datasource.write.operation='upsert',
    hoodie.datasource.write.table.name='{target_table_name}',
)
partitioned by ({partition_fields})
AS
SELECT *
FROM {source_db}.{source_table};

Throws Caused by: org.apache.avro.AvroRuntimeException: Not a valid schema field: ts . Notice that this is using the default value for preCombineField=ts instead of user specified custom field. This indicates that preCombineField option has no effect on Hudi when creating tables using Spark SQL.
Success Scenario:

run following SQL with hoodie properties specified in addition to SQL options

CREATE TABLE {target_db}.{target_table_name} using hudi
location 's3://{target_bucket_name}/{target_table_name}'
options (
    type = 'cow', primaryKey='{pk}', preCombineField='{precombine_field}',

    hoodie.table.name='{target_table_name}',
    hoodie.datasource.write.operation='upsert',
    hoodie.datasource.write.table.name='{target_table_name}',

    hoodie.datasource.write.recordkey.field='{primary_key}',
    hoodie.datasource.write.precombine.field='{precombine_field}',
    hoodie.datasource.write.partitionpath.field='{partition_fields}',
    hoodie.datasource.write.keygenerator.class='org.apache.hudi.keygen.ComplexKeyGenerator',
)
partitioned by ({partition_fields})
AS
SELECT *
FROM {source_db}.{source_table};

this is successful because both primaryKey and hoodie.datasource.write.recordkey.field are specified. both preCombineField and hoodie.datasource.write.precombine.field are specified and both partitioned by () and hoodie.datasource.write.partitionpath.field are specified.
Expected behavior
Failure scenario above should be successful.
Environment Description
EMR 6.7.0

Hudi version : 0.11.1
Spark version : 3.2.1
Hive version : 3.1.3
Hadoop version :
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : no

Additional context
Add any other context about the problem here.
Stacktrace
Traceback (most recent call last):
  File "/tmp/spark-618dec17-fd89-49da-abff-8893b20c3183/hudi_remodeller.py", line 66, in <module>
    args.primary_key, args.precombine_field, args.partition_fields, args.region)
  File "/tmp/spark-618dec17-fd89-49da-abff-8893b20c3183/hudi_remodeller.py", line 45, in remodel_table
    spark.sql(sql)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 723, in sql
  File "/usr/lib/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
  File "/usr/lib/spark/python/lib/py4j-0.10.9.3-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o90.sql.
: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20221129132817905
	at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:64)
	at org.apache.hudi.table.action.commit.SparkUpsertCommitActionExecutor.execute(SparkUpsertCommitActionExecutor.java:45)
	at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.upsert(HoodieSparkCopyOnWriteTable.java:113)
	at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.upsert(HoodieSparkCopyOnWriteTable.java:97)
	at org.apache.hudi.client.SparkRDDWriteClient.upsert(SparkRDDWriteClient.java:155)
	at org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:213)
	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:306)
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:171)
	at org.apache.spark.sql.execution.datasources.DataSource.writeAndRead(DataSource.scala:519)
	at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.saveDataIntoTable(createDataSourceTables.scala:228)
	at org.apache.spark.sql.execution.command.CreateDataSourceTableAsSelectCommand.run(createDataSourceTables.scala:182)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:115)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
	at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
	at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
	at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
	at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:112)
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:108)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:519)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:83)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:519)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:495)
	at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:108)
	at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:95)
	at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:93)
	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:221)
	at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:101)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:98)
	at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 18.0 failed 4 times, most recent failure: Lost task 7.3 in stage 18.0 (TID 270) ([2a05:d018:116b:f601:5602:379f:9471:369e] executor 2): org.apache.avro.AvroRuntimeException: Not a valid schema field: ts
	at org.apache.avro.generic.GenericData$Record.get(GenericData.java:256)
	at org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal(HoodieAvroUtils.java:501)
	at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$write$10(HoodieSparkSqlWriter.scala:272)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:65)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:133)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1474)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2610)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2559)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2558)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2558)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1200)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1200)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1200)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2798)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2740)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2729)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:978)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2215)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2236)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2255)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2280)
	at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
	at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362)
	at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361)
	at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
	at org.apache.hudi.data.HoodieJavaRDD.collectAsList(HoodieJavaRDD.java:157)
	at org.apache.hudi.index.simple.HoodieSimpleIndex.fetchRecordLocationsForAffectedPartitions(HoodieSimpleIndex.java:142)
	at org.apache.hudi.index.simple.HoodieSimpleIndex.tagLocationInternal(HoodieSimpleIndex.java:113)
	at org.apache.hudi.index.simple.HoodieSimpleIndex.tagLocation(HoodieSimpleIndex.java:91)
	at org.apache.hudi.table.action.commit.HoodieWriteHelper.tag(HoodieWriteHelper.java:49)
	at org.apache.hudi.table.action.commit.HoodieWriteHelper.tag(HoodieWriteHelper.java:32)
	at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:53)
	... 58 more
Caused by: org.apache.avro.AvroRuntimeException: Not a valid schema field: ts
	at org.apache.avro.generic.GenericData$Record.get(GenericData.java:256)
	at org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal(HoodieAvroUtils.java:501)
	at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$write$10(HoodieSparkSqlWriter.scala:272)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:65)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:133)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1474)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

I started spark/hudi today and this was happening to me. it's because the default precombine field is 'ts'. Set the precombine field to any (numeric?) field in your table and you'll get past this message.

Ok, I used spark 3.2.1 and Hudi 0.11.1 and did these steps:
Started spark-sql by running
$SPARK_HOME/bin/spark-sql \
--jars $HUDI_HOME/packaging/hudi-spark-bundle/target/hudi-spark3.2-bundle_2.12-0.11.1.jar  \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
--conf 'spark.sql.catalogImplementation=in-memory'

Then I created a table doing
create table no_ts (
  id int,
  name string,
  price double,
  age int,
  part string
) using hudi
tblproperties (
  type = 'mor',
  primaryKey = 'id',
  preCombineField = 'price',
  hoodie.datasource.meta.sync.enable = false,
  hoodie.datasource.hive_sync.enable = false
)
partitioned by (part);

and inserted data with:
insert into no_ts select 1, 'a1', 100, 10, 'p1';
insert into no_ts select 2, 'a2', 200, 12, 'p2';

I then created a second table from that one by doing
create table copy_no_ts using hudi
  options (
    type = 'mor',
    primaryKey = 'id',
    preCombineField = 'price',
    hoodie.datasource.meta.sync.enable = false,
    hoodie.datasource.hive_sync.enable = false
  )
  partitioned by (part)
  AS
  SELECT *
  FROM no_ts;

No exception occurred and I both tables contained the correct data when queried.

@atharvai : can you respond please. looks like Jon could not repro.

authors: atharvai haydenflinner jonvex nsivabalan

repo: apache/hudi


text:
[HUDI-7580] Fix order of fields when records inserted out of order

Change Logs
Fix order of fields when records inserted out of order.
Impact
Bug fix, no public api change.
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

Wouldn't it be better to fail create table if the partition columns are not at the end? This seems like it could lead to more confusion since you still need to do the insert into with the partition columns at the end.

Spark-sql has no such restriction, so I don't think it's a good idea to fail create table if the partition columns are not at the end. Also, note that the issue was fixed in Hudi 0.11.0 in https://github.com/apache/hudi/commit/ea1fbc71ecf9eec20815e2bc51bb8489e4ca1804, and unfortunately regressed in https://github.com/apache/hudi/commit/cfd0c1ee34460332053491fd1e68c2607c14e958. This PR is merely trying to restore the behavior of Hudi 0.11.0 while still keeping the perf improvements made by the latter commit intact.

Also, how would this affect the other sql commands?

This change is only in InsertIntoHoodieTableCommand, and should not affect any sql command other than INSERT INTO and CREATE TABLE AS SELECT.

From your test
        spark.sql(
          s"""
             |create table $tableName (
             |  id int,
             |  name string,
             |  price int,
             |  ts long
             |) using hudi
             | options (
             |  primaryKey ='id',
             |  type = 'mor',
             |  preCombineField = 'ts'
             | )
             | partitioned by(price)
             | location '$basePath'
       """.stripMargin)
        spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
        spark.sql(s"insert into $tableName values(2, 'a2', 100, 200000)")
        checkAnswer(s"select id, name, ts, price from $tableName")(
          Seq(1, "a1", 10, 1000),
          Seq(2, "a2", 100, 200000)
        )

we say the schema is "id, name, price, ts". But then in the insert "1, 'a1', 10, 1000" the 1000 corresponds to price. So the columns are not in the order they are defined in create table.

@jonvex Can you please take a look at the last commit? I have updated the test to make it more clear what is getting fixed here. Essentially, the order of fields in the table is (id, name, price, ts). The test inserts out-of-order i.e. (id, name, ts, price) and then it queries the table in the same order as defined when creating the table. The query order does not matter here, it is the out-of-order insert that is getting fixed.

@codope That makes sense if you need to specify the column names in insert into, but what happens if you did it in a different order than the one you tried?
In the previous test iteration, it was
spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
spark.sql(s"insert into $tableName values(2, 'a2', 100, 200000)")

and didn't specify which value corresponds to a column. Why did it assume that the order was "id, name, ts, price" but we defined the table as "id, name, price, ts". That is the part that is confusing me. The default order should match the table order. But from what I am seeing, the default order still has the partition column at the end.
Will the test still work if it is:
spark.sql(s"insert into $tableName (id, name, ts, price) values(1, 'a1', 10, 1000)")
spark.sql(s"insert into $tableName (name,  price, ts, id) values('a2', 200000, 100, 2)")

No, we have added validation and updated the docs.

authors: codope jonvex


text:
Make field-id of name-mapping optional

Feature Request / Improvement
See https://github.com/apache/iceberg-rust/pull/740/files#r1862755325

@barronw certainly, thanks!

repo: apache/iceberg-python