text:
Fix the comparison of generic types

The comparison cannot shortcut when a condition passes, only when one fails,
since it's basically a big AND. E.g. even if both lower bounds are null (and
therefore equal), the upper bounds still need to be compared. The only
obvious shortcut is if the operands are the same object, as the first
condition.
I did not manage to create a test that would fail without the change. However, it's a change similar to fae2911 which ended up causing GROOVY-7742, so I think it should only be applied to master for the moment.

the commit is ok, but it could have been a smaller one, if I look at your changes in line 1600 and following. Same for the xor parts, the old logic could have stayed, only the following line with == was really wrong here. You also don't really need to check for == to avoid a call to equalIncludingGenerics, the method will do that check anyway. Where it does makes sense, is of course in the upper bounds check, because they can be null now, thus you need to check or else you cannot get the length (you know that of course, it is just for a third party reader)

Yes, as I wrote in my earlier comment, I took the opportunity of the fix to make the code a bit more regular and consistent: the lower and upper bounds are compared in a similar way, itself similar to the way the generic types are compared in the second method (replace "upper" by "gt" and the blocks are identical), and the use of ^ is generalized (I introduced it in a previous PR). I felt it made the code easier to understand.
The lower1 != lower2 check protecting the equalIncludingGenerics call is indeed useless and can be removed to simplify.
However, I can also definitely limit the change to removing the lower1 == lower2 shortcut if you'd rather keep the change as small as possible. I think the code is more brittle (and a bit less readable) as it is -- you couldn't swap the lower and upper bounds checks as they are if you wanted to, for example, because of the upper1 == upper2 shortcut which would become as incorrect as lower1 == lower2 currently is -- but it certainly works. I just mean my change is not purely a style change.

Thanks! :+1:

authors: PascalSchumacher blackdrag fpavageau

repo: apache/groovy


text:
HADOOP-12345,HADOOP-11823 fixes

I have already filed the HADOOP-12345 and HADOOP-11823 jira.
The fix for HADOOP-12345 is to correctly compute the credential length which is passed as part of the NFS request. We need to round the XDR bytes to a multiple of 4 if the hostname in the credential is not a multiple of 4.
The fix for HADOOP-11823 is when RPC returns a denied reply, the code should not check for a verifier. It is a bug as it doesn't match the RPC protocol. (See Page 33 from NFS Illustrated book).
This is my first time contributing to Hadoop.

I will separate out the changes for HADOOP-12345 and HADOOP-11823 and submit two pull requests

source: gharchive/pull-request

repo: apache/hadoop

license_type: permissive

license_source: github-api


text:
HDFS-15371. Nonstandard characters exist in NameNode.java

@goiri The issue HDFS-9929 forget to delete the comma.
NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-XXXXX. Fix a typo in YYY.)
For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 36s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
27m 28s
trunk passed

+1 :green_heart:
compile
1m 25s
trunk passed

+1 :green_heart:
checkstyle
0m 58s
trunk passed

+1 :green_heart:
mvnsite
1m 35s
trunk passed

+1 :green_heart:
shadedclient
20m 40s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 48s
trunk passed

+0 :ok:
spotbugs
3m 44s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 41s
trunk passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 16s
the patch passed

+1 :green_heart:
compile
1m 10s
the patch passed

+1 :green_heart:
javac
1m 10s
the patch passed

-0 :warning:
checkstyle
0m 45s
hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 90 unchanged - 1 fixed = 91 total (was 91)

+1 :green_heart:
mvnsite
1m 23s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
shadedclient
17m 53s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 45s
the patch passed

+1 :green_heart:
findbugs
3m 53s
the patch passed

_ Other Tests _

-1 :x:
unit
138m 55s
hadoop-hdfs in the patch passed.

-1 :x:
asflicense
0m 34s
The patch generated 1 ASF License warnings.

226m  4s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestReadStripedFileWithMissingBlocks

hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier

hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped

hadoop.hdfs.server.datanode.TestDataNodeFaultInjector

hadoop.hdfs.TestStripedFileAppend

hadoop.hdfs.server.datanode.TestHSync

hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes

hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

hadoop.hdfs.server.namenode.TestReconstructStripedBlocks

hadoop.hdfs.TestReconstructStripedFile

hadoop.hdfs.server.namenode.TestAddStripedBlocks

hadoop.hdfs.server.datanode.TestBlockScanner

hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics

hadoop.hdfs.server.namenode.TestFileContextAcl

hadoop.hdfs.server.datanode.checker.TestDatasetVolumeCheckerTimeout

hadoop.hdfs.TestGetFileChecksum

hadoop.hdfs.server.namenode.TestDefaultBlockPlacementPolicy

hadoop.hdfs.server.namenode.ha.TestHASafeMode

hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

hadoop.hdfs.server.datanode.TestDataNodeReconfiguration

hadoop.hdfs.server.namenode.ha.TestHAAppend

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2032/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2032

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 060bd25010d1 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / f43a152b972

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-2032/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-2032/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-2032/1/testReport/

asflicense
https://builds.apache.org/job/hadoop-multibranch/job/PR-2032/1/artifact/out/patch-asflicense-problems.txt

Max. process+thread count
2851 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-2032/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

authors: hadoop-yetus zhaoyim


text:
HDFS-16437 ReverseXML processor doesn't accept XML files without the …

…SnapshotDiffSection

Description of PR
How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 56s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
35m 56s

trunk passed

+1 :green_heart:
compile
1m 33s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 22s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 57s

trunk passed

+1 :green_heart:
mvnsite
1m 27s

trunk passed

+1 :green_heart:
javadoc
1m  5s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 37s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 25s

trunk passed

+1 :green_heart:
shadedclient
25m 57s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 21s

the patch passed

+1 :green_heart:
compile
1m 23s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 23s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 51s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 25 unchanged - 0 fixed = 30 total (was 25)

+1 :green_heart:
mvnsite
1m 19s

the patch passed

+1 :green_heart:
javadoc
0m 53s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 22s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 27s

the patch passed

+1 :green_heart:
shadedclient
25m 27s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
325m 20s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 39s

The patch does not generate ASF License warnings.

434m 30s

Reason
Tests

Failed junit tests
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3926

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 9ab4ccd2efc5 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 485dd5590a9cdb04d6dd48dd82eb0a9a62307318

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/1/testReport/

Max. process+thread count
2069 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 54s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
34m 48s

trunk passed

+1 :green_heart:
compile
1m 29s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 19s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 56s

trunk passed

+1 :green_heart:
mvnsite
1m 28s

trunk passed

+1 :green_heart:
javadoc
1m  3s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 28s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 24s

trunk passed

+1 :green_heart:
shadedclient
25m 23s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 21s

the patch passed

+1 :green_heart:
compile
1m 23s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 23s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 50s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 25 unchanged - 0 fixed = 30 total (was 25)

+1 :green_heart:
mvnsite
1m 20s

the patch passed

+1 :green_heart:
javadoc
0m 54s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 21s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 27s

the patch passed

+1 :green_heart:
shadedclient
25m 50s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
329m 42s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 37s

The patch does not generate ASF License warnings.

437m 14s

Reason
Tests

Failed junit tests
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3926

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 63a42cc43505 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / cf5807e9fefb82cede403a6c5fbc882be9f75aa4

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/2/testReport/

Max. process+thread count
1980 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  4s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
35m 19s

trunk passed

+1 :green_heart:
compile
1m 30s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 19s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 57s

trunk passed

+1 :green_heart:
mvnsite
1m 28s

trunk passed

+1 :green_heart:
javadoc
1m  2s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 32s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 25s

trunk passed

+1 :green_heart:
shadedclient
26m  4s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 22s

the patch passed

+1 :green_heart:
compile
1m 23s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 23s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 52s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 25 unchanged - 0 fixed = 30 total (was 25)

+1 :green_heart:
mvnsite
1m 21s

the patch passed

+1 :green_heart:
javadoc
0m 53s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 23s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 26s

the patch passed

+1 :green_heart:
shadedclient
25m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
325m 37s

hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 38s

The patch does not generate ASF License warnings.

434m 39s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3926

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 28b98e65c898 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 87d3183ab24ec0c8f6dcba69b3be0173cd00e3cc

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/3/testReport/

Max. process+thread count
1911 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Hello @tasanuma  @jojochuang @goiri @aajisaka @ayushtkn ,PTAL. Tanks a lot.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
13m 50s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
33m 21s

trunk passed

+1 :green_heart:
compile
1m 28s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 21s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  0s

trunk passed

+1 :green_heart:
mvnsite
1m 27s

trunk passed

+1 :green_heart:
javadoc
1m  1s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 31s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 17s

trunk passed

+1 :green_heart:
shadedclient
22m 23s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m 54s
/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

-1 :x:
compile
0m 56s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt
hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.

-1 :x:
javac
0m 56s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.txt
hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04.

-1 :x:
compile
0m 54s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.

-1 :x:
javac
0m 54s
/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt
hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 25s
/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
The patch fails to run checkstyle in hadoop-hdfs

-1 :x:
mvnsite
0m 55s
/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

+1 :green_heart:
javadoc
0m 51s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 25s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

-1 :x:
spotbugs
0m 53s
/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

-1 :x:
shadedclient
13m 10s

patch has errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
0m 57s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch failed.

+1 :green_heart:
asflicense
0m 32s

The patch does not generate ASF License warnings.

96m 45s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3926

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux c0da5dde859c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / de7830c0dc4b6ad905bc318ae1f62f0911c8f278

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/4/testReport/

Max. process+thread count
617 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

The failed check has nothing to do with my changes, the last check passed. @jojochuang

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 43s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
32m 35s

trunk passed

+1 :green_heart:
compile
1m 27s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 21s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  1s

trunk passed

+1 :green_heart:
mvnsite
1m 32s

trunk passed

+1 :green_heart:
javadoc
1m  4s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 34s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 13s

trunk passed

+1 :green_heart:
shadedclient
22m 27s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 16s

the patch passed

+1 :green_heart:
compile
1m 18s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 18s

the patch passed

+1 :green_heart:
compile
1m 11s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 11s

the patch passed

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

-0 :warning:
checkstyle
0m 52s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 21 unchanged - 4 fixed = 23 total (was 25)

+1 :green_heart:
mvnsite
1m 19s

the patch passed

+1 :green_heart:
javadoc
0m 51s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 23s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 13s

the patch passed

+1 :green_heart:
shadedclient
22m  5s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
230m 41s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 46s

The patch does not generate ASF License warnings.

329m 33s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3926

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux c0a8a1c18ce2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / a680b82164902f27557a56ee9436d638d771bed6

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/5/testReport/

Max. process+thread count
3737 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
18m  1s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 30s

trunk passed

+1 :green_heart:
compile
1m 28s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
1m 21s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 57s

trunk passed

+1 :green_heart:
mvnsite
1m 27s

trunk passed

+1 :green_heart:
javadoc
1m  2s

trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 27s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 21s

trunk passed

+1 :green_heart:
shadedclient
25m 51s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 20s

the patch passed

+1 :green_heart:
compile
1m 24s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
1m 24s

the patch passed

+1 :green_heart:
compile
1m 14s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
1m 14s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 53s
/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 21 unchanged - 4 fixed = 23 total (was 25)

+1 :green_heart:
mvnsite
1m 19s

the patch passed

+1 :green_heart:
javadoc
0m 52s

the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
1m 23s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
3m 31s

the patch passed

+1 :green_heart:
shadedclient
25m 26s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
330m 46s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 38s

The patch does not generate ASF License warnings.

458m  8s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.mover.TestMover

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3926

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 9c95c06af0e4 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 150e9d95fe10919d5baa5ab5bf22b12a553e3740

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/6/testReport/

Max. process+thread count
1908 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3926/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

This check failure has nothing to do with my commit, please review! @jojochuang

Hi, @jojochuang  ,could you please merge this PR

authors: hadoop-yetus singer-bin


text:
HBASE-24853 The archiveLogFile should be called before decreasing tot…

…alLogSize in AbstractFSWAL.cleanOldLogs

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 29s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 44s
master passed

+1 :green_heart:
checkstyle
1m  6s
master passed

+1 :green_heart:
spotbugs
2m  0s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 22s
the patch passed

+1 :green_heart:
checkstyle
1m  4s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
11m  9s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1.

+1 :green_heart:
spotbugs
2m 10s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 15s
The patch does not generate ASF License warnings.

32m 29s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2240

Optional Tests
dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle

uname
Linux 2e78428d9760 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / ea130249ae

Max. process+thread count
94 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 30s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 45s
master passed

+1 :green_heart:
compile
0m 55s
master passed

+1 :green_heart:
shadedjars
5m 38s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 38s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 26s
the patch passed

+1 :green_heart:
compile
0m 53s
the patch passed

+1 :green_heart:
javac
0m 53s
the patch passed

+1 :green_heart:
shadedjars
5m 34s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 37s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
139m 32s
hbase-server in the patch passed.

163m 40s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2240

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 06f7fd1d1502 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / ea130249ae

Default Java
1.8.0_232

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/testReport/

Max. process+thread count
4631 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 47s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
6m  9s
master passed

+1 :green_heart:
compile
1m 25s
master passed

+1 :green_heart:
shadedjars
7m 56s
branch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 57s
hbase-server in master failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
5m  9s
the patch passed

+1 :green_heart:
compile
1m 26s
the patch passed

+1 :green_heart:
javac
1m 26s
the patch passed

+1 :green_heart:
shadedjars
7m 49s
patch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 51s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
150m 18s
hbase-server in the patch passed.

185m 49s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2240

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux f274b6db6e1d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / ea130249ae

Default Java
2020-01-14

javadoc
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/testReport/

Max. process+thread count
3833 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2240/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

authors: Apache-HBase bsglz

repo: apache/hbase


text:
HBASE-28340 Add trust/key store type to ZK TLS settings handled by HBase

POC for turning all hbase.zookeeper.property.* prefixed properties into system properties.
cc @Apache9

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 24s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 57s
master passed

+1 :green_heart:
compile
0m 15s
master passed

+1 :green_heart:
shadedjars
5m 27s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 15s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 42s
the patch passed

+1 :green_heart:
compile
0m 15s
the patch passed

+1 :green_heart:
javac
0m 15s
the patch passed

+1 :green_heart:
shadedjars
5m 28s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 14s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
2m 14s
hbase-common in the patch passed.

21m  7s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5669

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 813fe7c56a85 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 275d928a7d

Default Java
Eclipse Adoptium-11.0.17+8

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/testReport/

Max. process+thread count
396 (vs. ulimit of 30000)

modules
C: hbase-common U: hbase-common

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 15s
master passed

+1 :green_heart:
compile
0m 16s
master passed

+1 :green_heart:
shadedjars
5m 59s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 15s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 44s
the patch passed

+1 :green_heart:
compile
0m 16s
the patch passed

+1 :green_heart:
javac
0m 16s
the patch passed

+1 :green_heart:
shadedjars
5m 44s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 14s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
1m 59s
hbase-common in the patch passed.

22m 24s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5669

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 5c6b4f5c5816 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 275d928a7d

Default Java
Temurin-1.8.0_352-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/testReport/

Max. process+thread count
364 (vs. ulimit of 30000)

modules
C: hbase-common U: hbase-common

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 37s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 54s
master passed

+1 :green_heart:
compile
0m 49s
master passed

+1 :green_heart:
checkstyle
0m 20s
master passed

+1 :green_heart:
spotless
1m  2s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
0m 46s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 20s
the patch passed

+1 :green_heart:
compile
0m 36s
the patch passed

-0 :warning:
javac
0m 36s
hbase-common generated 1 new + 36 unchanged - 0 fixed = 37 total (was 36)

+1 :green_heart:
checkstyle
0m 16s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
5m 57s
Patch does not cause any errors with Hadoop 3.3.6.

+1 :green_heart:
spotless
0m 45s
patch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
0m 43s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 10s
The patch does not generate ASF License warnings.

26m 43s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5669

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile

uname
Linux 7c22c9df92ff 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 275d928a7d

Default Java
Eclipse Adoptium-11.0.17+8

javac
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-common.txt

Max. process+thread count
79 (vs. ulimit of 30000)

modules
C: hbase-common U: hbase-common

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5669/1/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Seems OK. Is this just a POC or it is nearly the final workable PR?

Seems OK. Is this just a POC or it is nearly the final workable PR?

If the approach is acceptable, it's final.

authors: Apache-HBase Apache9 anmolnar


text:
HIVE-25058: PTF: TimestampValueBoundaryScanner can be optimised during range computation pt2 - isDistanceGreater

What changes were proposed in this pull request?
Introduced faster TimestampWritableV2 handling into isDistanceGreater method.
Why are the changes needed?
Performance improvement.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test included, behavior is the same before/after the patch. Tested on cluster.

cc: @rbalamohan: we missed isDistanceGreater last time, please find flamegraph in jira
I'll include tests later

could please review this @rbalamohan? similar fix to HIVE-24746, unit test included

@rbalamohan , @pgaref : can I get a +1 on this one? the patch is working on customer's cluster right now

repo: apache/hive


text:
Optimize ExecSupport.getNextExchangeId()

While analyzing some JFR profiles from a service utilizing a high volume of async HTTP client requests, I noticed one of the top memory allocations of java.text.DecimalFormatSymbols coming from org.apache.hc.client5.http.impl.ExecSupport#getNextExchangeId, notably due to use of String#format. Given that this seems to be on a hot code path, we can optimize this code path.
Backtrace:
sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(Locale)
java.text.DecimalFormatSymbols.getInstance(Locale)
java.util.Formatter.zero()
java.util.Formatter$FormatSpecifier.getZero(Locale)
java.util.Formatter$FormatSpecifier.localizedMagnitude(StringBuilder, CharSequence, int, Formatter$Flags, int, Locale)
java.util.Formatter$FormatSpecifier.print(long, Locale)
java.util.Formatter$FormatSpecifier.printInteger(Object, Locale)
java.util.Formatter$FormatSpecifier.print(Object, Locale)
java.util.Formatter.format(Locale, String, Object[])
java.util.Formatter.format(String, Object[])
java.lang.String.format(String, Object[])
org.apache.hc.client5.http.impl.ExecSupport.getNextExchangeId()
org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(HttpHost, ClassicHttpRequest, HttpContext)
org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(ClassicHttpRequest, HttpContext)
org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(ClassicHttpRequest)

With a hand rolled a method to perform the same functionality of String.format("ex-%010d", long) we see a roughly 20x speed-up in execution time and ~16x reduction in allocations on a JMH microbenchmark for just this function (see https://github.com/schlosna/httpclient-format-performance ). This PR currently only optimizes the hot exchange ID code path and not the endpoint ID code paths that use similar formatting (see https://github.com/apache/httpcomponents-client/pull/249 )
Benchmarks all run on 16 core Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz.
JDK 11.0.13, OpenJDK 64-Bit Server VM, 11.0.13+8-LTS
Benchmark                                                   Mode  Cnt     Score     Error   Units
FormatBenchmarks.stringFormat                               avgt    5   674.730 ±   7.821   ns/op
FormatBenchmarks.stringFormat:·gc.alloc.rate                avgt    5   852.757 ±   9.814  MB/sec
FormatBenchmarks.stringFormat:·gc.alloc.rate.norm           avgt    5   704.000 ±   0.001    B/op
FormatBenchmarks.stringFormat:·gc.churn.G1_Eden_Space       avgt    5   862.955 ± 171.696  MB/sec
FormatBenchmarks.stringFormat:·gc.churn.G1_Eden_Space.norm  avgt    5   712.471 ± 146.197    B/op
FormatBenchmarks.stringFormat:·gc.churn.G1_Old_Gen          avgt    5     0.001 ±   0.002  MB/sec
FormatBenchmarks.stringFormat:·gc.churn.G1_Old_Gen.norm     avgt    5     0.001 ±   0.002    B/op
FormatBenchmarks.stringFormat:·gc.count                     avgt    5    53.000            counts
FormatBenchmarks.stringFormat:·gc.time                      avgt    5    33.000                ms
FormatBenchmarks.zeroPad                                    avgt    5    38.760 ±   1.082   ns/op
FormatBenchmarks.zeroPad:·gc.alloc.rate                     avgt    5  1855.680 ±  51.458  MB/sec
FormatBenchmarks.zeroPad:·gc.alloc.rate.norm                avgt    5    88.000 ±   0.001    B/op
FormatBenchmarks.zeroPad:·gc.churn.G1_Eden_Space            avgt    5  1859.501 ± 202.606  MB/sec
FormatBenchmarks.zeroPad:·gc.churn.G1_Eden_Space.norm       avgt    5    88.186 ±  10.045    B/op
FormatBenchmarks.zeroPad:·gc.churn.G1_Old_Gen               avgt    5     0.004 ±   0.002  MB/sec
FormatBenchmarks.zeroPad:·gc.churn.G1_Old_Gen.norm          avgt    5    ≈ 10⁻⁴              B/op
FormatBenchmarks.zeroPad:·gc.count                          avgt    5    79.000            counts
FormatBenchmarks.zeroPad:·gc.time                           avgt    5    62.000                ms

JDK 15.0.5, OpenJDK 64-Bit Server VM, 15.0.5+3-MTS
Benchmark                                                       Mode  Cnt     Score     Error   Units
FormatBenchmarks.stringFormat                                   avgt    5  1081.263 ±  50.140   ns/op
FormatBenchmarks.stringFormat:·gc.alloc.rate                    avgt    5  1094.601 ±  50.204  MB/sec
FormatBenchmarks.stringFormat:·gc.alloc.rate.norm               avgt    5  1448.105 ±   0.005    B/op
FormatBenchmarks.stringFormat:·gc.churn.G1_Eden_Space           avgt    5  1093.372 ±   0.707  MB/sec
FormatBenchmarks.stringFormat:·gc.churn.G1_Eden_Space.norm      avgt    5  1446.645 ±  67.105    B/op
FormatBenchmarks.stringFormat:·gc.churn.G1_Survivor_Space       avgt    5     0.004 ±   0.007  MB/sec
FormatBenchmarks.stringFormat:·gc.churn.G1_Survivor_Space.norm  avgt    5     0.006 ±   0.009    B/op
FormatBenchmarks.stringFormat:·gc.count                         avgt    5    55.000            counts
FormatBenchmarks.stringFormat:·gc.time                          avgt    5    51.000                ms
FormatBenchmarks.zeroPad                                        avgt    5    39.612 ±   1.489   ns/op
FormatBenchmarks.zeroPad:·gc.alloc.rate                         avgt    5  1815.969 ±  68.124  MB/sec
FormatBenchmarks.zeroPad:·gc.alloc.rate.norm                    avgt    5    88.006 ±   0.001    B/op
FormatBenchmarks.zeroPad:·gc.churn.G1_Eden_Space                avgt    5  1829.120 ± 209.669  MB/sec
FormatBenchmarks.zeroPad:·gc.churn.G1_Eden_Space.norm           avgt    5    88.641 ±   9.053    B/op
FormatBenchmarks.zeroPad:·gc.churn.G1_Survivor_Space            avgt    5     0.008 ±   0.002  MB/sec
FormatBenchmarks.zeroPad:·gc.churn.G1_Survivor_Space.norm       avgt    5    ≈ 10⁻³              B/op
FormatBenchmarks.zeroPad:·gc.count                              avgt    5    92.000            counts
FormatBenchmarks.zeroPad:·gc.time                               avgt    5    80.000                ms

JDK 17.0.1, OpenJDK 64-Bit Server VM, 17.0.1+12-LTS
Benchmark                                                       Mode  Cnt     Score     Error   Units
FormatBenchmarks.stringFormat                                   avgt    5   871.173 ±  11.990   ns/op
FormatBenchmarks.stringFormat:·gc.alloc.rate                    avgt    5  1313.506 ±  18.053  MB/sec
FormatBenchmarks.stringFormat:·gc.alloc.rate.norm               avgt    5  1400.111 ±   0.011    B/op
FormatBenchmarks.stringFormat:·gc.churn.G1_Eden_Space           avgt    5  1314.209 ± 139.635  MB/sec
FormatBenchmarks.stringFormat:·gc.churn.G1_Eden_Space.norm      avgt    5  1400.917 ± 157.487    B/op
FormatBenchmarks.stringFormat:·gc.churn.G1_Survivor_Space       avgt    5     0.006 ±   0.008  MB/sec
FormatBenchmarks.stringFormat:·gc.churn.G1_Survivor_Space.norm  avgt    5     0.006 ±   0.009    B/op
FormatBenchmarks.stringFormat:·gc.count                         avgt    5    81.000            counts
FormatBenchmarks.stringFormat:·gc.time                          avgt    5    53.000                ms
FormatBenchmarks.zeroPad                                        avgt    5    41.057 ±   0.172   ns/op
FormatBenchmarks.zeroPad:·gc.alloc.rate                         avgt    5  1751.849 ±   7.370  MB/sec
FormatBenchmarks.zeroPad:·gc.alloc.rate.norm                    avgt    5    88.007 ±   0.001    B/op
FormatBenchmarks.zeroPad:·gc.churn.G1_Eden_Space                avgt    5  1752.274 ± 171.164  MB/sec
FormatBenchmarks.zeroPad:·gc.churn.G1_Eden_Space.norm           avgt    5    88.027 ±   8.268    B/op
FormatBenchmarks.zeroPad:·gc.churn.G1_Survivor_Space            avgt    5     0.009 ±   0.005  MB/sec
FormatBenchmarks.zeroPad:·gc.churn.G1_Survivor_Space.norm       avgt    5    ≈ 10⁻³              B/op
FormatBenchmarks.zeroPad:·gc.count                              avgt    5   108.000            counts
FormatBenchmarks.zeroPad:·gc.time                               avgt    5    69.000                ms

@schlosna Awesome! Could you please fix style-check violations, though?

@michael-o I pulled out an IncrementingId class to enable reuse across Exchange, PoolingHttpClientConnectionManager, PoolingAsyncClientConnectionManager.
@carterkozak defer to you on which implementation path you'd prefer given https://github.com/schlosna/httpclient-format-performance/pull/1

hard-coding length 10 and using the switch would be nice, but I defer to your judgement

Updated to use switch

Excellent, thanks @ok2c !

authors: carterkozak ok2c schlosna

repo: apache/httpcomponents-client


text:
[HUDI-2332] Add clustering and compaction in Kafka Connect Sink

What is the purpose of the pull request
This PR adds the functionality of clustering and compaction in Kafka Connect Sink for Hudi.  To achieve that, the compaction and clustering logic are hooked up to Java client.
Brief change log

For compaction, the scheduling and execution are hooked up in HoodieJavaMergeOnReadTable, and interfaces in HoodieCompactionHandler are implemented in HoodieJavaCopyOnWriteTable to support compaction logic. HoodieJavaMergeOnReadTableCompactor is added for Java client as a plugin.   The validation of inflight instants is disabled for Kafka Connect.
For clustering, different clustering plan and execution strategies are added for Java client.  JavaClusteringPlanActionExecutor and JavaExecuteClusteringCommitActionExecutor are added, which are used in HoodieJavaCopyOnWriteTable to hook up clustering logic in Java client.
Note that compaction and clustering should not be executed inline in Hudi Kafka Connect Sink.
When ending the commit in a Hudi Kafka Connect transaction, schedules the compaction and clustering as needed.
Explicitly disables metadata table using "hoodie.metadata.enable": "false" in config-sink.json since the metadata table functionality needs to be validated for Java client.
Updates README.md with the instructions of running async compaction and clustering.

Verify this pull request
I run the end-to-end tests of Hudi Kafka Connect Sink to generate deltacommits in MOR table, which also schedules compaction and clustering.  Then running the async compaction and clustering using the following commands succeeds:
Compaction:
spark-submit  \
--class org.apache.hudi.utilities.HoodieCompactor \
hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.10.0-SNAPSHOT.jar \
--base-path /tmp/hoodie/hudi-test-topic \
--table-name hudi-test-topic \
--schema-file /repo/hudi/docker/demo/config/schema.avsc \
--parallelism 2 \
--spark-memory 1g

Clustering:
spark-submit \
--class org.apache.hudi.utilities.HoodieClusteringJob \
hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.10.0-SNAPSHOT.jar \
--props clusteringjob.properties \
--mode execute \
--base-path /tmp/hoodie/hudi-test-topic  \
--table-name sample_table \
--spark-memory 1g

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@vinothchandar @codope for review

lgmt, will let @codope give the final thumbs-up!

repo: apache/hudi


text:
Add additional S3 FileIO Attributes

Currently we are not supporting quite a few attributes in FileIO for S3.
Most important for me is Path-Style-Access. While on it, I added SSE as well.
Java Attributes: https://iceberg.apache.org/javadoc/1.4.1/constant-values.html
Relevant Opendal docs: https://nightlies.apache.org/opendal/opendal-docs-stable/docs/rust/opendal/services/struct.S3.html#server-side-encryption
What I am desperately missing and couldn't implement easily because its missing in Opendal is:

Everything around soft-deletes (s3.delete.*)
Remote signing ("s3.remote-signing-enabled")

@Xuanwo I am not too familiar with Opendal yet. Do you think supporting those would be a large effort?

@Xuanwo should I open a follow-up issue for:

Everything around soft-deletes (s3.delete.*)
Remote signing ("s3.remote-signing-enabled")
?

Everything around soft-deletes (s3.delete.*)

The Iceberg Java FileIO implements soft deletes using PutObjectTags and a user-configured lifecycle to support it. Deleted objects can still be scanned or listed because S3 does not recognize the concept of soft deletes. While it's possible for OpenDAL to support this, I doubt its usefulness.
I personally think adopting the object version is a better idea for that.

Remote signing ("s3.remote-signing-enabled")

I checked how iceberg-java implemetns remote-signing:
https://github.com/apache/iceberg/blob/f7585932a6d89b04c0d45b5f9dfe6f45483efd0b/aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java#L65-L76
It seems we need to implement this signer in iceberg-rust and pass it as a customized signer to opendal.

Everything around soft-deletes (s3.delete.*)
Remote signing ("s3.remote-signing-enabled")

Feel free to open issues for them. Thanks in advance.

authors: Xuanwo c-thiel

repo: apache/iceberg-rust


text:
[AIRFLOW-180] Fix timeout behavior for sensors

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:

https://issues.apache.org/jira/browse/AIRFLOW-180

In the previous state of the code, datetime.now was compared to
started_at and seconds was pulled out. It turns out that the seconds
attribute of a timedelta has a maximum of 86400 and the rolls up to 1 day.
The unintended consequence is that timeout larger than 86400 are
ignored, with sensors running forever.
To fix this we use the total_seconds method to get at the real
timedelta in seconds.

I should add that I tested this locally to produce the correct results by artificially modifying started_at in flight.

LGTM, but it would be nice to have the test that has been hinted at ;-)

@mistercrunch Addressed your comment.
@bolkedebruin I provided the test, though it required a more finicking than I had hoped for. I feel that there might be a better way with a fancier mock.patch but I am not seeing it right now.

[Current coverage][cc-pull] is 63.81%

Merging [#1547][cc-pull] into [master][cc-base-branch] will increase coverage by <.01%

@@             master      #1547   diff @@
==========================================
  Files           120        120
  Lines          8449       8450     +1
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           5391       5392     +1
  Misses         3058       3058
  Partials          0          0

Powered by Codecov. Last updated by [7c0f837...c2c5a21][cc-compare]
[cc-base-branch]: https://codecov.io/gh/apache/incubator-airflow/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/apache/incubator-airflow/compare/7c0f8373f59b0554d5ba15bb0e5e8669f0830313...c2c5a2110e6b1470ab0c95c8e6716b1b90b765e7
[cc-pull]: https://codecov.io/gh/apache/incubator-airflow/pull/1547?src=pr

@aoen Agreed. How about now?

LGTM

authors: aoen artwr bolkedebruin codecov-io

repo: apache/incubator-airflow

license_source: bigquery


text:
[AIRFLOW-1576] Added region param to Dataproc{*}Operators

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
-https://issues.apache.org/jira/browse/AIRFLOW-1576

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

Currently, the DataProcHadoopOperator, DataProcSparkOperator, DataProcPySparkOperator, and DataProcHiveOperator (found in dataproc_operator.py) default to using a global cluster. We add a region parameter to these operators with support in the corresponding gcp_dataproc_hook.py so that users can specify cluster regions (ex. us-central1, etc.)
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

My PR adds tests to test_dataproc_operator.py to ensure that the region parameter is passed through to the appropriate hook. Also, I've integraton-tested this by running a HadoopOperator with the new region param as follows.
    RUN_CENTRAL1_DATAPROC_HADOOP = DataProcHadoopOperator(
        task_id='run_central1_dataproc_hadoop',
        main_jar=WORDCOUNT_JAR,
        dataproc_cluster='test-cluster-a',
        region='us-central1',
        arguments=WORDCOUNT_ARGS
        )

Here is the successful result in Cloud Console.
Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Hi! My local Travis build seemed fine, and the error on the Travis log for master looks unrelated to this issue. Perhaps this is a master Travis issue?

Looks like you got caught with the docker operator commit I made that had to get reverted. Can you please try rebasing?

Codecov Report

Merging #2625 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2625      +/-   ##
==========================================
+ Coverage   71.71%   71.71%   +<.01%
==========================================
  Files         151      151
  Lines       11736    11736
==========================================
+ Hits         8416     8417       +1
+ Misses       3320     3319       -1

Impacted Files
Coverage Δ

airflow/models.py
87.15% <0%> (+0.04%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c897588...bed8671. Read the comment docs.

authors: cjqian codecov-io criccomini


text:
[AIRFLOW-1559][AIRFLOW-1665] Increase database layer robustness

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title.

https://issues.apache.org/jira/browse/AIRFLOW-1559
https://issues.apache.org/jira/browse/AIRFLOW-1665

Description

[x] Here are some details about my PR:

An overloaded or faulty database can easily take down Airflow. This PR addresses two dimensions of this problem: Airflow must be more diligent in gracefully closing unused connections, and it must reconnect if one of its existing connection fails unexpectedly.
For reviewers, I would suggest to read this PR commit by commit rather than whole diff at once. Each commit also comes with a more lengthy description of its intend.
The PR is partially based on #2767  by @vklogin and #2744 by @dhuang with a few improvements and clean-ups on top.
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

The database layer is used throughout the entire code base, so this improvement leverages the existing test coverage. The newly added behaviour depends on the interlinked behaviour of the main thread, sub- and child processes, and the database. It is therefore hard to test in isolation.
Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Looking good, please let me test this. Cheers

Codecov Report

Merging #2822 into master will increase coverage by <.01%.
The diff coverage is 60.56%.

@@            Coverage Diff             @@
##           master    #2822      +/-   ##
==========================================
+ Coverage   73.83%   73.84%   +<.01%
==========================================
  Files         159      160       +1
  Lines       12076    12114      +38
==========================================
+ Hits         8916     8945      +29
- Misses       3160     3169       +9

Impacted Files
Coverage Δ

airflow/operators/s3_file_transform_operator.py
0% <ø> (ø)
:arrow_up:

airflow/hooks/hive_hooks.py
39.52% <ø> (ø)
:arrow_up:

airflow/hooks/pig_hook.py
0% <ø> (ø)
:arrow_up:

airflow/minihivecluster.py
0% <ø> (ø)
:arrow_up:

airflow/configuration.py
84.03% <ø> (ø)
:arrow_up:

airflow/utils/db.py
86.6% <ø> (+1.33%)
:arrow_up:

airflow/task_runner/base_task_runner.py
73.33% <0%> (ø)
:arrow_up:

airflow/executors/local_executor.py
79% <0%> (ø)
:arrow_up:

airflow/bin/cli.py
53.91% <0%> (-0.08%)
:arrow_down:

airflow/executors/celery_executor.py
84.74% <100%> (ø)
:arrow_up:

... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0211219...94deac3. Read the comment docs.

This is good to see! I will test it out today for the SQL warnings I was trying to address in my PR.

@dhuang did you have a chance to test this? I am happy to iterate on the patch if you are still seeing issues.

Yup, I left it on overnight and didn't see any warnings about connection aborted. LGTM!

@bolkedebruin thanks a lot for merging!
Do you see a chance to get this into 1.9? Or has the release train already progressed too far? (Which I would also totally understand)

1.9.1 is the target. We are at rc for 1.9.0

That will do, thanks

authors: Fokko StephanErb bolkedebruin codecov-io dhuang


text:
[AIRFLOW-2313] Add optional TTL parameters for Dataproc operator

Make sure you have checked all steps below.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-2313

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

This PR adds three extra optional parameters to the DataprocClusterCreateOperator and relevant unit tests as well.
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

New tests:

test_build_cluster_data_with_autoDeleteTime check if autoDeleteTime is added to config
test_build_cluster_data_with_autoDeleteTtl check if autoDeleteTtl is added to config
test_build_cluster_data_with_autoDeleteTime_and_autoDeleteTtl check if autoDeleteTime overrides autoDeleteTtl if both set

Updated tests:

test_build_cluster_data additional checks for new parameters
test_init additional checks for new paramteres

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Code Quality

[x] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

There are some errors unrelated to this PR

Looks like Travis build failed at unrelated test case test_authorized - can someone retry it?
I like this change - we were looking at this option for some time already.

@ebartkus Could you rebase onto master please? The flaky test has been fixed earlier.

Codecov Report

Merging #3217 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3217      +/-   ##
==========================================
+ Coverage   75.87%   75.89%   +0.02%
==========================================
  Files         197      197
  Lines       14706    14706
==========================================
+ Hits        11158    11161       +3
+ Misses       3548     3545       -3

Impacted Files
Coverage Δ

airflow/jobs.py
82.83% <0%> (+0.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 12ab796...d13ec95. Read the comment docs.

Tests pass after rebase.

Great, thanks!

authors: Fokko codecov-io ebartkus exploy


text:
enable local DNS

Summary
enable local DNS resolver
Issues resolved
Fix #1164

@membphis @moonming  I have finish this feature, check please.

please take a look: https://github.com/apache/incubator-apisix/pull/1167#issuecomment-594976329

please take a look: #1167 (comment)

local=on it only valid in openresty. we can not use it in tengine.

please take a look: #1167 (comment)

local=on it only valid in openresty. we can not use it in tengine.

we can patch tengine as we do before.

@qiujiayu I think the following design will be better. What do you think?

When the dns_resolver in config.yaml is not set any IP, then it will read the local resolv.conf. This is equivalent to a hidden switch.
Support IPv6, remove the current restrictions.
Use openresty's local = on feature, and patch tengine.

The third can be TOOD.

@qiujiayu I think the following design will be better. What do you think?

When the dns_resolver in config.yaml is not set any IP, then it will read the local resolv.conf. This is equivalent to a hidden switch.
Support IPv6, remove the current restrictions.
Use openresty's local = on feature, and patch tengine.

The third can be TOOD.

Has been adjusted by the plan of what you said.
I didn't have an IPv6 environment to test.
Because APISIX resolves domain names using "resty.dns.resolver", DNS needs to be configured.

2. I didn't have an IPv6 environment to test.

we can create a new issue about this, fix it later.

@moonming @membphis   has any question for this PR?

I think we should discuss the complete design on the mailing list or issue
then submit the pr，instead of discussing these basic issues over and over
again.
YuanSheng Wang notifications@github.com于2020年3月25日 周三下午8:35写道：

@membphis commented on this pull request.
In conf/config.yaml
https://github.com/apache/incubator-apisix/pull/1217#discussion_r397817423
:

@@ -80,13 +80,13 @@ apisix:

udp:                        # UDP proxy port list
- 9200
- 9211

dns_resolver:                   # default DNS resolver, with disable IPv6 and enable local DNS

dns_resolver:                   # If not set, read from /etc/resolv.conf

I think it's easier to use extra separate fields to decide whether to use
local / etc / resolv.conf.
for example: dns_resolver_local: true
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-apisix/pull/1217#pullrequestreview-381102400,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGJZBK6SJIU36DOI4PUDWP3RJH3AVANCNFSM4LDPL4BQ
.

--
Thanks,
Ming Wen
Twitter: _WenMing

@moonming please take a view, I think we can merge this PR

@moonming 3. Use openresty's local = on feature, and patch tengine.

we can not use local = on, we need to parse the user domain by DNS resolver at Lua land.
For example:  https://github.com/apache/incubator-apisix/blob/master/lua/apisix.lua#L183

got it
Thanks,
Ming Wen
Twitter: _WenMing
YuanSheng Wang notifications@github.com 于2020年3月26日周四 上午9:45写道：

@moonming https://github.com/moonming 3. Use openresty's local = on
feature, and patch tengine.
we can not use local = on, we need to parse the user domain by DNS
resolver at Lua land.
For example:
https://github.com/apache/incubator-apisix/blob/master/lua/apisix.lua#L183
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-apisix/pull/1217#issuecomment-604180006,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGJZBK3IE5QA54GW6DBM76DRJKXSVANCNFSM4LDPL4BQ
.

authors: membphis moonming qiujiayu

repo: apache/incubator-apisix


text:
ARIA-290 Update Makefile for binary dist creation

Updated the Makefile to push required files for release
(LICENSE, NOTICE, DISCLAIMER) into the binary distribution
archive (wheel) automatically.

Can one of the admins verify this patch?

authors: asfgit ran-z

repo: apache/incubator-ariatosca


text:
[BEAM-930] Fix findbugs and re-enable Maven plugin in MongoDbIO and MongoDbGridFSIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[X] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[X] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[X] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[X] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @dhalperi

LGTM, jenkins is green and travis is a flake.

authors: dhalperi jbonofre

repo: apache/incubator-beam


text:
Support server protocol parsing order without changing ProtocolType(options.proto)

One possible plan: define an order for every protocol,
TypeOrder[] = {
{PROTOCOL_BAIDU_STD , 100} ,
{PROTOCOL_STREAMING_RPC, 200},
}

and sort the protocol in handlers according to the number that followed. The interval between two protocol should be large enough to adapt to more protocols in future.

the number for ordering should be passed to RegisterProtocol

source: gharchive/issue

authors: jamesge zyearn

repo: apache/incubator-brpc


text:
[CELEBORN-178][BUG] Default registered flag should be false, not null

What changes were proposed in this pull request?
When restart worker, throw NPE, caused by default registered flag is null before attach value
22/12/29 13:28:06,355 WARN [push-server-6-43] TransportChannelHandler: Exception in connection from /10.169.41.1:29528
java.lang.NullPointerException
        at org.apache.celeborn.service.deploy.worker.PushDataHandler.checkRegistered(PushDataHandler.scala:509)
        at org.apache.celeborn.common.network.server.TransportRequestHandler.checkRegistered(TransportRequestHandler.java:82)
        at org.apache.celeborn.common.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:76)
        at org.apache.celeborn.common.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:117)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at org.apache.celeborn.common.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:74)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?

@waitinfuture

@waitinfuture need backport to branch 0.2

repo: apache/incubator-celeborn


text:
[Fix-4054][UI]Repair the last Sunday of each month

What is the purpose of the pull request
[Feature-#4054][UI]Repair the last Sunday of each month
Brief change log
[Feature-3878][UI]Repair the last Sunday of each month

+1

authors: lgcareer xingchun-chen

repo: apache/incubator-dolphinscheduler


text:
[Bug][Load][Json] Load json format with stream load failed

Describe the bug
if the json data is larger than 4096 bytes, the data will be truncated, making it impossible to obtain the json data completely
To Reproduce
Load a large json file with stream load

@morningman
4096 bytes means a row or a file ?

@morningman
4096 bytes means a row or a file ?

file size.

authors: francisoliverlee morningman

repo: apache/incubator-doris


text:
Avoid SerDe for aggregation query with object pool

For https://github.com/apache/incubator-doris/pull/1610  "avoid serialize and deserialize from Scannode to Aggnode".
For aggregation query, we directly pass agg object from scan node to Agg node and avoid unnecessary serialize and deserialize.

@kangkaisen
I suddenly remembered that you need to modify HLL related functions in be/src/exprs/aggregate_functions.cpp. Otherwise, it will cause BE Crash during the upgrade process.

@kangkaisen
I suddenly remembered that you need to modify HLL related functions in be/src/exprs/aggregate_functions.cpp. Otherwise, it will cause BE Crash during the upgrade process.

Thanks for your reminder. update.

authors: imay kangkaisen


text:
《Dubbo客户端异步接口的实现背景和实践》打不开

博客《Dubbo客户端异步接口的实现背景和实践》打不开，链接：
http://dubbo.apache.org/zh-cn/blog/dubboAync_client.html
报404

#335 has been fixed this issue, the next time the build is complete, the access will be normal.

Now I will close this issue, thank you for your attention.

authors: lixiaojiee tswstarplanet

repo: apache/incubator-dubbo-website


text:
configurators 里面的参数不能动态生效

[ ] I have searched the issues of this repository and believe that this is not a duplicate.
[ ] I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

Dubbo version: 2.5.3
Operating System version: xxx
Java version: xxx

Step to reproduce this issue

configurators 里面的参数不能动态生效

Expected Result
动态生效
Actual Result
不生效
Just put your stack trace here!

可以把你的问题描述得再具体一些，比如给个具体的例子 @Nocturnana

@Nocturnana A more detailed case will help to reproduce the problem.

Please reopen this issue when you can provide more details.

authors: Nocturnana chickenlj luyuanwan

repo: apache/incubator-dubbo


text:
[Dubbo-619] Fix #619 and PR #2956 is not enough

What is the purpose of the change
Fix #619 , and PR #2956 is not enough in the following use case:
In souche.com, we add 'trace' function in dubbo, such as in FailoverClusterInvoker.doInvoke, we get stackTrace from exception to log detail exception info using the following:
org.apache.commons.lang3.exception.ExceptionUtils.getStackTrace(result.getException())
The above sentence will result the same problem with issue #619 .
So, we should solve this in decode stage.
#619 这个问题, PR #2956 貌似已经解决了, 但是我们公司(souche.com)在对dubbo加全链路埋点时, 还是遇到了之前的问题, 因为 #2956 是在InvokerInvocationHandler.invoke()方法的invoker.invoke(invocation)后解决了问题, 但是我们在FailoverClusterInvoker.doInvoke()方法里坐全链路埋点时, 为了能取到异常栈的信息, 用了如下的语句:
org.apache.commons.lang3.exception.ExceptionUtils.getStackTrace(result.getException())
导致了异常栈又出问题了..
所以应该在decode后, 对exception对象做判断才对.
Brief changelog
RpcResult.java
RpcResultTest.java
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[ ] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #3634 into master will decrease coverage by <.01%.
The diff coverage is 85.71%.

@@             Coverage Diff              @@
##             master    #3634      +/-   ##
============================================
- Coverage     63.95%   63.95%   -0.01%
  Complexity       71       71
============================================
  Files           700      700
  Lines         30646    30686      +40
  Branches       4946     4973      +27
============================================
+ Hits          19601    19625      +24
- Misses         8782     8794      +12
- Partials       2263     2267       +4

Impacted Files
Coverage Δ
Complexity Δ

.../src/main/java/org/apache/dubbo/rpc/RpcResult.java
94.11% <85.71%> (-2.76%)
0 <0> (ø)

...e/dubbo/remoting/transport/netty/NettyChannel.java
57.64% <0%> (-4.71%)
0% <0%> (ø)

.../apache/dubbo/remoting/transport/AbstractPeer.java
58.69% <0%> (-4.35%)
0% <0%> (ø)

...dubbo/common/status/support/LoadStatusChecker.java
46.15% <0%> (-3.85%)
0% <0%> (ø)

...he/dubbo/remoting/transport/netty/NettyServer.java
69.64% <0%> (-3.58%)
0% <0%> (ø)

...bo/rpc/cluster/support/FailbackClusterInvoker.java
67.21% <0%> (-3.28%)
0% <0%> (ø)

...ting/exchange/support/header/HeartbeatHandler.java
84.44% <0%> (-1.92%)
0% <0%> (ø)

.../remoting/transport/netty4/NettyClientHandler.java
68.42% <0%> (-1.23%)
0% <0%> (ø)

...e/dubbo/remoting/transport/netty4/NettyClient.java
74.6% <0%> (-1.21%)
0% <0%> (ø)

...he/dubbo/remoting/transport/netty/NettyClient.java
66.1% <0%> (-1.14%)
0% <0%> (ø)

... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f19335...935a67a. Read the comment docs.

@carryxyh 谢谢你的建议.
大家在看这个 PR 前建议先看下 PR #2956 , 那里分析了出现issue #619 的原因.
PR #2956 那样改在社区版里看上去是没有问题的, 但是在某些情况下会有问题:
比如我们公司在FailoverClusterInvoker.doInvoke()里加了全链路监控埋点的代码, 如下所示:

在finally块里, 我们会取详细的exception异常栈(如果有的话), 然后记录在链路日志中, 用的如下这段:
info.setExceptionStr(ExceptionUtils.getStackTrace(result.getException()));
在执行这段代码之前, result.getException()里的stackTrace是null, 这没有问题, 因为服务端本来就抛出了没有异常栈的NullPointerException, (如果这里不懂的话, 建议先看一下 #2956 , 那里有详细的分析).
但是执行完这段代码后, 见如下截图:

stackTrace 就变成了错误的异常栈, 也就是 issue #619 中出现那个误导用户的异常栈, 之所以在PR #2956 后还是有这个问题, 是因为是先执行的上面那块代码, 然后再执行RpcResult.recreate(), 而PR #2956 是在recreate里做的修改, 其实应该是在DecodeableRpcResult 类的handleException()方法里做异常栈的判断和修改:

本 PR 的目的是优化 PR #2956 , 在decode时就实现异常栈的检测和修改, 而不是等到InvokerInvocationHandler.invoke()后, recreate()时再做检测, 否则如果用户做二次开发时, 在诸如FailoverClusterInvoker.doInvoke()里用类似ExceptionUtils.getStackTrace()方法就破坏了异常栈, 所以要在源头就做好检测和修改, 这样更好一点.

@ralf0131 Could you help trigger CI again? This PR failed previously because the junit test failed: #3657

@carryxyh hi, could you tell me why merging is blocked, thank you.

authors: codecov-io zhaixiaoxiang


text:
如何获取yAxis里面的最大值？

Version
4.2.1
Steps to reproduce
// 如何获取yAxis里面的max值(系统生成的，不是用户自己设置的最大值)

What is expected?
有什么api能获取嘛？
What is actually happening?
在getOption里面获取不到系统生成的最大值

目前没有这一接口，只能自己手动计算最大值，然后赋给坐标轴最大值，这样可以确保一致性

@Ovilia 多谢回复
在源码中找到了... 把那个方法copy了出来了...
不过感觉这个可以加个api获取一下...
window.onload = function () {
  document.write('加载完成');

  let extendVal = [];

  let dataExtend = [0, 2006];
  let dataExtend1 = [0, 2000];
  let dataExtend2 = [0, 3381];
  let interval = 0;

  createNumber(dataExtend2)
}

function createNumber (dataExtend) {
  const splitNumber = 4;
  const result = Interval(dataExtend, splitNumber, undefined, undefined);
  const { interval } = result;

  // fixMin
  const minVal = Util.round(Math.floor(dataExtend[0] / interval) * interval);

  // fixMax
  const maxVal = Util.round(Math.ceil(dataExtend[1] / interval) * interval);

  console.log([minVal, maxVal]);
}

function Interval (extent, splitNumber, minInterval, maxInterval) {
  const result = {};
  var span = extent[1] - extent[0];
  var interval = result.interval = Util.nice(span / splitNumber, true);

  if (minInterval != null && interval < minInterval) {
    interval = result.interval = minInterval;
  }
  if (maxInterval != null && interval > maxInterval) {
    interval = result.interval = maxInterval;
  } // Tow more digital for tick.

  result.interval = interval;
  return result;
}

// 辅助 - Interval - 函数
function quantityExponent (val) {
  return Math.floor(Math.log(val) / Math.LN10);
}

class Util {
  static nice (val, round) {
    var exponent = quantityExponent(val);
    var exp10 = Math.pow(10, exponent);
    var f = val / exp10; // 1 <= f < 10

    var nf;

    if (round) {
      if (f < 1.5) {
        nf = 1;
      } else if (f < 2.5) {
        nf = 2;
      } else if (f < 4) {
        nf = 3;
      } else if (f < 7) {
        nf = 5;
      } else {
        nf = 10;
      }
    } else {
      if (f < 1) {
        nf = 1;
      } else if (f < 2) {
        nf = 2;
      } else if (f < 3) {
        nf = 3;
      } else if (f < 5) {
        nf = 5;
      } else {
        nf = 10;
      }
    }

    val = nf * exp10; // Fix 3 * 0.1 === 0.30000000000000004 issue (see IEEE 754).
    // 20 is the uppper bound of toFixed.

    console.log(val, '  -> val');
    console.log(exponent, '  -> exponent');
    console.log(exp10, '  -> exp10');
    console.log(f, '  -> f');
    console.log(nf, '  -> nf');
    console.log(val, '  -> val');

    return exponent >= -20 ? +val.toFixed(exponent < 0 ? -exponent : 0) : val;
  }

  static round(x, precision, returnStr) {
    if (precision == null) {
      precision = 10;
    }

    precision = Math.min(Math.max(0, precision), 20);
    x = (+x).toFixed(precision);
    return returnStr ? x : +x;
  }
}

authors: Ovilia djkloop

repo: apache/incubator-echarts


text:
Crashing Safari and reload page

Version
4.6.0
Reproduction link
https://jsfiddle.net/qipkiller/amxhrcs9
Steps to reproduce
Open the link and wait for updating the new data
What is expected?
Normal work and update new data to chart
What is actually happening?
Safari version 12.1 crushing and try to reload the page.

I can not reproduce it yet.
How long will be waiting for the reload in Safari?

The bug is triggered about 70%, I use browserstack.com fo test it, OS: Mojave, Safari: 12.1.
Can I record a video?

It is probably a Safari issue:
When both using gradient color and shadow to in HTML Canvas rendering at the first time, the memory will increase to over 1GB instantly, which might crash the Safari.
I'm not sure how to avoid this defect yet. Need to investigate for a while.
But, @qipkiller , to temporarily avoid this issue, please do not use gradient color and shadow at the same time. That is, in your demo, do not use echarts.graphic.LinearGradient and shadowColor/shadowBlue at the same time.
Other info:
I have simplified it to a minimal case for this issue:
<!DOCTYPE html>

        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1" />

            1. Open the page in Safari.

            2. Open Safari web inspector - Timeline panel. Start record. Watch "Memory" timeline.

            3. Click "Add Data" button. Check whether memory timeline appears a peak over 1G MB.

        <button id="addData">Add Data</button>

And the record of the memory timeline:

Notice picture above that the memory is over 1GB and the single JS task takes over 2 second.

The reason of this issue:
When the animation is enabled (by default) and add data item to a line chart, the coordinate of the new added data item will be calculated based on the old coordinate system, and the result coordinate will be used as the begin point of the animation. See https://github.com/apache/incubator-echarts/blob/4.6.0/src/chart/line/lineAnimationDiff.js#L121
See the demo code I give above, when there is only one data in the chart, the Y extent might be only be [0, 1]. When add a new data item 3547546 to the chart, the coordinate based on the previous Y extent [0, 1] is -993312512, which is a huge number. Even so, in most cases it works normally. But if we use both createLinearGradient and shadow in Safari with that huge size, the browser have a chance to crash.
@pissang Do you have any idea about it? Limit the result of the "previous coord calculation"?

@100pah Perhaps We can disable animation automatically if the diff between calculated coordinate of previous data and current data is too large.

Temporarily solved the problem simplification echarts.graphic.LinearGradient to color : 'rgba()', anddisable shadow for this, Thx

authors: 100pah pissang qipkiller


text:
fix:issues-11401 TypeError Cannot read property 'getAttribute' of null

https://github.com/apache/incubator-echarts/issues/11401

I think it doesn't do the fix.


text:
design(#12933): tweak the z value of axisPointer component to make it be displayed under the symbol by default.

Brief Information
This pull request is in the type of:

[ ] bug fixing
[ ] new feature
[x] others

What does this PR do?

Design Changes. Closes #12933
Fixed issues

#12933

Details
Before: What was the problem?

The axisPointer component displays above the symbol, which may destroy the integrity of the symbol.

After: How is it fixed in this PR?

Set the z value of axisPointer component as a lower value 0 by default.

Usage
Are there any API changes?

[ ] The API has been changed.

Related test cases or examples to use the new APIs
Any example with the option of axisPointer.

I'm thinking if the axisPointer line will be covered by the rectangle from splitArea if it's z is 0.

I'm thinking if the axisPointer line will be covered by the rectangle from splitArea if it's z is 0.

I've made a test in test/axis-splitArea.html and it seems that the axisPointer won't covered by splitArea.

Found it may be covered by areaStyle.

LGTM, @100pah Can you think of other cases may be broken?

authors: pissang plainheart


text:
Expose clientmanager/admin APIs as RESTful APIs

Enhancement Request

Is your enhancement request related to a problem? Please describe
Yes,  the clientmanager/admin APIs which are exposed from eventmesh-runtime module are not following the best practice of RESTful APIs . The problems could be seen by API urls such as "showClient", "rejectAllClient" ,"redirectClientByPath"

Describe the solution you'd like
Add a new module such as eventmesh-rest which optimizes the existing admin/clientmanager APIs and expose them as RESTful APIs.

Describe alternatives you've considered
I will follow up and provide more details later. However, I would like to report this to the community and initiate a discussion here. :) . Please provide your feedbacks.

Additional context

good suggestions, just go ahead.

#346 #349 are related to this.

@yzhao244 I think you can submit this pr first.

@yzhao244 Is there any updates here?

@qqeasonchen Yes,  Basically, my idea is to move httpserver context creation, handlers along with its dependencies such as Utils classes to eventmesh-rest module.  Inside the module, a new utility class called "RequestMapping" maps RESTful RequestUrl to the corresponding handler method and extracts values from path variables. As my sample handler class code indicates, any request invokes REST API "/schemaregistry/subjects/{subject}/versions" eventually calls createSchemaHandler method. Additionally, subject value is extracted from the path variable {subject} and passed to handler method as well.

HttpServer is still created and run inside eventmensh-runtime. Only the server.createContext part and handlers move to eventmesh-rest module.

@yzhao244 thanks, you can see the new pr #498 , maybe you can give some updates as well, after all, you are the first contributor of this feature :), i hope you can be continue on this, thanks again.

Hi, @yzhao244. I would like to coorperate with you on Open Schema related coding part. Any updates here?

@JunjieChou Great suggestions actually :) . Btw, I am planning contributing eventmesh-openschema-rest module along with a readme file and maybe some skeleton code under the main eventmesh-openschema-plugin module
for example, eventmesh-openschema-plugin/eventmesh-openschema-rest/readme.md which contains my proposed details of EventMesh Schema Registry REST APIs such as REST URL, payload and examples.
eventmesh-openschema-plugin/eventmesh-openschema-rest/.../controller/SchemaRegistryController.java which only contains some skeleton codes.
I will notify you as soon as I have a PR ready. :)

@yzhao244 Hi, good to hear that you are going to start a readme file.
Actually, I have started the coding of the separated RESTful Schema Registry.
Inspired by what you have coded, I created a new module called OpenSchemaRegistry under eventmesh-openschema-plugin as a separate service based on Spring-boot and spring-data-jpa.
For now, I have accomplished the /schemas  controller. Let me pull a request later. I am newly to these frameworks and welcome reviews and suggestions.

@yzhao244 I have created a pr to your forked repository. Hope for reviewing 😊 !

@JunjieChou Sure, I can definitely review. :) . Btw, did you have a chance to discuss with the community of using SpringBoot and Spring JPA for Schema Registry in EventMesh project. Maybe we can add Eason and other members to this issue to have an open discussion. :)
My initial thought was exposing Schema Registry REST APIs by leveraging the current EventMesh's ClientManagerController's design of using HttpServer.createContext(""). However, modifying the url and handlers to make the APIs more RESTful without using another Framework. Basically, Schema Registry and Runtime run on the same server. Therefore, I wasn't thinking of adding another framework to EventMesh.
Anyways, I can share some implementations with the community and it could be just my personal thought. lol
@qqeasonchen @ruanwenjun @jinrongluo WDYT

@yzhao244 I agree with using SpringBoot, since we may need to add a new module such like admin-server, it works as a control plane and provide some interface to manage the eventmesh-runtime module and other configuration(e.g. schema).
SpringBoot can help us a lot, and in runtime-module, I think we may don't need to import spring dependency.

I think eventmesh-admin moudle is great.

@yzhao244 @qqeasonchen The topology may look like below, this is a simple demo, the eventmesh-runtime is decentralization and ligntwight. For third-part storage, we may only need to rely on registry center,

@ruanwenjun @qqeasonchen @JunjieChou I agree with eventmesh-admin module, sounds a very good idea. eventmesh-admin module can in the future becomes a management layer which exposes Admin or Schema REST APIs to UI console for end-user to do managements.
I can rework my existing PR #513 to create eventmesh-admin module. Just double confirm, I will name this new module as "eventmesh-admin" instead of "eventmesh-admim-plugin", am I right? :)

@ruanwenjun Based on your topology image above, do you think schema registry should be a child module of eventmesh-admin?
For example
eventmesh-admin/eventmesh-admin-schemaregistry
eventmesh-admin/eventmesh-admin-clientmanager

@yzhao244 I think we can leave the plugins out for now, we can just add an admin module.

@yzhao244 I think we can leave the plugins out for now, we can just add an admin module.

Sounds good. :)

authors: JunjieChou qqeasonchen ruanwenjun yzhao244

repo: apache/incubator-eventmesh