text:
HBASE-22810 Initialize an separate ThreadPoolExecutor for taking/restoring snapshot

…oring snapshot

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
38
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

+1
mvninstall
337
master passed

+1
compile
54
master passed

+1
checkstyle
80
master passed

+1
shadedjars
274
branch has no errors when building our shaded downstream artifacts.

+1
javadoc
37
master passed

0
spotbugs
254
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
250
master passed

_ Patch Compile Tests _

+1
mvninstall
304
the patch passed

+1
compile
56
the patch passed

+1
javac
56
the patch passed

-1
checkstyle
77
hbase-server: The patch generated 4 new + 167 unchanged - 2 fixed = 171 total (was 169)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
274
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
946
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1
javadoc
33
the patch passed

+1
findbugs
255
the patch passed

_ Other Tests _

+1
unit
6636
hbase-server in the patch passed.

+1
asflicense
27
The patch does not generate ASF License warnings.

10077

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/486

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 4d977315dc04 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh

git revision
master / 8c1edb3bba

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/artifact/out/diff-checkstyle-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/testReport/

Max. process+thread count
5094 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/1/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
46
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

+1
mvninstall
331
master passed

+1
compile
52
master passed

+1
checkstyle
75
master passed

+1
shadedjars
275
branch has no errors when building our shaded downstream artifacts.

+1
javadoc
36
master passed

0
spotbugs
256
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
253
master passed

_ Patch Compile Tests _

+1
mvninstall
301
the patch passed

+1
compile
53
the patch passed

+1
javac
53
the patch passed

-1
checkstyle
75
hbase-server: The patch generated 4 new + 167 unchanged - 2 fixed = 171 total (was 169)

+1
whitespace
1
The patch has no whitespace issues.

+1
shadedjars
265
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
970
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1
javadoc
33
the patch passed

+1
findbugs
249
the patch passed

_ Other Tests _

-1
unit
6854
hbase-server in the patch failed.

+1
asflicense
32
The patch does not generate ASF License warnings.

10271

Reason
Tests

Failed junit tests
hadoop.hbase.master.assignment.TestOpenRegionProcedureHang

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/486

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 499a55e83057 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh

git revision
master / e69af5affe

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/artifact/out/diff-checkstyle-hbase-server.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/testReport/

Max. process+thread count
4524 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/2/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
37
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

+1
mvninstall
323
master passed

+1
compile
53
master passed

+1
checkstyle
74
master passed

+1
shadedjars
261
branch has no errors when building our shaded downstream artifacts.

+1
javadoc
38
master passed

0
spotbugs
207
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
206
master passed

_ Patch Compile Tests _

+1
mvninstall
297
the patch passed

+1
compile
52
the patch passed

+1
javac
52
the patch passed

-1
checkstyle
72
hbase-server: The patch generated 4 new + 167 unchanged - 2 fixed = 171 total (was 169)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
263
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
905
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1
javadoc
32
the patch passed

+1
findbugs
243
the patch passed

_ Other Tests _

+1
unit
6706
hbase-server in the patch passed.

+1
asflicense
24
The patch does not generate ASF License warnings.

9937

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/486

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux aec1cc3557ba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh

git revision
master / 27ed2ac071

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/artifact/out/diff-checkstyle-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/testReport/

Max. process+thread count
4995 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/3/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
84
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

0
mvndep
40
Maven dependency ordering for branch

+1
mvninstall
455
master passed

+1
compile
96
master passed

+1
checkstyle
136
master passed

+1
shadedjars
376
branch has no errors when building our shaded downstream artifacts.

+1
javadoc
64
master passed

0
spotbugs
308
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
366
master passed

_ Patch Compile Tests _

0
mvndep
17
Maven dependency ordering for patch

+1
mvninstall
413
the patch passed

+1
compile
100
the patch passed

+1
javac
100
the patch passed

-1
checkstyle
104
hbase-server: The patch generated 3 new + 162 unchanged - 7 fixed = 165 total (was 169)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
371
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
1282
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1
javadoc
65
the patch passed

+1
findbugs
393
the patch passed

_ Other Tests _

+1
unit
204
hbase-common in the patch passed.

-1
unit
13817
hbase-server in the patch failed.

+1
asflicense
66
The patch does not generate ASF License warnings.

18990

Reason
Tests

Failed junit tests
hadoop.hbase.util.TestFromClientSide3WoUnsafe

hadoop.hbase.client.TestFromClientSide3

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/486

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux cd6837ae2514 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh

git revision
master / 27ed2ac071

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/artifact/out/diff-checkstyle-hbase-server.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/testReport/

Max. process+thread count
4725 (vs. ulimit of 10000)

modules
C: hbase-common hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/4/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
38
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

0
mvndep
39
Maven dependency ordering for branch

+1
mvninstall
340
master passed

+1
compile
74
master passed

+1
checkstyle
93
master passed

+1
shadedjars
263
branch has no errors when building our shaded downstream artifacts.

+1
javadoc
54
master passed

0
spotbugs
257
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
305
master passed

_ Patch Compile Tests _

0
mvndep
14
Maven dependency ordering for patch

+1
mvninstall
286
the patch passed

+1
compile
71
the patch passed

+1
javac
71
the patch passed

-1
checkstyle
69
hbase-server: The patch generated 3 new + 162 unchanged - 7 fixed = 165 total (was 169)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
264
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
893
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1
javadoc
55
the patch passed

+1
findbugs
310
the patch passed

_ Other Tests _

+1
unit
174
hbase-common in the patch passed.

-1
unit
6870
hbase-server in the patch failed.

+1
asflicense
38
The patch does not generate ASF License warnings.

10643

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/486

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 2b37abec7345 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh

git revision
master / 53db390f60

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/artifact/out/diff-checkstyle-hbase-server.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/testReport/

Max. process+thread count
4656 (vs. ulimit of 10000)

modules
C: hbase-common hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/5/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

But please fix the checkstyle issues before merging...

The checkstyle is not an issue I think,  the '(' is preceded by a whitespace because we want to make the code align as before :
  /**
   * Messages originating from Client to Master.

   * C_M_CREATE_TABLE

   * Client asking Master to create a table.
   */
  C_M_CREATE_TABLE          (47, ExecutorType.MASTER_TABLE_OPERATIONS),
  /**
   * Messages originating from Client to Master.

   * C_M_SNAPSHOT_TABLE

   * Client asking Master to snapshot an offline table.
   */
  C_M_SNAPSHOT_TABLE        (48, ExecutorType.MASTER_SNAPSHOT_OPERATIONS),
  /**
   * Messages originating from Client to Master.

   * C_M_RESTORE_SNAPSHOT

   * Client asking Master to restore a snapshot.
   */
  C_M_RESTORE_SNAPSHOT      (49, ExecutorType.MASTER_SNAPSHOT_OPERATIONS),

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
58
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ master Compile Tests _

0
mvndep
39
Maven dependency ordering for branch

+1
mvninstall
426
master passed

+1
compile
100
master passed

+1
checkstyle
139
master passed

+1
shadedjars
343
branch has no errors when building our shaded downstream artifacts.

+1
javadoc
70
master passed

0
spotbugs
327
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
390
master passed

_ Patch Compile Tests _

0
mvndep
18
Maven dependency ordering for patch

+1
mvninstall
405
the patch passed

+1
compile
107
the patch passed

+1
javac
107
the patch passed

-1
checkstyle
104
hbase-server: The patch generated 3 new + 162 unchanged - 7 fixed = 165 total (was 169)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
359
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
1294
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1
javadoc
70
the patch passed

+1
findbugs
432
the patch passed

_ Other Tests _

+1
unit
217
hbase-common in the patch passed.

-1
unit
19484
hbase-server in the patch failed.

+1
asflicense
58
The patch does not generate ASF License warnings.

24660

Reason
Tests

Failed junit tests
hadoop.hbase.client.TestFromClientSide

hadoop.hbase.replication.TestReplicationDisableInactivePeer

hadoop.hbase.snapshot.TestFlushSnapshotFromClient

hadoop.hbase.client.TestFromClientSide3

hadoop.hbase.replication.TestReplicationKillSlaveRSWithSeparateOldWALs

hadoop.hbase.replication.TestReplicationSmallTests

hadoop.hbase.master.TestAssignmentManagerMetrics

hadoop.hbase.replication.TestReplicationSmallTestsSync

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/486

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 8e8f7bc61373 4.4.0-154-generic #181-Ubuntu SMP Tue Jun 25 05:29:03 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-486/out/precommit/personality/provided.sh

git revision
master / d9d5f69fc6

Default Java
1.8.0_181

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/artifact/out/diff-checkstyle-hbase-server.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/testReport/

Max. process+thread count
4829 (vs. ulimit of 10000)

modules
C: hbase-common hbase-server U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-486/6/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

authors: Apache-HBase Apache9 openinx

repo: apache/hbase


text:
HBASE-27309 Add major compact table or region operation on m…

…aster web table page

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 15s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2 Compile Tests _

-1 :x:
mvninstall
0m 19s
root in branch-2 failed.

+1 :green_heart:
spotless
0m 51s
branch has no errors when running spotless:check.

_ Patch Compile Tests _

-1 :x:
mvninstall
0m  6s
root in the patch failed.

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
spotless
0m 40s
patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 11s
The patch does not generate ASF License warnings.

4m 41s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4870

JIRA Issue
HBASE-27309

Optional Tests
dupname asflicense javac spotless

uname
Linux ef68985a85d6 5.4.0-1083-aws #90~18.04.1-Ubuntu SMP Fri Aug 5 08:12:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / ecf3debd42

Default Java
Eclipse Adoptium-11.0.17+8

mvninstall
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/artifact/yetus-general-check/output/branch-mvninstall-root.txt

mvninstall
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt

Max. process+thread count
44 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 51s
Docker mode activated.

-0 :warning:
yetus
0m  5s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
2m 32s
branch-2 passed

+1 :green_heart:
javadoc
0m 24s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m  4s
the patch passed

+1 :green_heart:
javadoc
0m 22s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
196m 45s
hbase-server in the patch passed.

207m 33s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4870

JIRA Issue
HBASE-27309

Optional Tests
javac javadoc unit

uname
Linux 6f9a8c17aa1e 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / ecf3debd42

Default Java
Temurin-1.8.0_352-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/testReport/

Max. process+thread count
2577 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 25s
Docker mode activated.

-0 :warning:
yetus
0m 10s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
2m 46s
branch-2 passed

+1 :green_heart:
javadoc
0m 28s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 41s
the patch passed

+1 :green_heart:
javadoc
0m 26s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
229m 35s
hbase-server in the patch passed.

242m 58s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4870

JIRA Issue
HBASE-27309

Optional Tests
javac javadoc unit

uname
Linux 0cf2918f4d9d 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / ecf3debd42

Default Java
Eclipse Adoptium-11.0.17+8

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/testReport/

Max. process+thread count
2355 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
3m 32s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
3m 15s
branch-2 passed

+1 :green_heart:
spotless
0m 44s
branch has no errors when running spotless:check.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 39s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
spotless
0m 41s
patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m  9s
The patch does not generate ASF License warnings.

12m 27s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4870

Optional Tests
dupname asflicense javac spotless

uname
Linux 9338b06d7ebd 5.4.0-1083-aws #90~18.04.1-Ubuntu SMP Fri Aug 5 08:12:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / e007e23625

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
60 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 44s
Docker mode activated.

-0 :warning:
yetus
0m  5s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
2m 23s
branch-2 passed

+1 :green_heart:
javadoc
0m 24s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m  6s
the patch passed

+1 :green_heart:
javadoc
0m 20s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
191m  2s
hbase-server in the patch passed.

202m 28s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4870

Optional Tests
javac javadoc unit

uname
Linux f340ffaa5ce6 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / e007e23625

Default Java
Temurin-1.8.0_352-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/testReport/

Max. process+thread count
2804 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
6m  7s
Docker mode activated.

-0 :warning:
yetus
0m  5s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
2m 52s
branch-2 passed

+1 :green_heart:
javadoc
0m 27s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 36s
the patch passed

+1 :green_heart:
javadoc
0m 26s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
207m 46s
hbase-server in the patch passed.

224m 48s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4870

Optional Tests
javac javadoc unit

uname
Linux 3cc9890179e5 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / e007e23625

Default Java
Eclipse Adoptium-11.0.17+8

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/testReport/

Max. process+thread count
2443 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4870/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

source: gharchive/pull-request

authors: Apache-HBase SiCheng-Zheng

license_type: permissive

license_source: github-api


text:
HBASE-22845 Revert MetaTableAccessor#makePutFromTableState access to …

…public
HBCK2 is dependent on it

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
62
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

-0
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ master Compile Tests _

+1
mvninstall
387
master passed

+1
compile
23
master passed

+1
checkstyle
30
master passed

+1
shadedjars
269
branch has no errors when building our shaded downstream artifacts.

+1
javadoc
24
master passed

0
spotbugs
72
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
69
master passed

_ Patch Compile Tests _

+1
mvninstall
294
the patch passed

+1
compile
24
the patch passed

+1
javac
24
the patch passed

+1
checkstyle
28
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
263
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
955
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

+1
javadoc
21
the patch passed

+1
findbugs
77
the patch passed

_ Other Tests _

+1
unit
108
hbase-client in the patch passed.

+1
asflicense
11
The patch does not generate ASF License warnings.

3044

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-489/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/489

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux cb8f6c7cb6e6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-489/out/precommit/personality/provided.sh

git revision
master / 8c1edb3bba

Default Java
1.8.0_181

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-489/1/testReport/

Max. process+thread count
291 (vs. ulimit of 10000)

modules
C: hbase-client U: hbase-client

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-489/1/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

authors: Apache-HBase jatsakthi


text:
HIVE-26656:Remove hsqldb dependency in hive due to CVE-2022-41853

What changes were proposed in this pull request?
Remove hsqldb dependency in hive
Why are the changes needed?
fix cve
Does this PR introduce any user-facing change?
no
How was this patch tested?
manual

@saihemanth-cloudera please see this

repo: apache/hive


text:
HIVE-26930: Support for increased retention of Notification Logs and Change Manager entries

What changes does this PR contain?
To support the Planned/Unplanned Failover, we need the capability to increase the retention period for both the Notification Logs and Change Manager entries until the successful reverse replication is done (i.e. the Optimized Bootstrap). A database-level property 'repl.db.under.failover.sync.pending' was introduced to signify this state. This PR contains the changes for,

selective deletion of notification events that are not relevant to the database(s) in failover
skipping the CM clearer thread execution until the time the Optimized Bootstrap is not done

Why this change is needed?
The change is needed to make the Optimized Bootstrap and Point-in-time consistency possible. If the relevant Notification logs and Change Manager entries are not retained, we can't perform the Optimized Bootstrap.
Were the changes tested?
I have included relevant unit tests in this PR, and will also perform manual verification after deploying the changes on a cluster.
Does this PR introduce any user-facing change?
No

Closing this PR, will raise another.


text:
[Task]: update MSSQL JDBC Driver

What needs to happen?
Update version of MSSQL jdbc driver included in Hop installation.
https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc
Issue Priority
Priority: 2
Issue Component
Component: Database

already done with #2445

source: gharchive/issue

repo: apache/hop


text:
5.3.x - HTTPCLIENT-2314

Make sure an UnknownHostException is thrown, even if a custom DnsResolver implementation is used.
Fixes the issue of HTTPCLIENT-2314

@ok2c Thanks for the review. Done

repo: apache/httpcomponents-client


text:
[SUPPORT] URI too long error

Describe the problem you faced
I'm using Spark3.5 + Hudi0.15.0 for partitioned table, when I choose req_date and req_hour for partition column name, I will get this error, but task would be executed successfully finally;
when I choose date and hour for partition column name, error disappeared.
Expected behavior
We should get no errors when we just make partition column names a bit longer.
Environment Description

Hudi version : 0.15.0

Spark version : 3.5.0

Hive version : NA

Hadoop version : 3.3.6

Storage (HDFS/S3/GCS..) : GCS

Running on Docker? (yes/no) : no

Stacktrace
2024-06-13 13:21:13 ERROR PriorityBasedFileSystemView:129 - Got error running preferred function. Trying secondary
org.apache.hudi.exception.HoodieRemoteException: URI Too Long
	at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.loadPartitions(RemoteHoodieTableFileSystemView.java:447) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.loadPartitions(RemoteHoodieTableFileSystemView.java:465) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.lambda$loadPartitions$6e5c444d$1(PriorityBasedFileSystemView.java:187) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:69) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.loadPartitions(PriorityBasedFileSystemView.java:185) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.table.action.clean.CleanPlanActionExecutor.requestClean(CleanPlanActionExecutor.java:133) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.table.action.clean.CleanPlanActionExecutor.requestClean(CleanPlanActionExecutor.java:174) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.table.action.clean.CleanPlanActionExecutor.execute(CleanPlanActionExecutor.java:200) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.scheduleCleaning(HoodieSparkCopyOnWriteTable.java:212) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieTableServiceClient.scheduleTableServiceInternal(BaseHoodieTableServiceClient.java:647) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieTableServiceClient.clean(BaseHoodieTableServiceClient.java:746) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:843) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:816) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:847) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:581) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.mayBeCleanAndArchive(BaseHoodieWriteClient.java:560) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:251) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:108) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.HoodieSparkSqlWriterInternal.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1082) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:508) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:125) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:168) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:107) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:201) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:108) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:107) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:473) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76) ~[spark-sql-api_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:473) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:449) ~[spark-catalyst_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:98) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:85) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:83) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142) ~[spark-sql_2.12-3.5.0.jar:0.15.0]
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:859) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:388) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:361) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:240) ~[spark-sql_2.12-3.5.0.jar:3.5.0]
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) [scala-library-2.12.18.jar:?]
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) [scala-library-2.12.18.jar:?]
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) [scala-library-2.12.18.jar:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) [spark-core_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1032) [spark-core_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194) [spark-core_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217) [spark-core_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91) [spark-core_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1124) [spark-core_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1133) [spark-core_2.12-3.5.0.jar:3.5.0]
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark-core_2.12-3.5.0.jar:3.5.0]
Caused by: org.apache.hudi.org.apache.http.client.HttpResponseException: URI Too Long
	at org.apache.hudi.org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.org.apache.http.client.fluent.Response.handleResponse(Response.java:90) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.org.apache.http.client.fluent.Response.returnContent(Response.java:97) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:189) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.loadPartitions(RemoteHoodieTableFileSystemView.java:445) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
	... 71 more

@michael1991 Thanks for raising this. Can you help me to reproduce this issue. I tried below but it was working fine for me.
fake = Faker()
data = [{"ID": fake.uuid4(), "EventTime": "2023-03-04 14:44:42.046661",
         "FullName": fake.name(), "Address": fake.address(),
         "CompanyName": fake.company(), "JobTitle": fake.job(),
         "EmailAddress": fake.email(), "PhoneNumber": fake.phone_number(),
         "RandomText": fake.sentence(), "CityNameDummyBigFieldName": fake.city(),  "ts":"1",
         "StateNameDummyBigFieldName": fake.state(), "Country": fake.country()} for _ in range(1000)]
pandas_df = pd.DataFrame(data)

hoodie_properties = {
    'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
    'hoodie.datasource.write.operation': 'upsert',
    'hoodie.datasource.write.hive_style_partitioning': 'true',
    'hoodie.datasource.write.recordkey.field': 'ID',
    'hoodie.datasource.write.partitionpath.field': 'StateNameDummyBigFieldName,CityNameDummyBigFieldName',
    'hoodie.table.name' : 'test'

}
spark.sparkContext.setLogLevel("WARN")
df = spark.createDataFrame(pandas_df)
df.write.format("hudi").options(**hoodie_properties).mode("overwrite").save(PATH)

for i in range(1, 50):
    df.write.format("hudi").options(**hoodie_properties).mode("append").save(PATH)

Hi @ad1happy2go , glad to see you again ~
Can you try column name with underscore, i'm not sure if enable urlencode for partition and partition column name with underscore could make this happen.

@michael1991
How many number of partitions in the table? Is it possible to get the URI? I was not able to reproduce this though.

@ad1happy2go  Partitions are hours, for example, gs://bucket/tables/hudi/r_date=2024-06-17/r_hour=00. But problem only occurs on two partitions and underscore, we are using one partition column like yyyyMMddHH and it's going on well. Not sure the exact cause.

Can you try reproducing this issue with the sample code. @michael1991 , That will help us to triage it better

I had the same problem when I deleted the data of some partitions ，The data uses secondary partition, event_date=2024-12-03/event_name=active ,about 200 partitions every day
Environment Description
Hudi version : 0.15.0
Spark version : 3.5.1
Hive version : NA
Hadoop version : 3.3.6
Storage (HDFS/S3/GCS..) : GCS
24/12/06 06:22:39 WARN HttpParser: URI is too large >8192
24/12/06 06:22:39 ERROR PriorityBasedFileSystemView: Got error running preferred function. Trying secondary
org.apache.hudi.exception.HoodieRemoteException: URI Too Long
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.loadPartitions(RemoteHoodieTableFileSystemView.java:447) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.loadPartitions(RemoteHoodieTableFileSystemView.java:465) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.lambda$loadPartitions$6e5c444d$1(PriorityBasedFileSystemView.java:187) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:69) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.loadPartitions(PriorityBasedFileSystemView.java:185) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.clean.CleanPlanActionExecutor.requestClean(CleanPlanActionExecutor.java:133) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.clean.CleanPlanActionExecutor.requestClean(CleanPlanActionExecutor.java:174) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.action.clean.CleanPlanActionExecutor.execute(CleanPlanActionExecutor.java:200) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.scheduleCleaning(HoodieSparkCopyOnWriteTable.java:212) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieTableServiceClient.scheduleTableServiceInternal(BaseHoodieTableServiceClient.java:647) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieTableServiceClient.clean(BaseHoodieTableServiceClient.java:746) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:843) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:816) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieWriteClient.clean(BaseHoodieWriteClient.java:847) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:581) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieWriteClient.mayBeCleanAndArchive(BaseHoodieWriteClient.java:560) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:251) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:108) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.HoodieSparkSqlWriterInternal.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1082) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:508) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:125) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:168) ~[hudi-spark3.5-bundle_2.12-0.15.0.jar:0.15.0]
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:48) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:107) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:201) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:108) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:107) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98) ~[spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:473) ~[spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76) [spark-sql-api_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:473) [spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:32) [spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267) [spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263) [spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32) [spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:32) [spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:449) [spark-catalyst_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:98) [spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:85) [spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:83) [spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142) [spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:859) [spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:388) [spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:361) [spark-sql_2.12-3.5.1.jar:3.5.1]
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:240) [spark-sql_2.12-3.5.1.jar:3.5.1]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) [py4j-0.10.9.7.jar:?]
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) [py4j-0.10.9.7.jar:?]
at py4j.Gateway.invoke(Gateway.java:282) [py4j-0.10.9.7.jar:?]
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) [py4j-0.10.9.7.jar:?]
at py4j.commands.CallCommand.execute(CallCommand.java:79) [py4j-0.10.9.7.jar:?]
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) [py4j-0.10.9.7.jar:?]
at py4j.ClientServerConnection.run(ClientServerConnection.java:106) [py4j-0.10.9.7.jar:?]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]

repo: apache/hudi


text:
[HUDI-7576] Improve efficiency of getRelativePartitionPath, reduce computation of partitionPath in AbstractTableFileSystemView

Change Logs

Improve the efficiency of getRelativePartitionPath by reducing the number of operations on the path object that are required to get the final result
Reduce the number of times a partitionPath is computed by supplying a partition path argument where possible in the AbstractFileSystemView

Impact
Reduces overhead of building FSViews with large numbers of files
Risk level (write none, low medium or high below)
None
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@the-other-tim-brown Can you fix the Azure CI failure?

@danny0405 error is:
TestUpsertPartitioner.testUpsertPartitionerWithSmallFileHandlingPickingMultipleCandidates:470 expected: <[BucketInfo {bucketType=UPDATE, fileIdPrefix=fg-1, partitionPath=2016/03/15}, BucketInfo {bucketType=UPDATE, fileIdPrefix=fg-2, partitionPath=2016/03/15}, BucketInfo {bucketType=UPDATE, fileIdPrefix=fg-3, partitionPath=2016/03/15}]> but was: <[BucketInfo {bucketType=UPDATE, fileIdPrefix=fg-3, partitionPath=2016/03/15}, BucketInfo {bucketType=UPDATE, fileIdPrefix=fg-2, partitionPath=2016/03/15}, BucketInfo {bucketType=UPDATE, fileIdPrefix=fg-1, partitionPath=2016/03/15}]>

I'll put up a separate minor pr to make the ordering deterministic for small file handling

@danny0405 https://github.com/apache/hudi/pull/11008

@danny0405 can you take another look when you get a chance? I have updated a few spots in the code


text:
[HUDI-3069] compact improve

Brief change log

compact improve

I found that when the compact plan is generated, the delta log files under each filegroup are arranged in the natural order of instant time. in the majority of cases,We can think that the latest data is in the latest delta log file, so we sort it from large to small according to the instance time, which can largely avoid rewriting the data in the compact process, and then optimize the compact time.
In addition, when reading the delta log file, we compare the data in the external spillablemap with the delta log data. If oldrecord is selected, there is no need to rewrite the data in the external spillablemap. Rewriting data will waste a lot of resources when data is spill to disk
This pull request is already covered by existing tests, such as (please describe tests).
Committer checklist

[*] Has a corresponding JIRA in PR title & commit()

[*] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@yihua : Can you follow up on the review please.

authors: nsivabalan scxwhite


text:
[HUDI-3931][DOCS] Guide to setup async metadata indexing

@nsivabalan @bhasudha @xushiyan Added this guide to go under Services tab. We can land this doc. I'll update the blog #5449 with more design elements. That can go after multi-modal index blog.


text:
[HUDI-4360] Fix HoodieDropPartitionsTool based on refactored meta sync

What is the purpose of the pull request
This PR fixes HoodieDropPartitionsTool based on refactored meta sync and the failed Java CI on master.
Brief change log

Fix the usage of old configs and APIs in HoodieDropPartitionsTool.

Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

The changes are limited to the HoodieDropPartitionsTool only and should not affect others.  Java CI passes.  Landing this to fix the master soon.


text:
[HUDI-6798] Add record merging mode and implement event-time ordering in the new file group reader

Change Logs
This PR adds a new table config hoodie.record.merge.mode to control the record merging mode and behavior in the new file group reader (HoodieFileGroupReader) and implements event-time ordering in it.  The table config hoodie.record.merge.mode is going to be the single config that determines how the record merging happens in release 1.0 and beyond.  Detailed changes include:

Adds RecordMergeMode to define three merging mode:

OVERWRITE_WITH_LATEST: using transaction time to merge records, i.e., the record from later transaction overwrites the earlier record with the same key.  This corresponds to the behavior of existing payload class OverwriteWithLatestAvroPayload.
EVENT_TIME_ORDERING: using event time as the ordering to merge records, i.e., the record with the larger event time overwrites the record with the smaller event time on the same key, regardless of transaction time. The event time or preCombine field needs to be specified by the user.  This corresponds to the behavior of existing payload class DefaultHoodieRecordPayload.
CUSTOM: using custom merging logic specified by the user.  When a user specifies a custom record merger strategy or payload class with Avro record merger, this is going to be specified so the record merging follows user-defined logic as before.

As of now, setting hoodie.record.merge.mode is not mandatory (HUDI-7850 as a follow-up to make it mandatory in release 1.0).  This PR adds the inference logic based on the payload class name, payload type, and record merger strategy in HoodieTableMetaClient to properly set hoodie.record.merge.mode in the table config.
Adds merging logic of OVERWRITE_WITH_LATEST and EVENT_TIME_ORDERING in HoodieBaseFileGroupRecordBuffer that do not have to go through the record merger APIs to simplify the implementation (opening up for further optimization when possible).  As a fallback, user can always set CUSTOM as the record merge mode to leverage payload class or record merger implementation for transaction or event time-based merging.
Adds a custom compareTo API in HoodieBaseFileGroupRecordBuffer to compare ordering field values of different types, due to an issue around ordering values in the delete records (HUDI-7848).
Adjusts tests to cover the new record merge modes.

New unit and functional tests are added around the new logic.  Existing unit and functional tests using file group readers on Spark cover all different merging modes.
Impact
Add record merging mode and implement event-time ordering in the new file group reader
Risk level
medium
Documentation Update
HUDI-7842 to update the docs on the website
Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

CI is green.


text:
IO Implementation using Go CDK

Extends PR #111
Implements #92. The Go CDK has well-maintained implementations for accessing objects stores from S3, Azure, and GCS via a io/fs.Fs-like interface. However, their file interface doesn't support the io.ReaderAt interface or the Seek() function that Iceberg-Go requires for files. Furthermore, the File components are private. So we copied the wrappers and implement the remaining functions inside of Iceberg-Go directly.
In addition, we add support for S3 Read IO using the CDK, providing the option to choose between the existing and new implementation using an extra property.
GCS connection options can be passed in properties map.

@dwilson1988 I saw your note about wanting to work on the CDK features, if you're able to provide some feedback that would be great.

@loicalleyne - happy to take a look. We use this internally in some of our software with Parquet and implemented a ReaderAt.  I'll do a more thorough review when I get a chance, but my first thought was to leave it completely separate from the blob.Bucket implementation and let the Create/New funcs simple accept a *blob.Bucket and leave the rest as an exercise to the user. This keeps it more or less completely isolated from the implementation. Thoughts on this direction?

My goal today was just to "get something on paper" to move this forward since the other PR has been stalled since July, I used the other PR as a starting point so I mostly followed the existing patterns. Very open to moving things around if it makes sense. Do you have any idea how your idea would work with the  interfaces defined in io.go?

Understood! I'll dig into your last question and get back to you.

Okay, played around a bit and here's where my head is at.
The main reason I'd like to isolated the creation of a *blob.Bucket is I've found that the particular implementation of bucket access can get tricky and rather than support it in this package for all situations, support the most common usage in io.LoadFS/inferFileIOFromSchema and change io.CreateBlobFileIO to accept a *url.URL and a *blob.Bucket. This enables a user to open a bucket with whatever implementation they so choose (GCS, Azure, S3, MinIO, Mem, FileSystem, etc) and there's less code here to maintain.
What I came up with is changing CreateBlobFileIO to:
// CreateBlobFileIO creates a new BlobFileIO instance
func CreateBlobFileIO(parsed *url.URL, bucket *blob.Bucket) (*BlobFileIO, error) {
	ctx := context.Background()
	return &BlobFileIO{Bucket: bucket, ctx: ctx, opts: &blob.ReaderOptions{}, prefix: parsed.Host + parsed.Path}, nil
}

The URL is still critical there, but now we don't have to concern ourselves with credentials to open the bucket except for in LoadFS.
Thoughts on this?

@dwilson1988
Sounds good, I've made the changes, please take a look.

@loicalleyne is this still on your radar?

hi @dwilson1988
yes, I'm wrapping up some work on another project and will be jumping back on this in a day or two.

Cool - just checking. I'll be patient. 🙂

@dwilson1988 made the suggested changes, there's a deprecation warning on the S3 config EndpointResolver methods that I haven't had time to look into, maybe you could take a look?

Hi @dwilson1988, do you think you'll have time to take a look at this?

Hi @dwilson1988, do you think you'll have time to take a look at this?

I opened a PR on your branch earlier today

@zeroshade hoping you can review when you've got time.

I should be able to give this a review tomorrow or Friday. In the meantime can you resolve the conflict in the go.mod? Thanks!

@loicalleyne looks like the integration tests are failing, unable to read the manifest files from the minio instance.

I did some debugging by copying some of the test scenarios into a regular Go program (if anyone can tell me how to run Delve in VsCode on a test that uses testify please let me know), running the docker compose file  and manually running the commands in iceberg-go\.github\workflows\go-integration.yml (note: to point to the local Minio in Docker I had to run export AWS_S3_ENDPOINT=http://127.0.0.1:9000).
It seems there's something wrong with the bucket prefix and how it interacts with subsequent calls, the prefix is assigned here
ie. it's trying to HEAD object
default/test_null_nan/metadata/00000-770ce240-af4c-49dd-bae9-6871f55f8be1.metadata.jsonwarehouse/default/test_null_nan/metadata/snap-2616202072048292962-1-6c011b0d-0f2a-4b62-bc17-158f94b1c470.avro

Unfortunately I don't have time to investigate any further right now, @dwilson1988 if you've seen this before please let me know.

I've been able to replicate and debug the issue myself locally. Aside from needing to make a bunch of changes to fix the prefix, bucket and key strings, I was still unable to get gocloud.dev/blob/s3blob to find the file appropriately.  I followed it down to the call to clientV2.GetObject and the s3v2.GetObjectInput has all the correct values: Bucket: "warehouse", Key: "default/test_all_types/....." etc. and yet minio still reports a 404. So I'm not sure what's going on.
I'll try poking at this tomorrow a bit more and see if i can make a small mainprog that is able to use s3blob to access a file from minio locally as a place to start.

Then I suspect it might be the s3ForcePathStyle option referred to  here. It affected Minio in particular once they moved to s3 V2.

@loicalleyne I haven't dug too far into the blob code, is it a relatively easy fix to handle that s3ForcePathStyle?

My understanding is that it's just another property to pass in props. Would also have to add it as a recognized property/constant in io/s3.go I should think.

s3.UsePathStyle
// Allows you to enable the client to use path-style addressing, i.e.,
// https://s3.amazonaws.com/BUCKET/KEY . By default, the S3 client will use virtual
// hosted bucket addressing when possible( https://bucket.s3.amazonaws.com/KEY ).
UsePathStyle [bool](https://pkg.go.dev/builtin#bool)

@loicalleyne can you take a look at the latest changes I made here?

Is it intended to not provide the choice between virtual hosted bucket addressing and path-style addressing?
LGTM otherwise - the tests are passing :)

@loicalleyne following pyiceberg's example, I've added an option to force virtual addressing. That work for you?

LGTM 👍

@dwilson1988 When you get a chance, can you take a look at the changes I made here. I liked your thought on isolating things, but there was still a bunch of specific options for particular bucket types that needed to get accounted for as the options are not always passed via URL due to how Iceberg config properties work.
So I'd like your thoughts or comments on what I ultimately came up with to simplify what @loicalleyne already had while solving the failing tests and whether it fits what you were thinking and using internally. Once this is merged, I'd definitely greatly appreciate contributions for Azure as you said :)

@zeroshade - I'll take a look this weekend!

authors: dwilson1988 loicalleyne zeroshade

repo: apache/iceberg-go


text:
Implement caching of manifest-files

Feature Request / Improvement
We currently loop over the manifests of a snapshot often just once. But now when we're compounding the operations (DELETE+APPEND), there is a fair chance that read a manifest more than once. The spec states that manifest files are immutable, this means that we can cache is locally using a method annotated with the lru_cache.

I am trying to working on this, is it possible to assign it to me?

@swapdewalkar Thanks for picking this up! I've just assigned it to you

Hi @swapdewalkar I wanted to check in and see if you have any updates on this task. If you need any assistance or if there are any obstacles, please let me know—I will be happy to help!

Hi, can we increase the scope of this issue to cache/store all_manifests, data_manifests & delete_manifests? Or do I create a new issue for this? This feature would be useful for tasks like Incremental Scans (Append, Changelog, etc) where we frequently access manifest files. I imagine this to be similar to the java implementation.
Also, since @swapdewalkar hasn't responded and if they do not have the time/bandwidth for the issue, I'm happy to give this a shot! :)

@chinmay-bhat I think we can generalize this quite easily, since from the spec:

Once written, data and metadata files are immutable until they are deleted.

I think we could go as easy to have a lru-cache based on the path to the metadata to cache it :)

Thanks @Fokko for the quick response.

based on the path to the metadata to cache it

I'm not clear on this. Are you saying we can simply add lru_cache to def manifests(self, io: FileIO) in class Snapshot? And then whenever we need data manifests or delete manifests, we iterate over the cached manifests? Wouldn't it be better to cache those too, since as you said, the files are immutable?
For ex:
@lru_cache
def manifests(self, io: FileIO):
......

@lru_cache
def data_manifests():
   return [manifest_file for manifest_file in self.manifests if manifest.content == ManifestContent.DATA]

@chinmay-bhat I don't think it is as easy as that. We should ensure that the manifest_list path is part of the cache. We could share the cache between calls, since if you do subsequent queries, and the snapshot hasn't been updated, this would speed up the call quite a bit.
We could also make the FileIO part of the caching key. I don't think that's stricktly required, but if something changed in the FileIO we might want to invalidate the cache, but I'm open to arguments here.

Thank you for clarifying! Here's how I imagine manifests() would look like :)
@lru_cache()
    def manifests(self, manifest_location: str) -> List[ManifestFile]:
        if manifest_location is not None:
            file = load_file_io().new_input(manifest_location)
            return list(read_manifest_list(file))
        return []

When we call snapshot.manifests(snapshot.manifest_list), if manifest_list is the same, we simply query the cached files. But if the snapshot is updated, manifest_list is also updated, and calling manifests() triggers a re-read of manifest files.
Is this similar to what you have in mind?

authors: Fokko MehulBatra chinmay-bhat swapdewalkar

repo: apache/iceberg-python


text:
Extend the DataFileWriterBuilder tests

In data_file_writer we write out a schema but don't have any tests for that. I think it would be good to write out a schema to validate that the field-IDs are there (they are, I checked by hand). And also add a test where we write DataFile that has a partition.

@Fokko I would like to try working on this, may I be assigned this?

authors: Fokko jonathanc-n

repo: apache/iceberg-rust


text:
[AIRFLOW-467] Allow defining of project_id in BigQeuryHook

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:
https://issues.apache.org/jira/browse/AIRFLOW-467
Testing Done:

Unit tests are added including backward compatibility tests

Awesome

authors: alexvanboxel bolkedebruin

repo: apache/incubator-airflow

license_source: bigquery


text:
Update README.md

Make sure you have checked all steps below.
JIRA

[ ] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX

Description

[ ] Here are some details about my PR, including screenshots of any UI changes:

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

[ ] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

I'm closing this as there has been no movement from the submitter.


text:
MLHR-1880 #resolve fixed default value documentation for setMaxLeng…

…th method.

@ilooner Please merge to 3.2

repo: apache/incubator-apex-malhar


text:
[CELEBORN-769] Change default value of celeborn.client.push.maxReqsIn…

…Flight to 16

What changes were proposed in this pull request?
Change default value of celeborn.client.push.maxReqsInFlight to 16.
Why are the changes needed?
Previous value 4 is too small, 16 is more reasonable.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.

cc @AngersZhuuuu @pan3793

changing the key from celeborn.client.push.maxReqsInFlight to celeborn.client.push.maxRequestsInFlight? it's better to avoid using abbreviations in configurations. WDYT? @pan3793 @waitinfuture

@cfmcgrady We use Reqs in several configurations, and "max" also is an abbr of "maximum" :)

@cfmcgrady We use Reqs in several configurations, and "max" also is an abbr of "maximum" :)

ok, Spark also has the key like spark.reducer.maxReqsInFlight

changing the key from celeborn.client.push.maxReqsInFlight to celeborn.client.push.maxRequestsInFlight? it's better to avoid using abbreviations in configurations. WDYT? @pan3793 @waitinfuture

+1 personally, but I think it's OK to use abbreviations, sometimes whole word is too long 😄

authors: cfmcgrady pan3793 waitinfuture

repo: apache/incubator-celeborn


text:
[BUG] dolphinscheduler compile failed

Problem description:
Now I have a problem. I have cloned the latest code from GitHub, but when I compile it, I fail to compile it.
Compile command:
mvn -U clean package -Prelease -Dmaven.test.skip=true

The following is the compile error information:

Current version:
1.2.0 dev branch
Compiling environment:
MacOS 10.15.2
Expected results:
Hope to be able to compile successfully in any environment and run it successfully.

please cat README.md, there would be 'how to build'.

need .proto file compile java class，idea have relate plugin

I changed the compile command to mvn clean install -Prelease  Still failed to compile

Current version:
1.2.0 dev branch ? dev or  1.2.0 branch ?
I compile dev，1.2.0 branch and incoming apache release 1.2.0 no problem

The current branch is dev
I can't import maven dependency with IntelliJ IDEA tool and report the error of Cannot resolve io.grpc:grpc-core:1.9.0

authors: lenboo qiaozhanwei sunnerrr wuchunfu

repo: apache/incubator-dolphinscheduler


text:
A Linux distribution for docker-compose

As I know，the image of  apache/dolphinscheduler was base on the Alpine image。But when I install Python dependencies，It takes a lot of time，at least an hour。What's more，It will need more any other  dependencies，for example
apk update && apk add --no-cache gcc g++ python3-dev libffi-dev librdkafka librdkafka-dev mariadb-connector-c-dev musl-dev libxml2-utils libxslt  libxslt-dev py3-numpy py3-pandas
But in the end, I fail to install python dependencies，because of the pandas。
So can you build another version based on other Linux，such as slim。 constrast
Thank you very much ！

@Joder5 This is not a problem. As for a any bare linux,  none of the libraries you mentioned will exist.
In order to improve the update speed, you can set the mirror source of alpine as follows:
# 1. install command/library/software
# If install slowly, you can replcae alpine's mirror with aliyun's mirror, Example:
# RUN sed -i "s/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g" /etc/apk/repositories
# RUN sed -i 's/dl-cdn.alpinelinux.org/mirror.tuna.tsinghua.edu.cn/g' /etc/apk/repositories

As for pip, you can also use the mirror source like https://pypi.tuna.tsinghua.edu.cn/simple
pip install --no-cache-dir -i https://pypi.tuna.tsinghua.edu.cn/simple

As for slim, we will consider it later.

I give you a version of Dockerfile based on debian:slim, which will be added to the warehouse after optimization in the future.
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

FROM openjdk:8-jdk-slim

ARG VERSION
ARG DEBIAN_FRONTEND=noninteractive

ENV TZ Asia/Shanghai
ENV LANG C.UTF-8
ENV DOCKER true

# 1. install command/library/software
# If install slowly, you can replcae alpine's mirror with aliyun's mirror, Example:
RUN echo \
"deb http://mirrors.tuna.tsinghua.edu.cn/debian/ buster main contrib non-free\n\
deb http://mirrors.tuna.tsinghua.edu.cn/debian/ buster-updates main contrib non-free\n\
deb http://mirrors.tuna.tsinghua.edu.cn/debian/ buster-backports main contrib non-free\n\
deb http://mirrors.tuna.tsinghua.edu.cn/debian-security buster/updates main contrib non-free" > /etc/apt/sources.list

RUN apt-get update && \
    apt-get install -y tzdata dos2unix python python3 procps netcat sudo tini postgresql-client && \
    echo "Asia/Shanghai" > /etc/timezone && \
    rm -f /etc/localtime && \
    dpkg-reconfigure tzdata && \
    rm -rf /var/lib/apt/lists/* /tmp/*

# 2. add dolphinscheduler
ADD ./apache-dolphinscheduler-incubating-${VERSION}-dolphinscheduler-bin.tar.gz /opt/
RUN ln -s /opt/apache-dolphinscheduler-incubating-${VERSION}-dolphinscheduler-bin /opt/dolphinscheduler
ENV DOLPHINSCHEDULER_HOME /opt/dolphinscheduler

# 3. add configuration and modify permissions and set soft links
COPY ./checkpoint.sh /root/checkpoint.sh
COPY ./startup-init-conf.sh /root/startup-init-conf.sh
COPY ./startup.sh /root/startup.sh
COPY ./conf/dolphinscheduler/*.tpl /opt/dolphinscheduler/conf/
COPY ./conf/dolphinscheduler/logback/* /opt/dolphinscheduler/conf/
COPY ./conf/dolphinscheduler/env/dolphinscheduler_env.sh /opt/dolphinscheduler/conf/env/
RUN dos2unix /root/checkpoint.sh && \
    dos2unix /root/startup-init-conf.sh && \
    dos2unix /root/startup.sh && \
    dos2unix /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh && \
    dos2unix /opt/dolphinscheduler/script/*.sh && \
    dos2unix /opt/dolphinscheduler/bin/*.sh && \
    rm -rf /bin/sh && \
    ln -s /bin/bash /bin/sh && \
    mkdir -p /tmp/xls /usr/lib/jvm && \
    ln -sf /usr/local/openjdk-8 /usr/lib/jvm/java-1.8-openjdk && \
    echo "Set disable_coredump false" >> /etc/sudo.conf

# 4. expose port
EXPOSE 5678 1234 12345 50051

ENTRYPOINT ["/usr/bin/tini", "--", "/root/startup.sh"]
``

close by #5158


text:
[Question] Stop the workflow, But can‘t Stop yarn application

I run a shell command waht is run sqoop to load data into hive. And it is running with mapreduce, when I stop the workflow，I think this MR application will be kill， but was not. The Mr applicaiton is still RUNNING. And I dont know why.
Which version of DolphinScheduler:
-[1.3.5]
This is the log

It is the same as #4862

fix by #4936

authors: CalvinKirs xingchun-chen zjw-zjw


text:
[FOLLOWUP] create table like clause support copy rollup

Proposed changes
for issue #6474

create table test.table1 like test.table with rollup (r1,r2) -- copy some rollup

create table test.table1 like test.table with rollup -- copy all rollup

create table test.table1 like test.table  -- only copy base table

Types of changes
What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)
[ ] Code refactor (Modify the code structure, format the code, etc...)
[ ] Optimization. Including functional usability improvements and performance improvements.
[ ] Dependency. Such as changes related to third-party components.
[x] Other.

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have created an issue on (Fix #6474) and described the bug/feature there in detail
[x] Compiling and unit tests pass locally with my changes
[x] I have added tests that prove my fix is effective or that my feature works
[x] If these changes need document changes, I have updated the document
[x] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Please rebase to master

repo: apache/incubator-doris


text:
Properly reset total size of segmentsToCompact in NewestSegmentFirstIterator

When NewestSegmentFirstIterator searches for segments to compact, it sometimes needs to clear the segments found so far and starts again. The total size of segments is also needed to be reset properly.

@gianm thanks for the quick review. Added a test.

repo: apache/incubator-druid


text:
The application.properties file specifies that the dubbo. protocol. port attribute is invalid.

The application.properties file specifies that the dubbo. protocol. port attribute is invalid.

The dubbo-spring-boot-starter.versionversion is 0.0.1.

You can used  dubbo.protocol.${name}.port

authors: huangxincheng mercyblitz


text:
I am confused with config server,  transporter config.

In my  understanding.
if we config  transporter="netty4" in provider side. It will  affect the consumer。
so we can use  server="netty4" in provider side.
but why dubbo  check sever key in DubboProtocol.initClient ?
the consumer with low version (not support netty4) will start failed.
And the transporter key will not  affect the consumer now.

transport config will not override.
the config that can override is like timeout, retries, loadbalance, actives and so on.

@whanice

if we config transporter="netty4" in provider side. It will affect the consumer。
Transporter is the protocolconfig attribute， the client does not have a protocolconfig concept, therefore the tranporter on the provider side will not be passed to the client.
but why dubbo check sever key in DubboProtocol.initClient ?
Because the value stored by the server will be passed to the client,therefore, the client will try to take the value of the server first.
the consumer with low version (not support netty4) will start failed.
I have tried neety4 as a provider, netty3 as a client, and can use it normally, Convenient to provide demo program verification?

Thank you very much for your support of dubbo.

Thanks for your guys reply.
Surely. the tcp connection should not be related to the nio framework,  No matter what you use, netty3, netty4 or mina in server. Will not affect the client use.
I care about migration.   when consumer use dubbo-2.5.x .  not have netty4 extension.
when provider  wants to upgrade dubbo verison to dubbo-2.6.x and use netty4. It will affect the consumer.
So I think in DubboProtol.initClient.  change to  will be better.
 // client type setting.
    String str = url.getParameter(Constants.CLIENT_KEY, url.getParameter(Constants.TRANSPORTER_KEY, Constants.DEFAULT_REMOTING_CLIENT));

And in the future. I think the sever ket shoud not pass to consumer.

You are right, I will confirm this question again.

Through communication, we keep netty as the extension point name, there will be no problem

And in the future. I think the sever key shoud not pass to consumer.

Agree, and this can be solved with #2030

authors: carryxyh chickenlj whanice zonghaishang

repo: apache/incubator-dubbo


text:
禁止某个Filter的执行

由于继承了别人的模块，能不能有办法在不修改别人源码的情况，在自己的模块中禁止掉某个Filter的执行

Maybe this documentation can help you. http://dubbo.apache.org/#!/docs/dev/impls/filter.md?lang=zh-cn

但是这个是只能禁止dubbo原生的filter，不能禁止用户自定义的

在 AbstractInterfaceConfig 中 直接限制住了 ，所以非dubbo framework下的会出现No such extension....异常
public void setFilter(String filter) {
        checkMultiExtension(Filter.class, "filter", filter);
        this.filter = filter;
    }

protected static void checkMultiExtension(Class<?> type, String property, String value) {
        checkMultiName(property, value);
        if (value != null && value.length() > 0) {
            String[] values = value.split("\\s*[,]+\\s*");
            String[] var4 = values;
            int var5 = values.length;

            for(int var6 = 0; var6 < var5; ++var6) {
                String v = var4[var6];
                if (v.startsWith("-")) {
                    v = v.substring(1);
                }

                if (!"default".equals(v) && !ExtensionLoader.getExtensionLoader(type).hasExtension(v)) {
                    throw new IllegalStateException("No such extension " + v + " for " + property + "/" + type.getName());
                }
            }
        }

    }


text:
Hover bug on 3D Scatter when give one valued data

Hey, when i set data as object (not directly two dimensional array) and object have one value, 3d scatter's on hover event is not functional.  You can try with below option code in this example page . I just get first item of current data and put it a array again.
    series: [{
        type: 'scatter3D',
        dimensions: [
            config.xAxis3D,
            config.yAxis3D,
            config.yAxis3D,
            config.color,
            config.symbolSiz
        ],
        data: [data.map(function (item, idx) {
            return [
                item[fieldIndices[config.xAxis3D]],
                item[fieldIndices[config.yAxis3D]],
                item[fieldIndices[config.zAxis3D]],
                item[fieldIndices[config.color]],
                item[fieldIndices[config.symbolSize]],
                idx
            ];
        })[0]],
        symbolSize: 12,
        // symbol: 'triangle',
        itemStyle: {
            borderWidth: 1,
            borderColor: 'rgba(255,255,255,0.8)'
        },
        emphasis: {
            itemStyle: {
                color: '#fff'
            }
        }
    }]

is there any solution?

repo: apache/incubator-echarts


text:
柱形图能分组吗？

如图能实现吗？

参考 http://gallery.echartsjs.com/editor.html?c=xBk7TY_hWx

authors: htjn pissang


text:
[4.2.0][bug]设置legend的背景色backgroundColor，图例依然被覆盖。

版本
[4.2.0]
重现条件
legend: {
data: ['line', 'bar'],
textStyle: {
color: ['#fff', '#ccc'],
backgroundColor: 'rgba(255,0,0,.5)'
}
},
设置textStyle: {
color: ['#fff', '#ccc'],
backgroundColor:'red'
}
主要是color为数组的时候, 同时设置backgroundColor, backgroundColor就会覆盖掉图例
截图

经测试 4.2.1 版本无法复现

authors: Ovilia ihwf


text:
fix call the same method twice

Fixes #2670 .
Motivation
Explain the content here.
Explain why you want to make the changes and what problem you're trying to solve.
Modifications
Describe the modifications you've done.
Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

fix for isse #2670

Pls add Motivation and Modifications description,and solve the build error.  @joaovitoras

repo: apache/incubator-eventmesh


text:
hawqextract column context does not exist error

When running hawq extract python stack trace is returned because pg_aoseg no longer has a column called content
[gpadmin@node2 ~]$ hawq extract -o rank_table.yaml foo 20160129:23:21:41:004538 hawqextract:node2:gpadmin-[INFO]:-try to connect database localhost:5432 gpadmin 20160129:23:21:41:004538 hawqextract:node2:gpadmin-[INFO]:-try to extract metadata of table 'foo' 20160129:23:21:41:004538 hawqextract:node2:gpadmin-[INFO]:--- detect FileFormat: AO 20160129:23:21:41:004538 hawqextract:node2:gpadmin-[INFO]:--- extract AO_FileLocations Traceback (most recent call last):   File "/usr/local/hawq-master/bin/hawqextract", line 551, in <module>     sys.exit(main())   File "/usr/local/hawq-master/bin/hawqextract", line 528, in main     metadata = extract_metadata(conn, args[0])   File "/usr/local/hawq-master/bin/hawqextract", line 444, in extract_metadata     cases[file_format]()   File "/usr/local/hawq-master/bin/hawqextract", line 363, in extract_AO_metadata     'Files': get_ao_table_files(rel_pgclass['oid'], rel_pgclass['relfilenode'])   File "/usr/local/hawq-master/bin/hawqextract", line 322, in get_ao_table_files     for f in accessor.get_aoseg_files(oid):   File "/usr/local/hawq-master/bin/hawqextract", line 164, in get_aoseg_files     return self.exec_query(qry)   File "/usr/local/hawq-master/bin/hawqextract", line 129, in exec_query     return self.conn.query(sql).dictresult() pg.ProgrammingError: ERROR:  column "content" does not exist LINE 2:         SELECT content, segno as fileno, eof as filesize

LGTM.
Verified this can fix generate yaml file.
Would you please file a jira and update the commit message? Like below:
HAWQ-XXX. hawqextract column context does not exist error
BTW, did you tested run with the generated yarm file? Thanks.

when attempting to create apache jira i get timeout or null pointer
exception.  I will try again later.
I only ran the YAML file through yamllint but did not test it in a
mapreduce job or anything. I stumbled on the error while working on
something else.  Also comparing it to GPDB the output looks similar.
On Fri, Mar 11, 2016 at 9:11 AM, Radar Lei notifications@github.com wrote:

LGTM.
Verified this can fix generate yaml file.
Would you please file a jira and update the commit message? Like below:
HAWQ-XXX. hawqextract column context does not exist error
BTW, did you tested run with the generated yarm file? Thanks.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-hawq/pull/305#issuecomment-195406454
.

Fix yaml file generation is good enough for this pull request. We can check if it work with mapreduce job in separate jira.
Please update the commit message, then we can merge it in. Thanks.

jira created
https://issues.apache.org/jira/browse/HAWQ-535

You might need to update the git commit message with the jira number, so when we finished merge, it can keep the origin author information.

thanks i amended the commit message
On Mon, Mar 14, 2016 at 9:42 AM, Radar Lei notifications@github.com wrote:

You might need to update the git commit message with the jira number, so
when we finished merge, it can keep the origin author information.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-hawq/pull/305#issuecomment-196343453
.

LGTM

Now the fix is in. Thanks.

@randomtask1155 , please close this pull request since it's already been merged. Thanks.

repo: apache/incubator-hawq


text:
[HIVEMALL-54][SPARK] Add an easy-to-use script for spark-shell

What changes were proposed in this pull request?
This pr added a script to automatically download the latest Spark version, compile Hivemall for the version, and invoke spark-shell with the compiled Hivemall binary.
This pr also included a documentation for hivemall-on-spark installation.
What type of PR is it?
Improvement
What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-54
How was this patch tested?
Manually tested.

Coverage increased (+0.3%) to 36.142% when pulling 5433fd56db138880c9e0ed0f2d20cf7396d4e5e8 on maropu:AddScriptForSparkShell into 85f8e173a2a97005c00b84140f4b9150060c4a56 on apache:master.

Coverage increased (+0.3%) to 36.142% when pulling 5433fd56db138880c9e0ed0f2d20cf7396d4e5e8 on maropu:AddScriptForSparkShell into 85f8e173a2a97005c00b84140f4b9150060c4a56 on apache:master.

@amaya382 @Lewuathe Could you confirm that the updated bin/spark-shell properly works if your spare time?

@maropu BTW, better to update incubator-hivemall-site once this PR is merged.

yea, I'll update just after this merged.

@myui 👌

Coverage remained the same at 35.844% when pulling 17f2dcf375a1c4c0ad979c80c6591143aefe8a1b on maropu:AddScriptForSparkShell into 85f8e173a2a97005c00b84140f4b9150060c4a56 on apache:master.

@amaya382 Have you confirmed?
@maropu please merge this PR if he say fine.

Looks mostly good, but I found minor issues not directly related to this PR.
Some declarations in define-all.spark are incorrect and duplicated.
e.g. train_arowh: https://github.com/maropu/incubator-hivemall/blob/AddScriptForSparkShell/resources/ddl/define-all.spark#L32-L36

@amaya382 Can you make prs to fix them?

Merged.

@amaya382 I made a JIRA ticket: https://issues.apache.org/jira/browse/HIVEMALL-65

@maropu okay, I'll do in a few days

Coverage remained the same at 35.814% when pulling ff03adf1de142fb9cdbe373bed718dda2e8e840e on maropu:AddScriptForSparkShell into 247e1aef87ddc789dacfb74e469263e7e2ab603e on apache:master.

Coverage remained the same at 35.814% when pulling ff03adf1de142fb9cdbe373bed718dda2e8e840e on maropu:AddScriptForSparkShell into 247e1aef87ddc789dacfb74e469263e7e2ab603e on apache:master.

@maropu Could you update incubator-hivemall-site? This feature should be documented.

authors: amaya382 coveralls maropu myui

repo: apache/incubator-hivemall


text:
[HIVEMALL-96-2] Added Geo Spatial UDFs

What changes were proposed in this pull request?
This PR added 5 Geo Spatial UDFs:  lat2tiley, lon2tilex, tilex2lon, tileytolat, and haversine_distance.
What type of PR is it?
Feature
What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-96
How was this patch tested?
Unit tests and manual tests
How to use this feature?
WITH data as (
  select 51.51202 as lat, 0.02435 as lon, 17 as zoom
  union all
  select 51.51202 as lat, 0.02435 as lon, 4 as zoom
  union all
  select null as lat, 0.02435 as lon, 17 as zoom
)
select
   lat, lon, zoom,
   tile(lat, lon, zoom) as tile,
   (lon2tilex(lon,zoom) + lat2tiley(lat,zoom) * cast(pow(2, zoom) as bigint)) as tile2,
   lon2tilex(lon, zoom) as xtile,
   lat2tiley(lat, zoom) as ytile,
   tiley2lat(lat2tiley(lat, zoom), zoom) as lat2,  -- tiley2lat returns center of the tile
   tilex2lon(lon2tilex(lon, zoom), zoom) as lon2 -- tilex2lon returns center of the tile
from
   data;

select
  haversine_distance(35.6833, 139.7667, 34.6603, 135.5232) as km,
  haversine_distance(35.6833, 139.7667, 34.6603, 135.5232, true) as mile;

Coverage increased (+0.2%) to 40.19% when pulling 6c391786cf877ef3080db9403192c55700491bae on myui:HIVEMALL-96-2 into c06378a81723e3998f90c08ec7444ead5b6f2263 on apache:master.

authors: coveralls myui


text:
Config the rest frontend service max worker thread

Why are the changes needed?

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

thanks, merged to master

repo: apache/incubator-kyuubi


text:
The Linkis website needs to add a github action to check whether the linkis website link is available

Some links on the linkis website have been broken.

#341 fix the issue

repo: apache/incubator-linkis-website


text:
[功能] Dev 1.3.1 metadatasource driver profiles not loaded by default

linkis-metadata-query-jdbc module driver profiles not loaded by default

最好可以添加下背景说明下为什么没有添加到到provide 依赖的原因
这些依赖POM配置建议以test模式放到pom文件中，用户只需要移除注解就能编译使用
添加对应驱动jar包的下载地址列表 方便直接使用官方安装包的同学使用，并告知应该拷贝到对应的那个目录下

authors: casionone dlimeng