text:
[Dubbo-fix]  fix typo CHARECTER -> CHARACTER

What is the purpose of the change
Fix typo CHARECTER -> CHARACTER.
I have searched in opened PR.
Fell free to close it if it has be included in other PRs or It is not expected to change.
Brief changelog
Only rename SPLIT_IPV4_CHARECTER &SPLIT_IPV6_CHARECTER
to SPLIT_IPV4_CHARACTER &SPLIT_IPV6_CHARACTER .
Verifying this change
These two private variables do not affect the other methods or classes except getRandomPort .
The current test case 'NetUtilsTest' is enough.
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[x] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[x] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #5744 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master    #5744      +/-   ##
============================================
- Coverage     61.32%   61.29%   -0.03%
+ Complexity      498      496       -2
============================================
  Files           928      928
  Lines         37947    37947
  Branches       5480     5480
============================================
- Hits          23270    23260      -10
- Misses        12153    12160       +7
- Partials       2524     2527       +3

Impacted Files
Coverage Δ
Complexity Δ

...n/java/org/apache/dubbo/common/utils/NetUtils.java
67.7% <100%> (ø)
0 <0> (ø)
:arrow_down:

...ng/exchange/support/header/HeartbeatTimerTask.java
73.68% <0%> (-5.27%)
0% <0%> (ø)

...e/dubbo/remoting/transport/netty/NettyChannel.java
55.68% <0%> (-4.55%)
20% <0%> (-1%)

...pache/dubbo/remoting/transport/AbstractServer.java
53.75% <0%> (-3.75%)
0% <0%> (ø)

...ache/dubbo/remoting/transport/mina/MinaClient.java
56.92% <0%> (-1.54%)
8% <0%> (-1%)

...g/apache/dubbo/registry/consul/ConsulRegistry.java
62.11% <0%> (-0.63%)
30% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2031110...a1e8c47. Read the comment docs.

LGTM

authors: CodingSinger codecov-io tangcent


text:
Fill conditionKeys collections when necessary

What is the purpose of the change
The action to fill the collection comes after the URL matching process to avoid invalid fill collections
Brief changelog
XXXXX
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #7462 (677342d) into master (eeac3a0) will increase coverage by 0.48%.
The diff coverage is 50.00%.

@@             Coverage Diff              @@
##             master    #7462      +/-   ##
============================================
+ Coverage     59.38%   59.86%   +0.48%
+ Complexity      502      489      -13
============================================
  Files          1079     1089      +10
  Lines         43329    43815     +486
  Branches       6318     6380      +62
============================================
+ Hits          25733    26232     +499
+ Misses        14759    14631     -128
- Partials       2837     2952     +115

Impacted Files
Coverage Δ

...rpc/cluster/configurator/AbstractConfigurator.java
61.90% <50.00%> (+0.96%)
:arrow_up:

...ookeeper/ZookeeperDynamicConfigurationFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...he/dubbo/rpc/protocol/rmi/RmiRemoteInvocation.java
0.00% <0.00%> (-80.00%)
:arrow_down:

...pport/zookeeper/ZookeeperDynamicConfiguration.java
0.00% <0.00%> (-62.07%)
:arrow_down:

...org/apache/dubbo/remoting/zookeeper/EventType.java
0.00% <0.00%> (-61.91%)
:arrow_down:

...org/apache/dubbo/rpc/protocol/rmi/RmiProtocol.java
0.00% <0.00%> (-61.67%)
:arrow_down:

.../configcenter/support/zookeeper/CacheListener.java
0.00% <0.00%> (-60.00%)
:arrow_down:

...g/context/properties/DefaultDubboConfigBinder.java
42.85% <0.00%> (-57.15%)
:arrow_down:

...a/org/apache/dubbo/rpc/filter/AccessLogFilter.java
28.35% <0.00%> (-50.75%)
:arrow_down:

...gcenter/wrapper/CompositeDynamicConfiguration.java
0.00% <0.00%> (-50.00%)
:arrow_down:

... and 243 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eeac3a0...677342d. Read the comment docs.

source: gharchive/pull-request

authors: BurningCN codecov-commenter

repo: apache/dubbo

license_type: permissive

license_source: github-api


text:
add echarty to extensions download page

Thank you for the invitation ZW 🎉

Thanks for your contribution! The website will be updated in several days.

authors: helgasoft plainheart

repo: apache/echarts-www


text:
The zoomlock property of datazoom is invalid

Version
5.0.0
Reproduction link
http://jsrun.net/SSIKp
Steps to reproduce
The zoomlock attribute of version 4. X is valid, and version 5.0.0 is invalid. The datalock attribute of datazoom in the online example of ecarts5 on the official website is also invalid
What is expected?
The zoomlock property of datazoom works
What is actually happening?
The zoomlock property of datazoom does not work

// dataZoom is Array
dataZoom: [
      {
              zoomLock: true,
              end: 50,
      },
      {
              zoomLock: true,
              end: 50,
        },
],

or
// dataZoom is Object
dataZoom: {
     zoomLock: true,
     end: 50,
},

@LongYue9608

// dataZoom is Array
dataZoom: [
      {
              zoomLock: true,
              end: 50,
      },
      {
              zoomLock: true,
              end: 50,
        },
],

or
// dataZoom is Object
dataZoom: {
     zoomLock: true,
     end: 50,
},

@LongYue9608

// dataZoom is Array
dataZoom: [
      {
              zoomLock: true,
              end: 50,
      },
      {
              zoomLock: true,
              end: 50,
        },
],

or
// dataZoom is Object
dataZoom: {
     zoomLock: true,
     end: 50,
},

@LongYue9608

Using the two ways you said, the zoomlock property still doesn't work
@susiwen8

// dataZoom is Array
dataZoom: [
      {
              zoomLock: true,
              end: 50,
      },
      {
              zoomLock: true,
              end: 50,
        },
],

or
// dataZoom is Object
dataZoom: {
     zoomLock: true,
     end: 50,
},

@LongYue9608

Using the two ways you said, the zoomlock property still doesn't work
@susiwen8

@LongYue9608

@LongYue9608

鼠标放在滑块中间会出现一个+号,单击拖动依然可以缩放设置start和end得值，4.x版本设置了zoomLock不会这样，我想让dataZoom组件设置了zoomLock为true之后只能平移，不能缩放 @susiwen8

鼠标放在滑块中间会出现一个+号,单击拖动依然可以缩放设置start和end得值，4.x版本设置了zoomLock不会这样，我想让dataZoom组件设置了zoomLock为true之后只能平移，不能缩放 @susiwen8

@LongYue9608 确实有这样的问题。

@LongYue9608 确实有这样的问题。

大概啥时候能修复好这个 @susiwen8

大概啥时候能修复好这个 @susiwen8

应该在 5.2.0里面修复。

应该在 5.2.0里面修复。

应该在 5.2.0里面修复。

我裂开来,现在才5.0.0,🐕产品一直让我改,回到4.x版本色调啥得视觉效果差了好多

应该在 5.2.0里面修复。

我裂开来,现在才5.0.0,🐕产品一直让我改,回到4.x版本色调啥得视觉效果差了好多

@LongYue9608 先提了个 PR，看能不能随 5.1发版，5.0.1是赶不上了

@LongYue9608 先提了个 PR，看能不能随 5.1发版，5.0.1是赶不上了

@LongYue9608 你可以先关闭这个 brush 功能

@LongYue9608 你可以先关闭这个 brush 功能

brushSelect这个是代替了之前的zoomLock功能吗 @pissang

brushSelect这个是代替了之前的zoomLock功能吗 @pissang

不是，两个功能

不是，两个功能

brushSelect:false现在呈现的效果和4.x版本的zoomLock:true好像是一样的-我用这个代替了

brushSelect:false现在呈现的效果和4.x版本的zoomLock:true好像是一样的-我用这个代替了

@LongYue9608 先提了个 PR，看能不能随 5.1发版，5.0.1是赶不上了 如果是急需，可以自己打一个包

您好, 现在这个问题得到解决了么, 我使用 "echarts": "^5.3.2" 还是不生效

source: gharchive/issue

repo: apache/echarts


text:
[Feature] 多Y轴的显示，用于显示多通道波形图

What problem does this feature solve?
比如 在信号分析领域，需要使用多Y轴查看波形图，支持放大缩小等
What does the proposed API look like?
可以查看EDFbrower,查看信号图

https://echarts.apache.org/examples/en/editor.html?c=area-rainfall

我的意思是 多Y轴垂直叠加，比如

核心是ecahrt能否 多Y轴垂直叠加

这种目前需要用循环 创建多个 Grid，多个 X轴，多个 Y轴，多个 series 实现

authors: Airkro Ovilia chenweihan


text:
Piecewise visualmap 需要和legend一样支持分页

Piecewise的visualmap，在散点图中分类变量的分类增多时，visuamap组件的布局会超出画布，很难做处理，希望这块和legend一样支持分页显示

这个问题有人处理吗？

同问，visualMap很长怎么处理，换行？or分页？

你好，可以看看这个回答，会不会帮你解决问题，https://github.com/apache/echarts/issues/8413#issuecomment-772348246

这个问题几件了还没人解决呢

authors: IEfucker Marsellen ToughRover lcyh


text:
test(pie): modify test data. close #18732

Brief Information
This pull request is in the type of:

[ ] bug fixing
[ ] new feature
[ ] others

What does this PR do?

Fixed issues

Details
Before: What was the problem?

After: How does it behave after the fixing?

Document Info
One of the following should be checked.

[ ] This PR doesn't relate to document changes
[ ] The document should be updated later
[ ] The document changes have been made in apache/echarts-doc#xxx

Misc
ZRender Changes

[ ] This PR depends on ZRender changes (ecomfe/zrender#xxx).

Related test cases or examples to use the new APIs
N.A.
Others
Merging options

[ ] Please squash the commits into a single one when merging.

Other information

It's not necessary to modify the test case. This is an example of echarts-examples repo. You can submit a PR there if you'd like to. The link of this example file is https://github.com/apache/echarts-examples/blob/gh-pages/public/examples/ts/pie-half-donut.ts

authors: MingYang-Yin plainheart


text:
Portfolio module

Description
Describe the changes made and why they were made. Ignore if these details are present on the associated Jira ticket
Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!

[ ] Commit message starts with the issue number from https://issues.apache.org/jira/projects/FINERACT/. Ex: FINERACT-646 Pockets API.

[x] Coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions have been followed.

[] API documentation at https://github.com/apache/fineract/blob/develop/api-docs/apiLive.htm has been updated with details of any API changes.

[x] Integration tests have been created/updated for verifying the changes made.

[x] All Integrations tests are passing with the new commits.

[ ] Submission is not a "code dump".  (Large changes can be made "in repository" via a branch.  Ask on the list.)

Our guidelines for code reviews is at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide

This code seems to include Commits from other PRs. Is that the correct way to do it?
Also, Travis is failing. Can you investigate?

@kangbreder: Given that the pr's you are sending out are a part of GSOC, can you please get in touch with your mentor and ensure that (s)he is reviewing and merging them in a timely manner?

authors: awasum kangbreder vishwasbabu

repo: apache/fineract


text:
FINERACT-972: Ensure Tomcat is started after compiling integration tests

Description
Tomcat was being started before compiling integration tests, meaning that if the integration test compilation failed then it was left running. After adding the dependencies, it now is started only after successful compilation of integration tests.
FINERACT-972

I don't know much about this. I'll just pull @vorburger to have a look

This is great! I had been wondering why, as your FINERACT-972 says, Tomcat sometimes didn't shut down.... now I know! Thanks for having spotted and fixed this.

authors: ptuomola vorburger xurror


text:
[FLINK-15341][client] Reset context classload in PackagedProgramUtils#createJobGraph

It happens we don't reset context classload in PackagedProgramUtils#createJobGraph. For fixing this, we can just reuse PackagedProgramUtils#getPipelineFromProgram.
cc @aljoscha @kl0u

@flinkbot run azure

@kl0u could you give this pull request a look?

Merged!

authors: TisonKun aljoscha

repo: apache/flink


text:
[FLINK-3534] [runtime] Prevent canceling Execution from failing

@StephanEwen, can you have a look at this?
This should handle cases where a cancelled execution is failed (as reported by @rmetzger), but still allow the ExecutionGraph to be failed (and not suffer from the point you raised in #1726).
Robert would like this in the next RC, but in general I think that it's better to have more exposure for a change to the ExecutionGraph... I'm undecided right now. What do you think?

Looks good to me!
+1 to merge this

I've fixed the test failures.
@StephanEwen, the changes can result in the following behaviour: if a slot is released, some of its executions can end up in cancelled rather than failed state. Is that OK? The following happens: The slot's first execution is failed. This triggers an execution graph fail, after which all executions are cancelled. When the slot fails further executions, these are already cancelling and go to state cancelled. Before they would still go state FAILED.

True, that is rather ugly behavior. I guess the important thing is that at least one of the Executions results in state failed.

My local Travis build has passed. I will merge this to master and release-1.0.

authors: StephanEwen uce


text:
[FLINK-25405] Let BlobServer/BlobCache check and delete corrupted blobs at start up

What is the purpose of the change
This commit introduces the BlobUtils.checkAndDeleteCorruptedBlobs that is called by the BlobServer and AbstractBlobCache at start up to delete all potentially corrupted blobs. The check is done by recomputing the MessageDigest for every blob.
This PR is based on #18164.
Verifying this change
Added tests in BlobServerGetTest, BlobServerPutTest, TransientBlobCachTest, PermanentBlobCacheTest.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Thanks for the review @zentol. I've answered your comments. Merging this PR once #18164 has been merged.


text:
[FLINK-25589][docs][connectors] Update Chinese version of Elasticsearch connector docs

What is the purpose of the change
Update Chinese version document of Elasticsearch sink interface related to FLINK-24326 (#17930)
Brief change log

Translate docs/content.zh/docs/connectors/datastream/elasticsearch.md
Translate docs/content.zh/docs/connectors/table/elasticsearch.md
Trivial fix of docs/content/docs/connectors/datastream/elasticsearch.md

Verifying this change
This is a document update.
Please build docs locally to verify document content and syntax.
cd docs
./build_docs.sh -p

Then you can preview http://localhost:1313 in the local browser.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@flinkbot run azure

@flinkbot run azure

@fapaul Would you like to merge this PR? Thanks.

authors: JingGe biyuhao


text:
[FLINK-30844][runtime] Increased TASK_CANCELLATION_TIMEOUT for testInterruptibleSharedLockInInvokeAndCancel

What is the purpose of the change
Fixing testInterruptibleSharedLockInInvokeAndCancel test
Brief change log

Increased TASK_CANCELLATION_TIMEOUT for testInterruptibleSharedLockInInvokeAndCancel since it looks that on the overloaded machine the current timeout is too small

Verifying this change
It is test itself
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@pnowojski , Could you take a look at this PR, please? I just increased the timeout since I didn't find any deadlocks or other suspicious things. It seems the test sometimes fails on the overloaded machines only because of this timeout.


text:
[FLINK-31876][QS] Migrate flink-queryable-state-client tests to JUnit5

What is the purpose of the change
Migrate flink-queryable-state-client testsD to JUnit5

@jiexray @reswqa Thanks for the review, I updated the PR according to your suggestions, please help take a look again.

@reswqa Thanks for the detailed review, fixed as suggested, please help take a look again.

My local CI https://dev.azure.com/fredia/flink/_build/results?buildId=518&view=results has passed.

My local CI https://dev.azure.com/fredia/flink/_build/results?buildId=518&view=results has passed.

I have squashed and rebased this to origin/master to see if it can pass CI.

authors: fredia reswqa


text:
[FLINK-1743] Add multinomial logistic regression to machine learning library

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

[x] General

The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
The pull request addresses only one issue
Each commit in the PR has a meaningful commit message (including the JIRA id)

[x] Documentation

Documentation has been added for new functionality
Old documentation affected by the pull request has been updated
JavaDoc for public methods has been added

[x] Tests & Build

Functionality added by the pull request is covered by tests
mvn clean verify has been executed successfully locally or a Travis build has passed

Refactor MultipleLinearRegression to extend GeneralizedLinearModel
Implement LogisticRegression
Change LogisticLoss with label in {0, 1}
Add LogisticLinerRegression test case

Hello @mtunique and thank you for you contribution!
Could you briefly describe your changes/additions and how you have tested the implementation for correctness?

@thvasilo

Add MultipleLinearRegression  to provide interfaces for the set of common properties and functions generalized linear models share.
Change LogisticLoss to let it be with label in {0, 1}
LogisticLinerRegression test: This test is originally from the Apache Spark project. Training Data: Weights are created randomly. Label is calculated by given scale and offset.

Thank you for the info!
In that case it's a bit ambiguous now which issue this PR is solving. If you are introducing an interface for GLMs we should be having this discussion under that context, because it changes the scope of the PR quite significantly re. design decisions vs. a simple implementation of logistic regression.
If you want, we can split this into different PRs and take it from there.
P.S. Any code that comes from the Spark project mus be clearly labeled as such, please make sure to give proper attribution.

Thanks all for the comments.
The code from Spark has be labeled like some other codes from Spark.
I will spilt this PR into two PRs. The other one is for FLINK-2013.

@thvasilo After #3756 is merged, I will change this PR.

Hi @mtunique any update on this PR?

Hello @bowenli86, AFAIK nobody's working on the PR's right now. I'm currently 100% focused on finishing my PhD so I won't have time to review.

Closing since flink-ml is effectively frozen.

authors: bowenli86 mtunique thvasilo zentol


text:
[FLINK-6216] [table] DataStream unbounded groupby aggregate with early firing

Implemented an unbounded groupby aggregate with early firing (period is 1, emit per every record)
Refactored the DataStreamAggregate to  DataStreamGroupWindowAggregate

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

[x] General

The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
The pull request addresses only one issue
Each commit in the PR has a meaningful commit message (including the JIRA id)

[ ] Documentation

Documentation has been added for new functionality
Old documentation affected by the pull request has been updated
JavaDoc for public methods has been added

[x] Tests & Build

Functionality added by the pull request is covered by tests
mvn clean verify has been executed successfully locally or a Travis build has passed

@fhueske, thanks for your review. I have addressed all your comments, and rebased the code to the master. Please take a look.

Thanks @shaoxuan-wang.
I merged the PR to table-retraction branch. The ASF GitBot only closes PRs which are merged to master. Can you close this PR?
Thank you!

authors: fhueske shaoxuan-wang


text:
[docs] Add clickable stack figure on overview page

This commit adds a figure giving an overview of Flink's stack. Users can click on components to go to the respective component docs.

I'm sure there will be people who vote for other colors etc. Please only provide comments with concrete suggestions on what to change. ;)
This is only for 0.9-SNAPSHOT. Do we want another version for 0.8 w/o Gelly, ML, Table, and, Tez?

Nice work, thank you.
I think its fine to add this only to 0.9.

I think it is great as it is.
+1 to add!

This is awesome. Should be in 0.9! I think it's an attraction to drive users to a newer release when they check out the documentation on the website.

Thanks. Merging this.

Thanks, great work:
I'd like to ask for the following renamings in the figure:

Optimizer -> Batch Optimizer
Stream Builder -> Streaming Optimizer

+1, looks great

authors: StephanEwen ktzoumas mbalassi mxm rmetzger uce


text:
[FLINK-8650] Tests for WINDOW clause and documentation update

What is the purpose of the change
This PR adds test and documentation coverage of WINDOW clause
Brief change log

Test that the same queries but with different specification of windows have the same plan
Mentioning in doc WINDOW syntax

Verifying this change
This change added tests and can be verified as follows:
via running of org.apache.flink.table.api.stream.sql.OverWindowTest
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

@fhueske thank you for your review
I have left comments and done changes. Please let me know if it is required some more efforts

Thanks for the update @snuyanzin.
Looks good. Will merge it.

authors: fhueske snuyanzin


text:
HDDS-2997. Support github comment based commands

What changes were proposed in this pull request?
Before we started to use github actions we had the opportunity to use some "commands" in github comments. For example when a /label xxx comment has been added to a PR, a bot added the label (by default just the committers can use labels, but with this approach it was possible for everyone).
Since the move to use github actions I got multiple question about re-triggering the test. Even it it's possible to do with pushing an empty commit (only by the owner or committer) I think it would be better to restore the support of comment commands.
This patch follows a very simple approach. The available commands are store in a separated subdirectory as shell scripts and they are called by a lightweight wrapper.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2997
How was this patch tested?
As it works only if merged to the master I tested it in my own fork:
Feel free to use this example PR to test it:
https://github.com/elek/hadoop-ozone/pull/9

Thanks the review and the test @adoroszlai I am merging it to the master right now...

repo: apache/hadoop-ozone


text:
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186)

branch-3.3 backport of PR https://github.com/apache/hadoop/pull/4186

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
10m 20s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ branch-3.3 Compile Tests _

+1 :green_heart:
mvninstall
38m 57s

branch-3.3 passed

+1 :green_heart:
compile
1m 26s

branch-3.3 passed

+1 :green_heart:
checkstyle
1m  4s

branch-3.3 passed

+1 :green_heart:
mvnsite
1m 34s

branch-3.3 passed

+1 :green_heart:
javadoc
1m 43s

branch-3.3 passed

+1 :green_heart:
spotbugs
3m 38s

branch-3.3 passed

+1 :green_heart:
shadedclient
28m 23s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 21s

the patch passed

+1 :green_heart:
compile
1m 15s

the patch passed

+1 :green_heart:
javac
1m 15s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 47s

the patch passed

+1 :green_heart:
mvnsite
1m 23s

the patch passed

+1 :green_heart:
javadoc
1m 29s

the patch passed

+1 :green_heart:
spotbugs
3m 32s

the patch passed

+1 :green_heart:
shadedclient
27m 54s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
210m 20s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 59s

The patch does not generate ASF License warnings.

333m 14s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade

hadoop.hdfs.server.datanode.TestDirectoryScanner

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4251

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 7bb0255fa870 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.3 / fe1cdcc1fddbfdffbd6aef932e8e276705296ea1

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/1/testReport/

Max. process+thread count
2170 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/1/console

versions
git=2.17.1 maven=3.6.0 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Failed unit tests seem to be unrelated to the change. I retriggered Jenkins to make sure.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 54s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ branch-3.3 Compile Tests _

+1 :green_heart:
mvninstall
41m  1s

branch-3.3 passed

+1 :green_heart:
compile
1m 29s

branch-3.3 passed

+1 :green_heart:
checkstyle
1m  9s

branch-3.3 passed

+1 :green_heart:
mvnsite
1m 39s

branch-3.3 passed

+1 :green_heart:
javadoc
1m 49s

branch-3.3 passed

+1 :green_heart:
spotbugs
4m  3s

branch-3.3 passed

+1 :green_heart:
shadedclient
28m 40s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 20s

the patch passed

+1 :green_heart:
compile
1m 23s

the patch passed

+1 :green_heart:
javac
1m 23s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 51s

the patch passed

+1 :green_heart:
mvnsite
1m 27s

the patch passed

+1 :green_heart:
javadoc
1m 29s

the patch passed

+1 :green_heart:
spotbugs
3m 46s

the patch passed

+1 :green_heart:
shadedclient
28m 28s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
227m 19s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
1m  8s

The patch does not generate ASF License warnings.

344m  9s

Reason
Tests

Failed junit tests
hadoop.hdfs.TestRollingUpgrade

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4251

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux bdc786fcc6c8 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.3 / fe1cdcc1fddbfdffbd6aef932e8e276705296ea1

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/2/testReport/

Max. process+thread count
1889 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/2/console

versions
git=2.17.1 maven=3.6.0 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 51s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ branch-3.3 Compile Tests _

+1 :green_heart:
mvninstall
40m 25s

branch-3.3 passed

+1 :green_heart:
compile
1m 29s

branch-3.3 passed

+1 :green_heart:
checkstyle
1m  8s

branch-3.3 passed

+1 :green_heart:
mvnsite
1m 39s

branch-3.3 passed

+1 :green_heart:
javadoc
1m 49s

branch-3.3 passed

+1 :green_heart:
spotbugs
3m 58s

branch-3.3 passed

+1 :green_heart:
shadedclient
28m 56s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 27s

the patch passed

+1 :green_heart:
compile
1m 17s

the patch passed

+1 :green_heart:
javac
1m 17s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 50s

the patch passed

+1 :green_heart:
mvnsite
1m 28s

the patch passed

+1 :green_heart:
javadoc
1m 28s

the patch passed

+1 :green_heart:
spotbugs
3m 48s

the patch passed

+1 :green_heart:
shadedclient
28m  5s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
228m 32s
/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 57s

The patch does not generate ASF License warnings.

344m 32s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.namenode.TestLargeDirectoryDelete

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4251

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux a429932b2c3f 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.3 / fe1cdcc1fddbfdffbd6aef932e8e276705296ea1

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/4/testReport/

Max. process+thread count
1914 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4251/4/console

versions
git=2.17.1 maven=3.6.0 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

The failed unit test is unrelated to the change.

authors: hadoop-yetus tomscut virajjasani

repo: apache/hadoop


text:
HDFS-16677. Add OP_SWAP_BLOCK_LIST as an operation code in FSEditLogOpCodes.

Description of PR
https://issues.apache.org/jira/browse/HDFS-16677
sub task of https://issues.apache.org/jira/browse/HDFS-14978
How was this patch tested?
add unit test testSwapBlockListEditLog
For code changes:
add swapBlockListEditLog & Feature.SWAP_BLOCK_LIST for NameNodeLayoutVersion

@ayushtkn @jojochuang    Hi, would you take some time to review it?


text:
HADOOP-18344. Upgrade AWS SDK to 1.12.262

Description of PR
How was this patch tested?
object store test in progress. first run had two failures I'd not see before

landsat test timeout
oom in a test writing data. I'd set my connector to use bytebuffer for block buffering and it ran out. But why now? SDK update or just upload delays triggering it?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 56s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
38m 13s

trunk passed

+1 :green_heart:
compile
0m 41s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
compile
0m 40s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
mvnsite
0m 46s

trunk passed

+1 :green_heart:
javadoc
0m 45s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
0m 37s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
61m 30s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 23s

the patch passed

+1 :green_heart:
compile
0m 22s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javac
0m 22s

the patch passed

+1 :green_heart:
compile
0m 22s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 22s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
0m 24s

the patch passed

+1 :green_heart:
javadoc
0m 23s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
0m 22s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
22m 10s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 31s

hadoop-project in the patch passed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

89m  1s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4637

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint

uname
Linux becb6109712b 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c191f50d8d902d4aac050217d3eb5046aac98080

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/1/testReport/

Max. process+thread count
750 (vs. ulimit of 5500)

modules
C: hadoop-project U: hadoop-project

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Test failures
[ERROR] Errors:
[ERROR]   ITestS3AFailureHandling.testMultiObjectDeleteLargeNumKeys » OutOfMemory Direct...
[ERROR]   ITestS3ADeleteFilesOneByOne.testBulkRenameAndDelete » OutOfMemory Direct buffe...
[ERROR]   ITestS3SelectLandsat.testSelectSeekFullLandsat:427->AbstractS3SelectTest.seek:701 » TestTimedOut
[INFO]

OOM in HADOOP-17937; ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete. This shows we would benefit from lazy creation of buffers.
ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier

[ERROR] Failures:
[ERROR]   ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier:129->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0001/test/testCreateAndCopyObjectWithStorageClassGlacier/file1]
Expecting:
 <null>
to be equal to:
 <"glacier">
ignoring case considerations
[ERROR]   ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassReducedRedundancy:104->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0001/test/testCreateAndCopyObjectWithStorageClassReducedRedundancy/file1]
Expecting:
 <null>
to be equal to:
 <"reduced_redundancy">
ignoring case considerations

I will need to see if this is a regression. I believe i have disabled setting storage class.
* [ERROR] testSelectSeekFullLandsat(org.apache.hadoop.fs.s3a.select.ITestS3SelectLandsat)  Time elapsed: 600.013 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 600000 milliseconds

CLI tests good. but they are all logging a warning about unset storage class. Filed HADOOP-18371

setup failure in ILoadTestS3ABulkDeleteThrottling; filed HADOOP-18372
[INFO] Running org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[ERROR] Tests run: 12, Failures: 12, Errors: 0, Skipped: 0, Time elapsed: 10.114 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling
[ERROR] test_010_Reset[bulk-delete-aws-retry=false-requests=100-size=250](org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling)  Time elapsed: 2.291 s  <<< FAILURE!
java.lang.AssertionError:
[page size]
Expecting actual not to be null
        at org.apache.hadoop.fs.s3a.scale.ILoadTestS3ABulkDeleteThrottling.setup(ILoadTestS3ABulkDeleteThrottling.java:175)

test failures occur on trunk even without sdk upgrade
[ERROR] Failures:
[ERROR]   ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier:129->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0003/test/testCreateAndCopyObjectWithStorageClassGlacier/file1]
Expecting:
 <null>
to be equal to:
 <"glacier">
ignoring case considerations
[ERROR]   ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassReducedRedundancy:104->assertObjectHasStorageClass:215 [Storage class of object s3a://stevel-london/fork-0003/test/testCreateAndCopyObjectWithStorageClassReducedRedundancy/file1]
Expecting:
 <null>
to be equal to:
 <"reduced_redundancy">
ignoring case considerations
[ERROR] Errors:
[ERROR]   ITestS3ADeleteFilesOneByOne.testBulkRenameAndDelete » OutOfMemory Direct buffe...
[ERROR]   ITestS3SelectLandsat.testSelectSeekFullLandsat:427->AbstractS3SelectTest.seek:701 » TestTimedOut
[INFO]

ILoadTestS3ABulkDeleteThrottling (also failing without the upgrade) is fixed in https://github.com/apache/hadoop/pull/4642

the storage class tests all work when i switch from bytebuffer as the upload buffering, as does the OOM.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  3s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 52s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
28m 10s

trunk passed

+1 :green_heart:
compile
24m 36s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
compile
20m 46s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
mvnsite
2m 41s

trunk passed

+1 :green_heart:
javadoc
2m 24s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
2m 27s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
117m 43s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 50s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m  0s

the patch passed

+1 :green_heart:
compile
24m  6s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javac
24m  6s

the patch passed

+1 :green_heart:
compile
22m 46s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
22m 46s

the patch passed

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
mvnsite
2m 19s

the patch passed

+1 :green_heart:
javadoc
1m 57s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
2m  9s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
32m 22s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m  3s

hadoop-project in the patch passed.

+1 :green_heart:
unit
3m 21s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m 17s

The patch does not generate ASF License warnings.

207m 35s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4637

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint

uname
Linux 4d5009939df0 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 9800dd2f1d5ee5d6fd728aae748f6dd7d7897faa

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/2/testReport/

Max. process+thread count
719 (vs. ulimit of 5500)

modules
C: hadoop-project hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/2/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Okay looks good. running the tests once complete will give a +1

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  5s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m  4s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
28m  2s

trunk passed

+1 :green_heart:
compile
26m 24s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
compile
21m 47s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
mvnsite
2m 21s

trunk passed

+1 :green_heart:
javadoc
1m 52s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
2m  9s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
119m  5s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 47s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m  3s

the patch passed

+1 :green_heart:
compile
22m 27s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javac
22m 27s

the patch passed

+1 :green_heart:
compile
20m 55s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
20m 55s

the patch passed

-1 :x:
blanks
0m  0s
/blanks-eol.txt
The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply

+1 :green_heart:
mvnsite
2m 39s

the patch passed

+1 :green_heart:
javadoc
2m 21s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
2m 28s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
33m  5s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m  9s

hadoop-project in the patch passed.

+1 :green_heart:
unit
3m 22s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m 37s

The patch does not generate ASF License warnings.

206m 49s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4637

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint

uname
Linux 0a1dfd256d92 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 7e1b620d2fea3a4003a80cdddc5aa14f0812e388

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/3/testReport/

Max. process+thread count
648 (vs. ulimit of 5500)

modules
C: hadoop-project hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/3/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

aah, forgot that license-binary change. pushed up a followup, and for the backports merged the two changes

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 56s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
15m 18s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
26m  2s

trunk passed

+1 :green_heart:
compile
24m 50s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
compile
22m 13s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
mvnsite
2m 36s

trunk passed

+1 :green_heart:
javadoc
2m 23s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
2m  8s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
117m  3s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 47s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 59s

the patch passed

+1 :green_heart:
compile
25m 15s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javac
25m 15s

the patch passed

+1 :green_heart:
compile
22m 42s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
22m 42s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
2m 12s

the patch passed

+1 :green_heart:
javadoc
1m 48s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1

+1 :green_heart:
javadoc
1m 59s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
shadedclient
30m 32s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m  8s

hadoop-project in the patch passed.

+1 :green_heart:
unit
3m 20s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m 35s

The patch does not generate ASF License warnings.

206m 30s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4637

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint

uname
Linux 58a8f285c8fd 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e67b4e8885b5ae8a0cb9092a6850c09ade99adbb

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/4/testReport/

Max. process+thread count
698 (vs. ulimit of 5500)

modules
C: hadoop-project hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4637/4/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

authors: ahmarsuhail hadoop-yetus mukund-thakur steveloughran


text:
HADOOP-18543. AliyunOSSFileSystem#open(Path path, int bufferSize) use buffer size as its downloadPartSize

Description of PR
In our application, different components have their own suitable buffer size to download.
But currently, AliyunOSSFileSystem#open(Path path, int bufferSize) just get downloadPartSize from configuration.
We cannnot use different value for different components in our programs.
I think we should the method should use the buffer size from the paramater.
AliyunOSSFileSystem#open(Path path) could have default value as current default downloadPartSize.
How was this patch tested?
Added method TestAliyunOSSInputStream#testConfiguration to test.
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[x] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  4s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
41m 19s

trunk passed

+1 :green_heart:
compile
0m 39s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
0m 44s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
0m 39s

trunk passed

+1 :green_heart:
mvnsite
0m 45s

trunk passed

+1 :green_heart:
javadoc
0m 46s

trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 45s

trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
1m  6s

trunk passed

+1 :green_heart:
shadedclient
20m 47s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 24s

the patch passed

+1 :green_heart:
compile
0m 25s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
0m 25s

the patch passed

+1 :green_heart:
compile
0m 24s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 24s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 22s
/results-checkstyle-hadoop-tools_hadoop-aliyun.txt
hadoop-tools/hadoop-aliyun: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

+1 :green_heart:
mvnsite
0m 28s

the patch passed

+1 :green_heart:
javadoc
0m 27s

the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 26s

the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
0m 50s

the patch passed

+1 :green_heart:
shadedclient
20m 21s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 31s

hadoop-aliyun in the patch passed.

+1 :green_heart:
asflicense
0m 45s

The patch does not generate ASF License warnings.

96m 17s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5172/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5172

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 5bc2bfc76972 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 7580a7cb63c066cc56d2277a5f1829e207b072a3

Default Java
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5172/1/testReport/

Max. process+thread count
731 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5172/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 38s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
39m 59s

trunk passed

+1 :green_heart:
compile
0m 40s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
compile
0m 40s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
checkstyle
0m 36s

trunk passed

+1 :green_heart:
mvnsite
0m 37s

trunk passed

+1 :green_heart:
javadoc
0m 49s

trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javadoc
0m 42s

trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
1m  0s

trunk passed

+1 :green_heart:
shadedclient
20m 32s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 23s

the patch passed

+1 :green_heart:
compile
0m 23s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javac
0m 23s

the patch passed

+1 :green_heart:
compile
0m 25s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
javac
0m 25s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 22s
/results-checkstyle-hadoop-tools_hadoop-aliyun.txt
hadoop-tools/hadoop-aliyun: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)

+1 :green_heart:
mvnsite
0m 28s

the patch passed

+1 :green_heart:
javadoc
0m 22s

the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04

+1 :green_heart:
javadoc
0m 26s

the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08

+1 :green_heart:
spotbugs
0m 50s

the patch passed

+1 :green_heart:
shadedclient
20m 11s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
0m 31s

hadoop-aliyun in the patch passed.

+1 :green_heart:
asflicense
0m 41s

The patch does not generate ASF License warnings.

93m 23s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5172/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5172

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 2276ff2601e8 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 76ce3cb5dcd36949f916098aa8a58e29c6f7664a

Default Java
Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5172/2/testReport/

Max. process+thread count
679 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5172/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

sorry, but I'm going to say -1 to using the normal IO buffer size as the GET range. The default value of 4k is way too small even for parquet/orc reads, it will break all existing apps in performance terms: distcp, parquet library, avro, ORC, everything, as they all use the default value.

there is a configuration option for multipart download size, which is filesystem-wide. Not as flexible, but something everyone will expect to work.
If you want better control of read policy, buffer sizes etc, then this connector needs to implement openFile(), as s3a and abfs do. that will let you add a new option to specify the range for GET calls.

(note also that includes letting you declare read policy (whole-file, sequential, random, vectored....that can be used to change default block size too)

sorry, but I'm going to say -1 to using the normal IO buffer size as the GET range. The default value of 4k is way too small even for parquet/orc reads, it will break all existing apps in performance terms: distcp, parquet library, avro, ORC, everything, as they all use the default value.

there is a configuration option for multipart download size, which is filesystem-wide. Not as flexible, but something everyone will expect to work.
If you want better control of read policy, buffer sizes etc, then this connector needs to implement openFile(), as s3a and abfs do. that will let you add a new option to specify the range for GET calls.

Thanks a lot for your detailed explaination and suggestion.
I agree that it's better not to change the implementaion of current interface.
I'd like to make oss also implement openFile() in this pr as s3a does which could also meet our needs.
WDYT?

I'd like to make oss also implement openFile() in this pr as s3a does which could also meet our needs.

This is exactly what the API was designed for -to let people provide extra hints/options. to the object stores.
Do read the filesystem.md and related docs on the topic, and look at the abfs implementation as well as the s3a one.
If your implementation takes a FileStatus or length option, it can then skip the HEAD request on opening and save time and money. All the hadoop internal uses of openFile() do this. My own copies of the parquet and avro readers also do it for better cloud reading performance

authors: hadoop-yetus masteryhx steveloughran


text:
HADOOP-18679. Add API for bulk/paged object deletion

Initial pass at writing an API for bulk deletes,
targeting S3 and any store with paged delete support.
Minimal design of a RemoteIterator to provide the list of paths to delete; a progress report will be provided after pages are deleted so as to provide an update of files deleted, and a way for the application code to abort an ongoing delete -such as after a failure.
Aspects of implementation to make clear in markdown spec

no guarantee of page size being > 1; or constant through entire operation.
after progress callback requests abort, more operations may continue (should we include an "aborting" flag?)
no guarantee order of execution. implementations may shuffle paths before posting.
no expectation that parent directories will exist after the operation completes; if an object store needs to explicitly look for and create directory markers, that step will be omitted.
background option is a hint to prioritise over other write operations vs add an interval between pages/different page size
callback: guarantee of thread callback comes from or whether it blocks further work
any exception raised by the iterator is an unrecoverable failure. unsubmitted paths may/may not be submitted before reporting the failure
no timeouts on iterator next()/hasNext().

How was this patch tested?
No tests yet; working on API first.
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 29s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
31m 32s

trunk passed

+1 :green_heart:
compile
10m 44s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
9m 34s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
0m 51s

trunk passed

+1 :green_heart:
mvnsite
1m 11s

trunk passed

+1 :green_heart:
javadoc
0m 58s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 42s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 41s

trunk passed

+1 :green_heart:
shadedclient
22m 25s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 39s

the patch passed

+1 :green_heart:
compile
9m 49s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
9m 49s

the patch passed

+1 :green_heart:
compile
9m 33s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
9m 33s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 49s

the patch passed

+1 :green_heart:
mvnsite
1m  8s

the patch passed

+1 :green_heart:
javadoc
0m 54s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 42s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 45s

the patch passed

+1 :green_heart:
shadedclient
22m 19s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 49s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 50s

The patch does not generate ASF License warnings.

148m 24s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5993

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux f297046e6241 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / f25a930b8893f2c9f358fd839e4fb943d250c726

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/1/testReport/

Max. process+thread count
1253 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

writing up spec made me decide we should have a .opt to indicate when a bulk delete is a "background" operation, which may be executed at a rate to interfere less with live queries, e.g: smaller pages, rate limited buildup of pages, different throttle retry policy.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 34s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
30m 19s

trunk passed

+1 :green_heart:
compile
10m 29s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
compile
9m 28s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
checkstyle
0m 52s

trunk passed

+1 :green_heart:
mvnsite
1m 10s

trunk passed

+1 :green_heart:
javadoc
0m 59s

trunk passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 43s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 39s

trunk passed

-1 :x:
shadedclient
20m 56s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 40s

the patch passed

+1 :green_heart:
compile
9m 48s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javac
9m 48s

the patch passed

+1 :green_heart:
compile
9m 29s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
9m 29s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 49s
/results-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1 :green_heart:
mvnsite
1m 10s

the patch passed

+1 :green_heart:
javadoc
0m 52s

the patch passed with JDK Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04

+1 :green_heart:
javadoc
0m 42s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
spotbugs
1m 45s

the patch passed

-1 :x:
shadedclient
21m 16s

patch has errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 47s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

145m 56s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5993

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 7926fe6849b6 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 93174773ea7456127183541cc1c65d8435de98a4

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/2/testReport/

Max. process+thread count
1320 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ahmarsuhail

caller provides a remote iterator, such as the ones we do for listing or another source/transformation (see RemoteIterators)
build() call returns some result
implementation kicks off a worker thread to process the iterator, reading its values in until there's enough to kick off a DELETE request (page or maybe a parallel set in a thread pool)
after each page/set of deletes, invokes the supplied callback of results
then continues, unless told to stop
finish only on: iterator has nothing, iterator raises an exception
or maybe on reaching some limit on failures
including maybe those considered unrecoverable

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 21s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
13m 47s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m 28s

trunk passed

+1 :green_heart:
compile
8m 19s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
7m 27s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
2m  4s

trunk passed

+1 :green_heart:
mvnsite
1m 24s

trunk passed

+1 :green_heart:
javadoc
1m  4s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m  0s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
2m  9s

trunk passed

+1 :green_heart:
shadedclient
19m 50s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 20s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 50s

the patch passed

+1 :green_heart:
compile
7m 52s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
7m 52s

the patch passed

+1 :green_heart:
compile
7m 27s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
7m 27s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m 59s
/results-checkstyle-root.txt
root: The patch generated 5 new + 3 unchanged - 0 fixed = 8 total (was 3)

+1 :green_heart:
mvnsite
1m 18s

the patch passed

+1 :green_heart:
javadoc
1m  2s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 57s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
2m 20s

the patch passed

+1 :green_heart:
shadedclient
19m 45s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 22s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 10s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 36s

The patch does not generate ASF License warnings.

143m 28s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5993

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux e75e83010c54 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d69fac0192c14889f0b3aa62bdb76e1d196eec8c

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/3/testReport/

Max. process+thread count
2432 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 20s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 17s

Maven dependency ordering for branch

-1 :x:
mvninstall
4m 17s
/branch-mvninstall-root.txt
root in trunk failed.

-1 :x:
compile
3m 53s
/branch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in trunk failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
3m 36s
/branch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in trunk failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

+1 :green_heart:
checkstyle
1m 54s

trunk passed

+1 :green_heart:
mvnsite
1m 12s

trunk passed

+1 :green_heart:
javadoc
0m 50s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 37s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
1m 49s

trunk passed

-1 :x:
shadedclient
4m 52s

branch has errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 20s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
0m 45s

the patch passed

-1 :x:
compile
3m 48s
/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
javac
3m 48s
/patch-compile-root-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt
root in the patch failed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.

-1 :x:
compile
3m 36s
/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

-1 :x:
javac
3m 36s
/patch-compile-root-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt
root in the patch failed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m 48s
/results-checkstyle-root.txt
root: The patch generated 5 new + 3 unchanged - 0 fixed = 8 total (was 3)

+1 :green_heart:
mvnsite
1m  6s

the patch passed

+1 :green_heart:
javadoc
0m 39s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
0m 41s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
2m  1s

the patch passed

-1 :x:
shadedclient
4m 57s

patch has errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
16m 13s

hadoop-common in the patch passed.

+1 :green_heart:
unit
2m 24s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 24s

The patch does not generate ASF License warnings.

79m 47s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/5993

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux de98fcc15d5f 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / c7b4e99c212f8a3e60da456560c629be94326f76

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/4/testReport/

Max. process+thread count
3149 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5993/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

authors: hadoop-yetus steveloughran


text:
HADOOP-19045. CreateSession Timeout - followup

Remove all declarations of fs.s3a.connection.request.timeout in

hadoop-common/src/main/resources/core-default.xml
hadoop-aws/src/test/resources/core-site.xml

New test in TestAwsClientConfig to verify that the value defined in fs.s3a.Constants class is used.
This is brittle to someone overriding it in their test setups, but as this test is intended to verify that the option is not explicitly set, there's no workaround.
Contributed by Steve Loughran
For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

testing in progress

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 10s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
markdownlint
0m  0s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 49s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
31m 33s

trunk passed

+1 :green_heart:
compile
17m 53s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
16m 32s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
4m 43s

trunk passed

+1 :green_heart:
mvnsite
2m 27s

trunk passed

+1 :green_heart:
javadoc
1m 42s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 25s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
3m 52s

trunk passed

+1 :green_heart:
shadedclient
33m 47s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 28s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 28s

the patch passed

+1 :green_heart:
compile
17m  4s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
17m  4s

the patch passed

+1 :green_heart:
compile
16m 15s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
16m 15s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
4m 27s
/results-checkstyle-root.txt
root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1 :green_heart:
mvnsite
2m 23s

the patch passed

+1 :green_heart:
javadoc
1m 36s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 27s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
4m 14s

the patch passed

+1 :green_heart:
shadedclient
34m  4s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 30s

hadoop-common in the patch passed.

+1 :green_heart:
unit
3m 20s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 54s

The patch does not generate ASF License warnings.

243m 19s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6532/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint spotbugs checkstyle

uname
Linux 322964176d71 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d664389e715dde3461d2b607d3969a59120f3494

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6532/1/testReport/

Max. process+thread count
1253 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6532/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 33s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+0 :ok:
markdownlint
0m  1s

markdownlint was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m 53s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
30m 57s

trunk passed

+1 :green_heart:
compile
18m 13s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
16m 10s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
checkstyle
4m 35s

trunk passed

+1 :green_heart:
mvnsite
2m 31s

trunk passed

+1 :green_heart:
javadoc
1m 41s

trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 28s

trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
3m 54s

trunk passed

+1 :green_heart:
shadedclient
33m 50s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 29s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 25s

the patch passed

+1 :green_heart:
compile
17m 25s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
17m 25s

the patch passed

+1 :green_heart:
compile
16m 21s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
javac
16m 21s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
4m 34s
/results-checkstyle-root.txt
root: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1 :green_heart:
mvnsite
2m 26s

the patch passed

+1 :green_heart:
javadoc
1m 34s

the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
1m 30s

the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08

+1 :green_heart:
spotbugs
4m 14s

the patch passed

+1 :green_heart:
shadedclient
33m 57s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 43s

hadoop-common in the patch passed.

+1 :green_heart:
unit
3m 21s

hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
1m  0s

The patch does not generate ASF License warnings.

242m 38s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6532/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint markdownlint spotbugs checkstyle

uname
Linux 60f63f6cde82 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d664389e715dde3461d2b607d3969a59120f3494

Default Java
Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6532/2/testReport/

Max. process+thread count
1349 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6532/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.