text:
Kafka ingestion ignores segment granularities that aren't "simple" granularities

Affected Version
0.20.0
Description
This is the granularity spec being used in a kafka ingestion task
"granularitySpec": {
        "type": "uniform",
        "segmentGranularity": {"type": "period", "period": "PT3H"},
        "queryGranularity": "NONE",
        "rollup": false
}

The taskDuration is also set to 3H or more. But druid is creating segments for each hour. For example, this is one of the segment ids: mydatasource_2021-04-28T20:00:00.000Z_2021-04-28T21:00:00.000Z_2021-04-28T20:03:38.441Z. The max total rows or the max rows per segment limits are not being hit. Even if those limits were being hit, I'd still expect the segment to be marked for 3 hours and more partitions to get created as more data comes in for the same time period.
The same thing happens even with an equivalent duration granularity (10800000).
The documentation states that "Any granularity can be provided here." with a link to the query granularities page. So it seems like either this is expected behaviour and the documentation is incorrect or this is a bug and what the documentation says is the expected behaviour.
After unsuccessfully trying to get a response on an older issue #6170 (https://github.com/apache/druid/issues/6170#issuecomment-815946739, https://github.com/apache/druid/issues/6170#issuecomment-817947549, https://github.com/apache/druid/issues/6170#issuecomment-821286942) that seems to be the same bug and on the google group too, I am opening this new issue.

Bump

For append tasks such as Kafka indexing, the segment identifier is identified using the allocate API of the overlord. This is different from the overwrite task behavior since append tasks cannot change the segment granularity of the existing segments.
Therefore during the allocate action in the overlord, it will initially try to find an existing segment which can be used to add the current row. If that's not possible, it will try to fit the current row into one of the predefined granularities that is closest to the segment granularity defined in the ingestion spec. In your case, the closest predefined granularity finer than 3H would be HOUR which would be why an hourly segment got created in your case. It looks like the relevant documentation needs to be clarified in this aspect.

Does that mean that I can only use one of the predefined granularities?
And if so, how is one supposed to achieve segment size optimization?
Does compaction task support non-predefined granularities? Even if it does, that seems like an unnecessary overhead when one could have ingested the data with the proper granularity in the first place.
If it doesn't, we're back to the question, how to optimize?

Looking at SegmentAllocateAction.tryAllocateFirstSegment, it looks like only the predefined granularities would be used. Any other granularities would be treated as explained earlier. Thinking about it, instead of fitting it into one of the closest predefined granularities, the allocate API should be able to accommodate the user specified segment granularity  if there are no existing segments for the current row. Not sure why the design is as such. Maybe @jihoonson  can help clarify that.

Does compaction task support non-predefined granularities?

Compaction is an overwrite task and yes it can support the non-predefined granularity specified.

Hey @scimas, sorry for the late response. I think the current workaround is setting up a compaction service outside Druid that periodically issues the compaction job. https://github.com/apache/druid/pull/10843 will allow you to change segment granularity using auto compaction, but will be included in our next major release.

Thinking about it, instead of fitting it into one of the closest predefined granularities, the allocate API should be able to accommodate the user specified segment granularity if there are no existing segments for the current row. Not sure why the design is as such.

This code was written a long time ago, and I don't know why it does this. But I agree. I don't see any reason that it behaves this way.

authors: a2l007 jihoonson scimas

repo: apache/druid


text:
TIMESTAMP (value), MILLIS_TO_TIMESTAMP()/TIMESTAMP_TO_MILLIS() not honoring sqlTimeZone

Affected Version
25.0.0
Description
I have a date/time value converted via TIMESTAMP_TO_MILLIS() on ingestion.  When I use MILLIS_TO_TIMESTAMP() at query time with the sqlTimeZone query context parameter set, the TIMESTAMP returned is still in UTC time.
I believe that a timestamp comparison using BETWEEN is also failing because it is comparing timestamps of two different time zones incorrectly.
The documentation reads:
Sets the time zone for this connection, which will affect how time functions and timestamp literals behave.

This is inaccurate, the dates were being truncated to day and stored as midnight in UTC. When converted according to sqlTimeZone it was correctly converting the incorrectly stored time.

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
complex metrics -> complex types

Description
This PR rebrands ComplexMetricSerde as ComplexTypeSerde, ComplexMetrics as ComplexTypes, and ComplexMetricExtractor as ComplexTypeExtractor. All of the former are marked as deprecated, and their logic moved to (and implement) the latter to minimize disruption. All current serdes still implement ComplexMetricSerde rather than the new type, to allow anything using ComplexMetrics.getSerdeForType to continue functioning until their future removal, at which point the implementations shall be moved to extend ComplexTypeSerde directly.
Additionally, it adds a method to ComplexTypeExtractor, coerceValue, to use to allow COMPLEX types to coerce values to the appropriate type when read from a column value selector from RowBasedColumnSelectorFactory. This allows COMPLEX typed dimensions to perform value coercion, similar to the other use of the extractValue method when reading COMPLEX metrics for queries against IncrementalIndex.

This PR has:

[x] been self-reviewed.
[x] added documentation for new or modified features or behaviors.
[x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[x] been tested in a test Druid cluster.

btw, I tested that this change is minimally disruptive by doing some tests using extensions (datasketches specifically, for my complex types) built against the master branch without the changes in this PR, and everything was still functioning correctly as far as I can tell, so I think it should be relatively chill despite changing stuff marked @ExtensionPoint.

source: gharchive/pull-request


text:
请问 2.7.0 以后的dubbo不在使用 接口禁用功能了吗

2.7.7还是可以禁用的 但是之前的注解被弃用了 换Dubbo* 注解了

2.7.6 没有更改 注解 且 可以使用禁用

经过 多次测试发现 2.7.6 在 只有一个service 服务时 禁用 是不起作用的，两个 和以上 就有作用，相当的人性化，避免了 测试完后 忘记取消禁用 导致的服务器调用异常

repo: apache/dubbo-admin


text:
Bmap在本初子午线附近只能显示一边的scatter点以及俄罗斯东部部分领土无法渲染

Version
5.0.0
Steps to reproduce
使用bmap的demo【https://echarts.apache.org/next/examples/zh/editor.html?c=effectScatter-bmap 】 ，将data和geoCoordMap替换成如下
var data=[
{name:"Ashburn", value:17},
{name:"Beijing", value:26},
{name:"Chengdu", value:1},
{name:"Hangzhou", value:11},
{name:"Iron", value:1},
{name:"Moscow", value:1},
{name:"Shanghai", value:81},
{name:"Timişoara", value:2},
{name:"Vilnius", value:3},
{name:"Xiamen", value:44},
];
var geoCoordMap={
"Ashburn":[-77.4875,39.0437],
"Beijing":[116.3972,39.9075],
"Chengdu":[104.0667,30.6667],
"Hangzhou":[120.1614,30.2936],
"Iron":[-88.0660,45.8202],
"Moscow":[37.6156,55.7522],
"Shanghai":[121.4581,31.2222],
"Timişoara":[21.2257,45.7537],
"Vilnius":[25.2798,54.6892],
"Xiamen":[118.0819,24.4798],
};
What is expected?
①当我的中心在太平洋时，且zoom得可见中国和美国时，位于中国和美国的数据点都能现实。
②俄罗斯的领土是完整显示的。
What is actually happening?
①当位于本处子午线向东边滑动时，发现美国的点是丢失的；但是当滑动超过了本初子午线，中国的数据点就丢失，但是美国的数据点就会显示。

这是还在本初子午线左边，所以只显示了中国的数据点

这是过了本初子午线，到了右边，所以只显示美国的数据点。
②白令海西北角，部分俄罗斯领土没有显示，没有渲染。这个是在SVG和CANVAS都有，firefox和chrome都有。请问应该怎么解决。

我在本地重复这个图和在demo上直接改数据都是一样的，这两个问题都存在。请问应该怎么样解决呢？

应该是跨了半球导致的，可以试试原生的百度地图 Marker 是否有此问题。

应该是跨了半球导致的，可以试试原生的百度地图 Marker 是否有此问题。

@ytsxstone 这个是百度地图问题，ECharts 无法解决。

authors: HamletShaoE plainheart

repo: apache/echarts


text:
隐藏图例时，希望轴单位也自动隐藏

Version
5.3.2
Link to Minimal Reproduction
No response
Steps to Reproduce
点击图例隐藏图表，轴单位未隐藏
Current Behavior

隐藏图例的时候，轴的单位未自动隐藏
Expected Behavior
期望隐藏图例的时候，轴的单位也隐藏
Environment
- OS:
- Browser:
- Framework:

Any additional comments?
No response

Come now
On Thursday, May 12, 2022, echarts-bot[bot] @.***>
wrote:

Closed #17020 https://github.com/apache/echarts/issues/17020.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/echarts/issues/17020#event-6595977122, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AQXZROMY6LKL7U25CNROCPDVJS2MBANCNFSM5VXKYCNQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Yes
On Thursday, May 12, 2022, echarts-bot[bot] @.***>
wrote:

@YXYCZXY https://github.com/YXYCZXY It seems you are not using English,
I've helped translate the content automatically. To make your issue
understood by more people and get helped, we'd like to suggest using
English next time. 🤗
TRANSLATED
TITLE
When hiding the legend, want the axis units to also be automatically hidden
BODY
Version
5.3.2
Link to Minimal Reproduction
No response
Steps to Reproduce
Click the legend to hide the chart, the axis units are not hidden
Current Behavior
[image: image]
https://user-images.githubusercontent.com/49504688/168019976-c963c358-4008-438c-9a15-27d294a84bce.png
When you hide a legend, the units of the axis are not automatically hidden
Expected Behavior
When you expect to hide the legend, the units of the axis are also hidden
Environment

OS:
Browser:
Framework:

Any additional comments?
No response
—
Reply to this email directly, view it on GitHub
https://github.com/apache/echarts/issues/17020#issuecomment-1124651587,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQXZROND5BZRCZQ55X5JGQLVJS2PNANCNFSM5VXKYCNQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Yes
On Thursday, May 12, 2022, Brian @.***> wrote:

Yes
On Thursday, May 12, 2022, echarts-bot[bot] @.***>
wrote:

@YXYCZXY https://github.com/YXYCZXY It seems you are not using
English, I've helped translate the content automatically. To make your
issue understood by more people and get helped, we'd like to suggest using
English next time. 🤗
TRANSLATED
TITLE
When hiding the legend, want the axis units to also be automatically
hidden
BODY
Version
5.3.2
Link to Minimal Reproduction
No response
Steps to Reproduce
Click the legend to hide the chart, the axis units are not hidden
Current Behavior
[image: image]
https://user-images.githubusercontent.com/49504688/168019976-c963c358-4008-438c-9a15-27d294a84bce.png
When you hide a legend, the units of the axis are not automatically hidden
Expected Behavior
When you expect to hide the legend, the units of the axis are also hidden
Environment

OS:
Browser:
Framework:

Any additional comments?
No response
—
Reply to this email directly, view it on GitHub
https://github.com/apache/echarts/issues/17020#issuecomment-1124651587,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQXZROND5BZRCZQ55X5JGQLVJS2PNANCNFSM5VXKYCNQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

可以使用legendselected事件去操控你的option

使用legendselected 可以去烧烤你的选项

这个我知道这样的确可以解决，但是echarts自带的隐藏图例的时候可以把对应轴的label给隐藏，那为什么不也隐藏轴的单位呢？
我觉得这个是一个bug

使用legendselected 可以去烧烤你的选项

这个我知道这样的确可以解决，但是echarts自带的隐藏图例的时候可以把对应轴的label给隐藏，那为什么不也隐藏轴的单位呢？ 我觉得这个是一个bug

这个是看个人需求的吧，有的需求会要求显示y轴的名称，我们在制作图表的时候是需要根据自己的需求二次封装的

authors: YXYCZXY nelsonbbarriento23 zengyuehao


text:
[Bug] Height Exceeds 65535px Error

Version
5.4.1
Link to Minimal Reproduction
https://codesandbox.io/s/basic-bar-forked-74z08m?file=/style.css
Steps to Reproduce
1.Use the following link to create, has error example
https://codesandbox.io/s/basic-bar-forked-74z08m?file=/style.css
2.Replacing [height: 65536px] for [height: 1000px], Back to normal.
Current Behavior
ECharts height exceeds 65535px will report an error
some computers in 59353px will report an error

Expected Behavior
Expected

Environment
- OS: Windows 10
- Browser: Chrome 108.0.5359.125
- Framework:

Any additional comments?
No response

This is up to the limit of the browser, and it's not related to ECharts.


text:
[hotfix][docs] fix mistakes in Modules document

What is the purpose of the change
fix mistakes in module document from Catalogs to Modules.

@JingsongLi @bowenli86
Have a look  when you guys are free😀

repo: apache/flink


text:
[FLINK-23528][kinesis] Do not interrupt main thread during cancellation.

What is the purpose of the change
FlinkKinesisConsumer#cancel() (inside KinesisDataFetcher#shutdownFetcher()) shouldn't be interrupting source thread. Otherwise, as described in FLINK-23527, network stack can be left in an invalid state and downstream tasks can encounter deserialisation errors.
Brief change log

Use a future for a soft wakeup during cancellation.

Verifying this change
Added a test that verifies that cancellation still works. Testing that no InterruptedException is thrown proved to be impossible.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

LGTM subject to e2e tests passing
✅  1. The [description] looks good.
✅  2. There is [consensus] that the contribution should go into to Flink.
✅  3. Needs [attention] from. N/A
✅  4. The change fits into the overall [architecture].
✅  5. Overall code [quality] is good.
✅  6. License check N/A
⏲  7. E2e tests pass
✅  8. Commit contains the Jira ID

authors: AHeise dannycranmer


text:
[FLINK-31836][table] Upgrade Calcite version to 1.34.0

Based on the following PR:
[FLINK-31362][table] Upgrade Calcite version to 1.33.0
What is the purpose of the change
This PR is about to upgrade Calcite to 1.34.0
once upgrade to 1.33.0 will be approved and merged I will rebase this one
Since 1.34.0 was a short release the upgrade is relatively simple, so only the latest commit is for 1.34.0 upgrade
Verifying this change
This change is already covered by existing tests
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): ( yes)
The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
The serializers: ( no)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no )
The S3 file system connector: ( no)

Documentation

Does this pull request introduce a new feature? ( no)
If yes, how is the feature documented? (not applicable)

@flinkbot run azure

@flinkbot run azure


text:
[FLINK-12820][Connectors / Cassandra] Support ignoring writing nulls

What is the purpose of the change
Added support to ignore null fields when writing to c* for tuple types.
Brief change log

Added 'ignoreNullFields' config parameter which can be enabled
Unset nulls from bound statement.

Verifying this change
This change added tests and can be verified as follows:
Added a test case to CassandraConnectorITCase testing partial column updates
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (JavaDocs)

@flinkbot approve-until architecture

Overall approach looks good.
I would ask you to switch to use primitive booleans rather than boxed booleans, to be consistent with the approach in the remainder of Flink. This also avoid accidental nullability and late null pointer exceptions.
Otherwise looks good to me.

@flinkbot approve all

Looks good to me, merging this...

authors: StephanEwen ozancicek


text:
[FLINK-2456][fix]add a repository ID for hadoop2 in flink-hbase module

add a repository for hadoop-2 in flink-hbase.
It can be downloaded through a specified mirror like:

repo2
Human Readable Name for this Mirror.
http://repo2.maven.org/maven2/
hadoop-2-repo2

Or it will not affect the normal network environment when default.

Thanks for providing this path!
Can you help us understand this fix a bit? Why is this extra repository entry required for the HBase module, and not for any other modules?

This is just a method to solve the problem that some modules' dependencies cannot be downloaded from central maven repository.
You can specify every module dependency ID so that we can downloading dependencies from either central maven repository by default or other channel by setting maven mirrors when mvn package.
As for HBase, it`s just that I get some trouble in downloading this module dependency.
And sorry for the url.
I just want to push a example about maven mirror on the comment but it was blocked because of the XML format.
This complete code should be like this:
"<mirror>"
    "<id>repo2</id>"
    "<name>Human Readable Name for this Mirror.</name>"
    "<url>http://repo2.maven.org/maven2/</url>"
"<mirrorOf>hadoop-2-repo2</mirrorOf>"
"</mirror>"

The repository you add is the default Maven repository - it should be used by default anyways.
Is the change there override your mirror repository for HBase with the default repository?

The most important In the HBase pom.xml I add is the repository ID:
""
"	"
"		hadoop-2-repo2"
"					https://repo.maven.apache.org/maven2"
"					"
"						true"
"					"
"					"
"						false"
"					"
"				"
"			"
By using this id, you can get the hadoop-2 from any way(for example using mirror with this ID).
The default repository is the url(https://repo.maven.apache.org/maven2) above.
And the default means you need not do anything with maven configuration just print "mvn package".
As I see, every flink dependency is using this url.
Did I misunderstand? Is the hadoop-2 default repository path is another?

The most important In the HBase pom.xml I add is the repository ID:
""
""
"hadoop-2-repo2"
"https://repo.maven.apache.org/maven2"
""
"true"
""
""
"false"
""
""
""
By using this id, you can get the hadoop-2 from any way(for example using mirror with this ID).
The default repository is the url(https://repo.maven.apache.org/maven2) above.
And the default means you need not do anything with maven configuration just print "mvn package".
As I see, every flink dependency is using this url.
Did I misunderstand? Is the hadoop-2 default repository path is another?

The most important in the Flink-hbase pom.xml I add is the repository ID:
""
By using this id, you can get the hadoop-2 from any way(for example using mirror with this ID).
The default repository is the usl(https://repo.maven.apache.org/maven2) above.
And the default means you need not do anything with maven configuration just print "mvn pakcage".
As I see, every flink dependency is using this url.
Did I misunderstand?Is the hadoop-2 default repository path is another?

The most important in the Flink-hbase pom.xml I add is the repository ID:

By using this id, you can get the hadoop-2 from any way(for example using mirror with this ID).
The default repository is the usl(https://repo.maven.apache.org/maven2) above.
And the default means you need not do anything with maven configuration just print "mvn pakcage".
As I see, every flink dependency is using this url.
Did I misunderstand?Is the hadoop-2 default repository path is another?

I understand.
Will merge this...

Thank you for taking the time to explain the reason behind this issue...

My plessure.

authors: HuangWHWHW StephanEwen


text:
GEODE-6768: "Connection disconnect detected by EOF" will be logged as…

… warning message and not info message
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[X] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[X] Is your initial contribution a single, squashed commit?

[X] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

Hello
I see that "concourse-ci/UpgradeTestOpenJDK11 " check has failed earlier and now with an empty commit, it is in pending state. This check has nothing to do with my changes. Please review.

repo: apache/geode


text:
GEODE-6717: Logging out when all JMX connections are closed

Co-authored-by: mkevo
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[ ] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[ ] Is your initial contribution a single, squashed commit?

[ ] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

@mkevo

Hi @jinmeiliao, thank you for a help.
I tested it, and it works great. We can close previous PR #3811

authors: jinmeiliao mkevo


text:
GEODE-9492: Ignore default port test in stress tests

PROBLEM
GeodeRedisServerStartupDUnitTest.startupOnDefaultPort() tests the
system's use of a fixed port. Repeat test tasks run multiple instances
of this test concurrently. Most of the concurrently running instances of
the test are likely to fail when they find that port already in use.
SOLUTION

Add an IgnoreInRepeatTestTasks test category.
Configure repeat tests to exclude tests with the
IgnoreInRepeatTestTasks category.
Add the IgnoreInRepeatTestTasks category to
GeodeRedisServerStartupDUnitTest.startupOnDefaultPort()

That seems reasonable. Odd that this is the first we have seen of this kind of problem. One would have thought other default ports would have this problem. But perhaps, this test is flawed. Perhaps breaking it into a test of the default port number and a test of the starting up on a specified port would be more appropriate then altering our test framework?

@mhansonp I think it started appearing because Dale removed use of Docker for these tests.

Odd that this is the first we have seen of this kind of problem. One would have thought other default ports would have this problem.

It's a consequence of two things. First, we are no longer dockerizing test workers. Until a few weeks ago, each test worker ran in a separate docker container, and within each container fewer processes were competing for ports. Now that dozens of concurrently running tests compete for the same ports, we will see more port conflicts like this.
Second, very few tests use default ports, so there are likely to be few conflicts… except in stress tests, which pit multiple instances of a test against each other.

But perhaps, this test is flawed. Perhaps breaking it into a test of the default port number and a test of the starting up on a specified port would be more appropriate then altering our test framework?

I chose this approach because it affects only tests that explicitly test the use of default ports. And my guess (or perhaps merely my hope) is that very few tests do that: Only one per service. So very few tests will use this technique to avoid stress. I hope.
As for altering the test framework, my assessment (for now) is that including these few tests in stress test runs isn't worth the extra complexity in the test framework.

As for altering the test framework, my assessment (for now) is that including these few tests in stress test runs isn't worth the extra complexity in the test framework.

I suggested that we modify this test to be repeated test capable rather than adding a new category which you have done.
E.g. Pulling the default port in into a default port test. Then modify this code to test that it works fine with a configured port. I am not very attached to this idea, but it seems cleaner. Because then the redis code will get repeat testing.

I suggested that we modify this test to be repeated test capable rather than adding a new category which you have done.

Ah. I misread "then" as meaning "and then," and my misreading changed the entire meaning of what you wrote. Sorry about that.

authors: demery-pivotal kirklund mhansonp


text:
[Improvement] Support Trino-446 version

What would you like to be improved?
Gravitino connector

Property
Type
Default Value

connector.name
string
dlable

gravitino.metalake
string
xxxxxx

gravitino.uri
string
http://localhost:8090

ReasonWhen using the latest Trino v446 version supported by JDK21 (LTS), after configuring the above-mentioned connector, the following startup error occurs:Unsupported Trino-446 version. The supported version for the Gravitino-Trino-connector is from Trino-435 to Trino-439.Could the community consider supporting the latest version of the Trino connector?
How should we improve?
Would you like to support the lastest version of Trino?

@jerryshao @shaofengshi  Should we prioritize supporting this requirement?

I think we don't support newer version of Trino, neither do JDK 21. This is not a trivial work, we need to think more about it.

authors: diqiu50 jerryshao xxzhky

repo: apache/gravitino


text:
HDDS-1716. Smoketest results are generated with an internal user

[~eyang] reported the problem in HDDS-1609 that the smoketest results are generated a user (the user inside the docker container) which can be different from the host user.
There is a minimal risk that the test results can be deleted/corrupted by an other users if the current user is different from uid=1000
I opened this issue because [~eyang] said me during an offline discussion that HDDS-1609 is a more complex issue and not only about the ownership of the test results.
I suggest to handle the two problems in different way. With this patch, the permission of the test result files can be fixed easily.
In HDDS-1609 we can discuss about general security problems and try to find generic solution for them.
Steps to reproduce this the problem:
Use a user which is different from uid=1000
Create a new ozone build (mvn clean install -f pom.ozone.xml -DskipTests)
Go to a compose directory (cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/)
Execute tests (./test.sh)
check the ownership of the results (ls -lah ./results)
Current result: the owner of the result files are the user uid=1000
Expected result: the owner of the files should be always the current user (even if the current uid is different)

See: https://issues.apache.org/jira/browse/HDDS-1716

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
72
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

0
shelldocs
0
Shelldocs was not available.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
494
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
959
branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1
mvninstall
535
the patch passed

+1
mvnsite
0
the patch passed

-1
shellcheck
0
The patch generated 6 new + 1 unchanged - 0 fixed = 7 total (was 1)

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
813
patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1
unit
114
hadoop-hdds in the patch passed.

+1
unit
191
hadoop-ozone in the patch passed.

+1
asflicense
48
The patch does not generate ASF License warnings.

3435

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1002

Optional Tests
dupname asflicense mvnsite unit shellcheck shelldocs

uname
Linux 66b9ae8a8755 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 83dcb9d

shellcheck
https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/1/artifact/out/diff-patch-shellcheck.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/1/testReport/

Max. process+thread count
306 (vs. ulimit of 5500)

modules
C: hadoop-ozone/dist U: hadoop-ozone/dist

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/1/console

versions
git=2.7.4 maven=3.3.9 shellcheck=0.4.6

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
70
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

0
shelldocs
1
Shelldocs was not available.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
474
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
839
branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1
mvninstall
437
the patch passed

+1
mvnsite
0
the patch passed

+1
shellcheck
2
There were no new shellcheck issues.

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
745
patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1
unit
111
hadoop-hdds in the patch passed.

+1
unit
187
hadoop-ozone in the patch passed.

+1
asflicense
49
The patch does not generate ASF License warnings.

3112

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1002

Optional Tests
dupname asflicense mvnsite unit shellcheck shelldocs

uname
Linux 3dbb9f3b1d3f 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / d203045

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/2/testReport/

Max. process+thread count
307 (vs. ulimit of 5500)

modules
C: hadoop-ozone/dist U: hadoop-ozone/dist

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/2/console

versions
git=2.7.4 maven=3.3.9 shellcheck=0.4.6

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
74
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

0
shelldocs
1
Shelldocs was not available.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
514
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
831
branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1
mvninstall
434
the patch passed

+1
mvnsite
0
the patch passed

+1
shellcheck
2
There were no new shellcheck issues.

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
730
patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1
unit
99
hadoop-hdds in the patch passed.

+1
unit
170
hadoop-ozone in the patch passed.

+1
asflicense
43
The patch does not generate ASF License warnings.

3084

Subsystem
Report/Notes

Docker
Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1002

Optional Tests
dupname asflicense mvnsite unit shellcheck shelldocs

uname
Linux 9837b950f400 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 564758a

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/3/testReport/

Max. process+thread count
306 (vs. ulimit of 5500)

modules
C: hadoop-ozone/dist U: hadoop-ozone/dist

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1002/3/console

versions
git=2.7.4 maven=3.3.9 shellcheck=0.4.6

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

authors: elek hadoop-yetus

repo: apache/hadoop


text:
HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete

After HDFS-13101, it was observed in one of our customer deployments that delete snapshot ends up cleaning up inodes from active fs which can be referred from only one snapshot as the isLastReference() check for the parent dir introduced in HDFS-13101 may return true in certain cases. The aim of this Jira to add a check to ensure if the Inodes are being referred in the active fs , should not get deleted while deletion of snapshot happens.

/retest

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
26m 33s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
21m 43s
trunk passed

+1 :green_heart:
compile
1m  8s
trunk passed

+1 :green_heart:
checkstyle
0m 51s
trunk passed

+1 :green_heart:
mvnsite
1m 13s
trunk passed

+1 :green_heart:
shadedclient
17m 30s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 42s
trunk passed

+0 :ok:
spotbugs
3m  3s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m  1s
trunk passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  6s
the patch passed

+1 :green_heart:
compile
1m  3s
the patch passed

+1 :green_heart:
javac
1m  3s
the patch passed

-0 :warning:
checkstyle
0m 43s
hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 218 unchanged - 0 fixed = 221 total (was 218)

+1 :green_heart:
mvnsite
1m  9s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
shadedclient
15m 14s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 38s
the patch passed

+1 :green_heart:
findbugs
3m  4s
the patch passed

_ Other Tests _

-1 :x:
unit
107m  2s
hadoop-hdfs in the patch passed.

+1 :green_heart:
asflicense
0m 35s
The patch does not generate ASF License warnings.

204m 23s

Reason
Tests

Failed junit tests
hadoop.TestRefreshCallQueue

hadoop.hdfs.TestRollingUpgrade

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1989/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1989

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux e5b15b11f2f0 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / b5b45c53a4e

Default Java
Private Build-1.8.0_252-8u252-b09-1~18.04-b09

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1989/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1989/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1989/1/testReport/

Max. process+thread count
2914 (vs. ulimit of 5500)

modules
C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1989/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

authors: bshashikant hadoop-yetus


text:
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics.

Part of HADOOP-16830 IOStatistics Support.
Contributed by Steve Loughran.
Tests are in hadoop-aws so excluded here
full PR already reviewed by @bgaborg in #2553; this is the feature split up by module for better isolation/backport

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 30s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
32m 59s

trunk passed

+1 :green_heart:
compile
0m 40s

trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
compile
0m 36s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
checkstyle
0m 29s

trunk passed

+1 :green_heart:
mvnsite
0m 43s

trunk passed

+1 :green_heart:
shadedclient
16m 16s

branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 26s

trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
0m 25s

trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+0 :ok:
spotbugs
1m 16s

Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
1m 14s

trunk passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 31s

the patch passed

+1 :green_heart:
compile
0m 31s

the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javac
0m 31s

the patch passed

+1 :green_heart:
compile
0m 27s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
javac
0m 27s

the patch passed

+1 :green_heart:
checkstyle
0m 20s

the patch passed

+1 :green_heart:
mvnsite
0m 32s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
shadedclient
14m 58s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 20s

the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04

+1 :green_heart:
javadoc
0m 20s

the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

+1 :green_heart:
findbugs
1m 16s

the patch passed

_ Other Tests _

+1 :green_heart:
unit
6m 58s

hadoop-mapreduce-client-core in the patch passed.

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

83m 15s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2579/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2579

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 8776cfff27cc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 99d08a19ba5

Default Java
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2579/1/testReport/

Max. process+thread count
1614 (vs. ulimit of 5500)

modules
C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2579/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.0.6

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

authors: hadoop-yetus steveloughran


text:
HADOOP-19252. Upgrade hadoop-thirdparty to 1.3.0

Update the version of hadoop-thirdparty to 1.3.0
across all shaded artifacts used.
This synchronizes the shaded protobuf library with those of all other shaded artifacts (guava, avro)
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[X] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
18m  3s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+0 :ok:
xmllint
0m  1s

xmllint was not available.

+0 :ok:
shelldocs
0m  1s

Shelldocs was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ branch-3.4.1 Compile Tests _

+0 :ok:
mvndep
4m  4s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
44m 45s

branch-3.4.1 passed

+1 :green_heart:
compile
19m 27s

branch-3.4.1 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
17m 53s

branch-3.4.1 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
mvnsite
23m 13s

branch-3.4.1 passed

+1 :green_heart:
javadoc
9m 13s

branch-3.4.1 passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
7m 45s

branch-3.4.1 passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
shadedclient
54m 58s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s

Maven dependency ordering for patch

-1 :x:
mvninstall
0m 34s
/patch-mvninstall-root.txt
root in the patch failed.

-1 :x:
compile
0m 30s
/patch-compile-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt
root in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.

-1 :x:
javac
0m 30s
/patch-compile-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt
root in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.

-1 :x:
compile
0m 29s
/patch-compile-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt
root in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.

-1 :x:
javac
0m 29s
/patch-compile-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt
root in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-1 :x:
mvnsite
0m 25s
/patch-mvnsite-root.txt
root in the patch failed.

+1 :green_heart:
shellcheck
0m  0s

No new issues.

-1 :x:
javadoc
0m 33s
/patch-javadoc-root-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt
root in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.

-1 :x:
javadoc
0m 40s
/patch-javadoc-root-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt
root in the patch failed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05.

-1 :x:
shadedclient
3m 10s

patch has errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
7m 32s
/patch-unit-root.txt
root in the patch failed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

199m 15s

Subsystem
Report/Notes

Docker
ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7024/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/7024

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs

uname
Linux 6a301af30d4d 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.4.1 / cf53b0994d7bc957aa904c022cf928bc3293e657

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7024/1/testReport/

Max. process+thread count
613 (vs. ulimit of 5500)

modules
C: hadoop-project . U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7024/1/console

versions
git=2.25.1 maven=3.6.3 shellcheck=0.7.0

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@zeekling thanks for that -hadn't seen it.
That is just Junit's temp directory: not a production issue
https://nvd.nist.gov/vuln/detail/CVE-2020-15250
doesn't mean it's not good to upgrade again, but not a blocker to shipping the production code.

ok, got it. @steveloughran

was done manually

authors: hadoop-yetus steveloughran zeekling


text:
HBASE-24704 Make the Table Schema easier to view even there are multi…

…ple families

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  1s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

-0 :warning:
whitespace
0m  0s
The patch 8 line(s) with tabs.

_ Other Tests _

+1 :green_heart:
asflicense
0m 12s
The patch does not generate ASF License warnings.

2m 26s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
dupname asflicense

uname
Linux fe2ffe84582a 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 1db89773e6

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/artifact/yetus-general-check/output/whitespace-tabs.txt

Max. process+thread count
47 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 26s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
5m  6s
master passed

-0 :warning:
javadoc
0m 50s
hbase-server in master failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 50s
the patch passed

-0 :warning:
javadoc
0m 50s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
222m 57s
hbase-server in the patch passed.

238m  2s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux edeec2d18a48 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 1db89773e6

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/testReport/

Max. process+thread count
3026 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 26s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 15s
master passed

+1 :green_heart:
javadoc
0m 40s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 57s
the patch passed

+1 :green_heart:
javadoc
0m 38s
the patch passed

_ Other Tests _

-1 :x:
unit
235m 59s
hbase-server in the patch failed.

248m 52s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux dc85ece9bcd4 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 1db89773e6

Default Java
1.8.0_232

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/testReport/

Max. process+thread count
3724 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 14s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

-0 :warning:
whitespace
0m  0s
The patch 8 line(s) with tabs.

_ Other Tests _

+1 :green_heart:
asflicense
0m 12s
The patch does not generate ASF License warnings.

2m 28s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
dupname asflicense

uname
Linux 103e7de76cb3 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 1db89773e6

whitespace
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/artifact/yetus-general-check/output/whitespace-tabs.txt

Max. process+thread count
42 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  2s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 44s
master passed

-0 :warning:
javadoc
0m 44s
hbase-server in master failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 33s
the patch passed

-0 :warning:
javadoc
0m 40s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
193m 49s
hbase-server in the patch passed.

207m  8s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux 8e3ce6c556d1 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 1db89773e6

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/testReport/

Max. process+thread count
3231 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 22s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m  6s
master passed

+1 :green_heart:
javadoc
0m 38s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 48s
the patch passed

+1 :green_heart:
javadoc
0m 35s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
202m 31s
hbase-server in the patch passed.

214m 35s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux a05db99516ca 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 1db89773e6

Default Java
1.8.0_232

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/testReport/

Max. process+thread count
3460 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  5s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

_ Other Tests _

+1 :green_heart:
asflicense
0m 12s
The patch does not generate ASF License warnings.

2m 19s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
dupname asflicense

uname
Linux 7ed9666e6694 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 724f0478ed

Max. process+thread count
43 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 31s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
5m  7s
master passed

-0 :warning:
javadoc
0m 52s
hbase-server in master failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 50s
the patch passed

-0 :warning:
javadoc
0m 51s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
222m 42s
hbase-server in the patch passed.

237m 40s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux 4e824fbd936c 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 724f0478ed

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/testReport/

Max. process+thread count
3156 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 30s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 18s
master passed

+1 :green_heart:
javadoc
0m 40s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m  1s
the patch passed

+1 :green_heart:
javadoc
0m 39s
the patch passed

_ Other Tests _

-1 :x:
unit
236m 19s
hbase-server in the patch failed.

249m  2s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux 2b19aaac98ae 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 724f0478ed

Default Java
1.8.0_232

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/testReport/

Max. process+thread count
3182 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/3/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 28s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

_ Other Tests _

+1 :green_heart:
asflicense
0m 13s
The patch does not generate ASF License warnings.

1m 59s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
dupname asflicense

uname
Linux 4b3af99a2228 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 3bd54217a0

Max. process+thread count
48 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 29s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 39s
master passed

+1 :green_heart:
javadoc
0m 39s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 28s
the patch passed

+1 :green_heart:
javadoc
0m 38s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
140m  9s
hbase-server in the patch passed.

151m  8s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux c14e4986a3b3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 3bd54217a0

Default Java
1.8.0_232

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/testReport/

Max. process+thread count
4728 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 12s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 45s
master passed

-0 :warning:
javadoc
0m 45s
hbase-server in master failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 38s
the patch passed

-0 :warning:
javadoc
0m 40s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
196m 31s
hbase-server in the patch passed.

210m 23s

Subsystem
Report/Notes

Docker
Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux 0986825ebc0d 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 3bd54217a0

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/testReport/

Max. process+thread count
3597 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

PTAL. Thanks. @wchevreuil @virajjasani

@wchevreuil @virajjasani What do you think of this proposal?
Thanks.

rebase

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 29s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

_ Other Tests _

+1 :green_heart:
asflicense
0m 14s
The patch does not generate ASF License warnings.

2m  2s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
dupname asflicense

uname
Linux fcf807eab978 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 7c4d66a277

Max. process+thread count
48 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 51s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 41s
master passed

-0 :warning:
javadoc
0m 51s
hbase-server in master failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 58s
the patch passed

-0 :warning:
javadoc
0m 46s
hbase-server in the patch failed.

_ Other Tests _

-1 :x:
unit
8m 13s
hbase-server in the patch failed.

21m 48s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux 42a83860b585 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 7c4d66a277

Default Java
2020-01-14

javadoc
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/testReport/

Max. process+thread count
762 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 31s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 43s
master passed

+1 :green_heart:
javadoc
0m 38s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 31s
the patch passed

+1 :green_heart:
javadoc
0m 36s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
145m 54s
hbase-server in the patch passed.

157m 11s

Subsystem
Report/Notes

Docker
Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2042

Optional Tests
javac javadoc unit

uname
Linux 1e184ed97aeb 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 7c4d66a277

Default Java
1.8.0_232

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/testReport/

Max. process+thread count
3996 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2042/4/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@bsglz with updated usage of ColumnFamilyDescriptors properties, we might have a slight different UI. Can you please upload screenshots with updated PR results of UI on Jira? For both master and branch-2.
Thanks

@bsglz with updated usage of ColumnFamilyDescriptors properties, UI might looks slight differently. Can you please upload screenshots with updated PR results of UI on Jira? For both master and branch-2.
Thanks

No prob, will update later.

@virajjasani Updated the screenshots.
Thanks.

Thank you @bsglz
If you can raise PR for branch-1, that would be really great. Merging this PR in a while.

Thank you @bsglz
If you can raise PR for branch-1, that would be really great. Merging this PR in a while.

No prob, will do it tomorrow.

authors: Apache-HBase bsglz virajjasani

repo: apache/hbase


text:
HBASE-25207 Revisit the implementation and usage of RegionStates.include

Remove the RegionStates.include method as its name is ambiguous.
Add more comments to describe the logic on why we filter region like
this.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  6s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 55s
master passed

+1 :green_heart:
checkstyle
1m  8s
master passed

+1 :green_heart:
spotbugs
1m 59s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 24s
the patch passed

+1 :green_heart:
checkstyle
1m  6s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
17m 45s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
spotbugs
2m 11s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 13s
The patch does not generate ASF License warnings.

40m 22s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2571

Optional Tests
dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle

uname
Linux a7437c83add0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 200343c59c

Max. process+thread count
94 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
5m  3s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 54s
master passed

+1 :green_heart:
compile
1m  8s
master passed

+1 :green_heart:
shadedjars
7m 23s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 39s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m  0s
the patch passed

+1 :green_heart:
compile
1m  7s
the patch passed

+1 :green_heart:
javac
1m  7s
the patch passed

+1 :green_heart:
shadedjars
7m 22s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 37s
the patch passed

_ Other Tests _

-1 :x:
unit
239m 46s
hbase-server in the patch failed.

273m 52s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2571

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 8b449d462b86 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 200343c59c

Default Java
1.8.0_232

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/1/testReport/

Max. process+thread count
2828 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Let me check the failed UTs.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 27s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  1s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 44s
master passed

+1 :green_heart:
checkstyle
1m 10s
master passed

+1 :green_heart:
spotbugs
2m  2s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 26s
the patch passed

+1 :green_heart:
checkstyle
1m  6s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
17m  3s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
spotbugs
2m  8s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 16s
The patch does not generate ASF License warnings.

38m 37s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2571

Optional Tests
dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle

uname
Linux 66b839112d55 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6c8f5f30d5

Max. process+thread count
94 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 29s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m  2s
master passed

+1 :green_heart:
compile
1m  6s
master passed

+1 :green_heart:
shadedjars
6m 30s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 40s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m  8s
the patch passed

+1 :green_heart:
compile
1m  7s
the patch passed

+1 :green_heart:
javac
1m  7s
the patch passed

+1 :green_heart:
shadedjars
6m 42s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 40s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
135m 39s
hbase-server in the patch passed.

163m 14s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2571

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux f3104324e673 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6c8f5f30d5

Default Java
2020-01-14

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/testReport/

Max. process+thread count
3975 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 33s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 30s
master passed

+1 :green_heart:
compile
1m  6s
master passed

+1 :green_heart:
shadedjars
7m 18s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 40s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 54s
the patch passed

+1 :green_heart:
compile
0m 59s
the patch passed

+1 :green_heart:
javac
0m 59s
the patch passed

+1 :green_heart:
shadedjars
7m 25s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 40s
the patch passed

_ Other Tests _

-1 :x:
unit
229m 18s
hbase-server in the patch failed.

259m 14s

Subsystem
Report/Notes

Docker
Client=19.03.13 Server=19.03.13 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/2571

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 0eae4a3aa019 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6c8f5f30d5

Default Java
1.8.0_232

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/testReport/

Max. process+thread count
2747 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2571/2/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@brfrn169 Mind taking a look? Let's get this in first and then work on HBASE-25206.
Thanks.

+1 LGTM


text:
[Bug]: Updated Component Field Values are not reflecting in the Pipeline

Apache Hop version?
2.3.0
Java version?
11.0.14
Operating system
Windows
What happened?
When working with Database related components/transforms like Database lookup, Insert & Update, Table Output or etc. the updated field values (such as Field mapping, Connection, Transform name or etc.) are not saved.
Issue Priority
Priority: 2
Issue Component
Component: Database, Component: Hop Gui, Component: Pipelines, Component: Transforms

Can you give a specific example? it sounds like a really odd bug.

I am pretty sure I am having the same problem. And my specs are pretty similar.
Apache Hop version?
2.3.0
Java version?
OpenJDK11U 11.0.18_10
Operating system
Windows Server 2019 Datacenter
Example:
At the Dimension lookup/update step

Even after choosing one type for every field

Once saved (pressing Ok) it's not saved successfuly and is set tu None again once I reopen the transform.
I checked the hpl file and it looks to be missing a tag at

Tried adding this tag. As I saw other transformations having type features used it.
<type>String</type>
However it didn't work and it was overwritten to its original state. I think it's best to continue in this issue. If that's not the case I can create a new Issue

authors: hansva kirana-ks martinezcajm

repo: apache/hop


text:
[SUPPORT] Urgent: Please Help: MOR seems not working as expected

Hi,
I am doing POC about Hudi, but I am quite new to Hudi, I am stuck here! I would like to ask about Hudi merge on read table
(query out only the basic parquet data, not include the delta data)
I am using Hudi 0.6.0 + Spark 2.4.4
I use the following code to do 3 operations: 1. overwrite 2. upsert 3. upsert.

After the operations are done, when I look at the HDFS files, there are 3 parquets there, I have thought that only one parquet be there, the two upsert operations should be written to avro files as delta commits.

[root@host1]hdfs dfs -ls /data/hudi_demo/hudi_hive_read_write_mor_4/2020-11-19
Found 4 items
-rw-r--r--   2 rootsupergroup         93 2020-11-21 14:45 /data/hudi_demo/hudi_hive_read_write_mor_4/2020-11-19/.hoodie_partition_metadata
-rw-r--r--   2 rootsupergroup     435000 2020-11-21 14:45 /data/hudi_demo/hudi_hive_read_write_mor_4/2020-11-19/3f7b1807-8099-4629-8fb7-c8a74ba9298d-0_0-21-21_20201121144541.parquet
-rw-r--r--   2 root supergroup     434787 2020-11-21 14:45 /data/hudi_demo/hudi_hive_read_write_mor_4/2020-11-19/3f7b1807-8099-4629-8fb7-c8a74ba9298d-0_0-51-49_20201121144550.parquet
-rw-r--r--   2 root supergroup     434842 2020-11-21 14:45 /data/hudi_demo/hudi_hive_read_write_mor_4/2020-11-19/3f7b1807-8099-4629-8fb7-c8a74ba9298d-0_0-81-77_20201121144554.parquet

In the code, I have used the following code snippet to disable compaction, not sure it is correct to disable compaction, I disable compaction simply want to see the delta commit files

      //disable async compact
      .option(DataSourceWriteOptions.ASYNC_COMPACT_ENABLE_OPT_KEY, "false")
      .option(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS_PROP, 100)
      //disable inline compact
      .option(HoodieCompactionConfig.INLINE_COMPACT_PROP, "false")

In Hive,

I do the following query, I want to query only the basic parquet data, not include the delta commit data.
select * from xyz.hudi_hive_read_write_mor_4_ro;, it query out all the recent records(with the larget creation_date for the updated record) data, like a snapshot query

In spark shell,

I do the same query:
val basepath="/data/hudi_demo/hudi_hive_read_write_mor_4"

val x= spark.read.format("hudi").option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_READ_OPTIMIZED_OPT_VAL)

val df =x.load(basepath + "/*")

df.show

The result is same as query in hive.
Following is the code, that I use to do the POC
package org.example.hudi

import org.apache.hudi.DataSourceWriteOptions
import org.apache.hudi.config.{HoodieCompactionConfig, HoodieIndexConfig, HoodieWriteConfig}
import org.apache.hudi.index.HoodieIndex
import org.apache.spark.sql.{SaveMode, SparkSession}

case class MyOrder(
                    name: String,
                    price: String,
                    creation_date: String,
                    dt: String)

object MORTest {
  val overwrite1Data = Seq(
    MyOrder("A", "12.7", "2020-11-18 14:43:32", "2020-11-19"),
    MyOrder("B", "13.2", "2020-11-18 14:42:21", "2020-11-19"),
    MyOrder("C", "11.6", "2020-11-18 14:47:19", "2020-11-19"),
    MyOrder("D", "10.4", "2020-11-18 14:46:50", "2020-11-19")
  )

  val insertUpdate1Data = Seq(
    MyOrder("A", "13.7", "2020-11-18 14:53:32", "2020-11-19"), //update A
    MyOrder("E", "11.6", "2020-11-18 14:47:19", "2020-11-19"), //add E,F
    MyOrder("F", "10.4", "2020-11-18 14:46:50", "2020-11-19")
  )

  val insertUpdate2Data = Seq(
    MyOrder("X", "13.7", "2020-11-18 14:53:32", "2020-11-19"), //add X, Y
    MyOrder("Y", "18.2", "2020-11-19 14:42:21", "2020-11-19"),
    MyOrder("F", "17.4", "2020-11-18 15:46:50", "2020-11-19") //Update F
  )

  val spark = SparkSession.builder.appName("MORTest")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .config("spark.sql.warehouse.dir", "hdfs:///user/hive/warehouse")
    .enableHiveSupport().getOrCreate()

  val hudi_table = "hudi_hive_read_write_mor_4"

  val base_path = s"/data/hudi_demo/$hudi_table"

  def run(op: Int) = {
    val (data, saveMode) = op match {
      case 1 => (overwrite1Data, SaveMode.Overwrite)
      case 2 => (insertUpdate1Data, SaveMode.Append)
      case 3 => (insertUpdate2Data, SaveMode.Append)
    }
    import spark.implicits._
    val insertData = spark.createDataset(data)
    insertData.write.format("hudi")
      .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "name")
      .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "creation_date")
      .option(DataSourceWriteOptions.HIVE_DATABASE_OPT_KEY, "xyz")
      .option(DataSourceWriteOptions.HIVE_TABLE_OPT_KEY, hudi_table)
      .option(DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY, "true")
      .option(HoodieIndexConfig.BLOOM_INDEX_UPDATE_PARTITION_PATH, "true")
      .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "dt")

      //table type: MOR
      .option(DataSourceWriteOptions.TABLE_TYPE_OPT_KEY, DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL)

      //disable async compact
      .option(DataSourceWriteOptions.ASYNC_COMPACT_ENABLE_OPT_KEY, "false")
      .option(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS_PROP, 100)
      //disable inline compact
      .option(HoodieCompactionConfig.INLINE_COMPACT_PROP, "false")

      .option(DataSourceWriteOptions.HIVE_URL_OPT_KEY, "jdbc:hive2://127.0.0.1:10000")
      .option(HoodieWriteConfig.TABLE_NAME, hudi_table)
      .option(DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY, "org.apache.hudi.hive.MultiPartKeysValueExtractor")
      .option(HoodieIndexConfig.INDEX_TYPE_PROP, HoodieIndex.IndexType.GLOBAL_BLOOM.name())
      .option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, "dt")
      .option("hoodie.insert.shuffle.parallelism", "2")
      .option("hoodie.upsert.shuffle.parallelism", "2")
      .mode(saveMode)
      .save(base_path);
  }

  def main(args: Array[String]): Unit = {
    //do overwrite
    run(1)

    //do upsert
    run(2)

    //do upsert
    run(3)

    println("===MOR is done=====")
  }

}

@bithw1 : The 3 Parquet files are different versions of the same file.
Your second and third batch has both new records and updates. Hudi will write new records to parquet files but can route updates to delta files. As you have very few records in the initial parquet file, Hudi will route both new records and updates to the same Parquet file and a new version of the file is created.
If you do only have updates in batch 2, you will see the delta files.

Hi @bvaradar , thanks a lot for the great help. I tried your instruction, and it works as expected now.
So, I am closing this issue now.

repo: apache/hudi


text:
[SUPPORT] Disaster Recovery (DR) Setup? Questions.

Are there any tips or support on setting up a Disaster Recovery (DR) environment with Apache Hudi?
We are creating our Datalake, stored on AWS S3, by running a Spark structured streaming application on AWS EMR. The Spark application is processing incoming data from a AWS Kinesis stream and saving them into Hudi tables on S3 and syncing with the AWS Glue catalog. All of this happens in a single AWS region (us-east-1).
In the event where we need to failover to a different region or our main region (us-east-1) goes down, what is the suggested approach to get start up again in another AWS region with our existing Datalake data? We can set up S3 replication to replicate the parquet files (and .hoodie files) to another S3 bucket residing in a different AWS region, but S3 replication happens asynchronously which means files may get replicated out of order and cause issues when querying (due to possible missing files). We will need to look at how to replicate Glue databases/tables from 1 AWS region into another, so that other AWS services and/or query engines can query.
Would love to hear some ideas/thoughts :)
Trying to workaround an issue like this: https://www.datacenterdynamics.com/en/news/aws-us-east-1-outage-brings-down-services-around-the-world/
Environment Description

Hudi version : 0.7.0-amzn-1

Spark version : 2.4.7

Hive version : 2.3.7

Hadoop version : 2.10.1

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

I ran a test where I deleted some .parquet files from a Hudi table (stored on S3) to simulate S3 replication lagging behind in the copy process and therefore causing the target S3 bucket to have missing data files. Then used AWS Athena to query the Hudi table with missing data files.
Result:

Query ran successfully with missing data files (.parquet) - AWS Athena did not error out.
The returned results were incorrect (count and data returned did not match with original Datalake). This makes sense because .parquet data files are missing.

Next, I wonder what happens if S3 replication is slow and it's not able to replicate the files in .hoodie folder fast enough - what happens when I run SQL queries on a Hudi table with missing .hoodie files?

There is a preview available for AWS Backup for S3, this might be interesting to use in future. AWS claims that you can do point in time recovery with it: Continuous backups create point-in-time backups, and allow you to restore S3 resources to any point-in-time within the last 35 days.

Thanks @kazdy for your input. I read the documentation for AWS Backup and S3 support.
Based on the documentation, I am assuming we can do the following (using the us-east-1 issue that occurred on 2021-DEC-7):
FAILURE SCENARIO:

Create a backup of Hudi Datalake S3 bucket on us-east-1 using AWS Backup and create a copy of that in region X.
When us-east-1 failure/issue occurs, then restore Hudi Datalake S3 bucket backup (created from us-east-1) into a different S3 bucket in region X.

FALLBACK SCENARIO (when us-east-1 is stable again):

Create a backup of Hudi Datalake S3 bucket on region X using AWS Backup and create a copy of that in us-east-1.
When ready to fallback to us-east-1, then restore Hudi Datalake S3 bucket backup (created from region X) into a S3 bucket in us-east-1.

Sounds like above scenario should work - won't be able to test this until AWS Backup for S3 is generally available.

@WTa-hash have you tried using savepoint feature? Maybe it'll allow you to deal with these issues when using s3 replication and missing .hoodie files? There will be some missing data, but maybe tables will not be corrupted?

@xushiyan @bhasudha @bvaradar @yanghua : Do you folks have any pointes on this regard.

@WTa-hash have you tried using savepoint feature? Maybe it'll allow you to deal with these issues when using s3 replication and missing .hoodie files? There will be some missing data, but maybe tables will not be corrupted?

No, I haven't. Can you link me to documentation about the savepoint feature.

We don't have any documentation as such. You need to directly use writeClient or go via hudi-cli.
But here is how you can do savepoint and restore using hudi-cli
connect --path /tmp/hudi_trips_cow
commits show
set --conf SPARK_HOME=[SPARK_HOME_DIR]
savepoint create --commit 20220105222853592 --sparkMaster local[2]

// restore

refresh
savepoint rollback --savepoint 20220106085108487 --sparkMaster local[2]

btw, savepoint and restore for MOR is added just few weeks back and so is available only from 0.11. But should work for COW for older releases too.

I will try to add more documentation around savepoint/restore to our website.

recently we also updated instructions on how to use hudi-cli for S3 dataset.
https://hudi.apache.org/docs/next/cli/
just incase you you interested.

Closing this as hudi has savepoint and restore for both table types. Feel free to reopen or create new github issue if you need further assistance. thanks!

For anyone looking at this question now, I see there's documentation available for "current" version (0.11):
https://hudi.apache.org/docs/next/disaster_recovery

authors: WTa-hash kazdy nsivabalan