text:
Hope xAxis of Stack Line support value

What problem does this feature solve?
Currently, stack line only support category xAxis. Could xAxis support value so the x direction could be scaled by x value?
If so, Echarts could be used for simulating the geological profile, which is composed of connected rock/soil layer points.
What does the proposed API look like?
Just providing the value type for xAxis for stack line.

yes, xAxis of Stack Line supports value, Demo Code

@helgasoft In the demo you provided, only the default value of xAxis: category is used. Below is a screenshot of the complete options in the demo. In fact, when the value of xAxis.type is set to value, xAxis.data cannot be assigned, and all data will be in series, such as series[0].data = [[1, 2], [2, 3], [3, 4]].

@carlshe, thank you for pointing that out.
I forgot what the documentations says - when "axis.data is specified, the type is auto set as 'category'"
I still do not understand why type='value' is needed for your data.
Could you provide a data sample to see the problem with "x direction...scaled by x value" ?
Geological profiles are interesting, maybe we can come up with a solution.

Stack line with value axis may have some logic problems. For example, if the data of one series is [[1.1, 20], [2, 31], [3, 2]] and another series [[1, 12], [2, 31], [3, 2]], how do you expect them to stack? Should [1.1, 20] stack with [1, 12]? If not, the two bars should probably overlap. So I think you probably should use category x axis by logic.

Dear helgasoft,
Thank you for your quick reply!
I use ECharts' stack area map to express the geological profile/section. The geological profile data comes from the sampling analysis of geological boreholes. Each borehole has a spatial location. Generally, the boreholes on the section line are not uniformly distributed on the profile line. But stack area map treats the data of xAxis as discrete quantities (‘category’) uniformly distributed on the x-axis, and then cannot reflect the distance relationship between the boreholes.
As follow map shown, there are 3 boreholes located on the profile line, the distance between first and second borehole is 16m, the distance between second and third borehole is 8m. The presented 3 boreholes are uniformly distributed on the xAxis. It will be best the visual interval between second and third is half of that between first and second.
Could please help giving a solution for it?
Thank you again!
Carlshe
At 2023-03-30 03:52:48, "helgasoft" @.***> wrote:
@carlshe, thank you for pointing that out.
I forgot what the documentations says - when "axis.data is specified, the type is auto set as 'category'"
I still do not understand why type='value' is needed for your data.
Could you provide a data sample to see the problem with "x direction...scaled by x value" ?
Geological profiles are interesting, maybe we can come up with a solution.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Stack line with value axis may have some logic problems. For example, if the data of one series is [[1.1, 20], [2, 31], [3, 2]] and another series [[1, 12], [2, 31], [3, 2]], how do you expect them to stack? Should [1.1, 20] stack with [1, 12]? If not, the two bars should probably overlap. So I think you probably should use category x axis by logic.

Thank you Ovilia,
I just hope while using stacked area map, the xAxis can reflect comparable continuous quantity. Currently the data of xAxis is treated as discrete quantity.
Could please refer my reply to helgasoft?

@carlshe, I think themeRiver chart will work best for you
Demo Code

Thank you very much, helgasoft!
I tried your demo. It seems not suitable for me. Because the lines are not stable. Any a item change will result in vertical variation of bottom, top or middle lines. In stack area map, item change only take effect on its neighbor.

if boreholes are of different depth, an additional 'unknown' layer can be added at the bottom
Demo Code - example: deepest borehole is at 16.

If stack area map really doesn't support continuous x value, is it possible to derive sub type from stack area map to improve its rendering?

Thank you very much, helgasoft!
The circumscribed polygon of the thematic river map is a symmetric graph. Any a item change, even small, will result in vertical adjust of bottom, top or middle lines to ensure it becomes symmetric again. In stack area map, item change only take effect on its neighbor. The bottom is always stable.
I tried your new demo, and found that at x-24, the bottom moves a little up, and the top moves a little down.
If stack area map really doesn't support continuous x value, is it possible to derive a sub type from stack area map to improve its rendering? by overriding/modifying part of its rendering process?

I tried your new demo, and found that at x-24, the bottom moves a little up, and the top moves a little down.

My mistake: last data line should be [24, 19, 'unknown'] instead of [24, 18, 'unknown']. If values sums of all three boreholes are the same (80), then the chart remains a rectangle.

is it possible to derive a sub type from stack area map to improve its rendering? by overriding/modifying part of its rendering process?

I cannot really answer that because I work at API level and do not do development (yet). But by what @Ovilia said earlier about the logic problem, it looks, hmm... impossible.
It would be helpful to investigate further if any JS charting software does exactly what you want and then post it here as a question with a code example.

Stack line with value axis may have some logic problems. For example, if the data of one series is [[1.1, 20], [2, 31], [3, 2]] and another series [[1, 12], [2, 31], [3, 2]], how do you expect them to stack? Should [1.1, 20] stack with [1, 12]? If not, the two bars should probably overlap. So I think you probably should use category x axis by logic.

I understand your care.
I have a question? For 'category', if the data is ordered number, is it possible to expose an option to tell echart to treat these categories as x values, so the line points could be rendered at expected x-locations?
Or is it possible to derive a sub type from stack area map to improve its rendering? by overriding/modifying part of its rendering process, especally the function that get the coords from [category,y-value]?
Thanks!

I found hichart can render stacked area map with x-value.
In fact, it still define xAxis as category, but it accept x-value as category and render line points at expected location.

ECharts can do the same - Demo Code

Thank you, helgasoft!
I tested your demo code. I think it is essentially still original category type of xAxis. All the data defined for x_Axis is discrete identification. All the x-coord node must be prdefined in xAxis and evenly distributed over x-axis, and all feature points must locate at exactly one of the predefined x-coords. But for geological profile or any other similar polylines, the feature point could be at any a x-coord.
Highchart has no much limit on x of Stack area map, except that all the layers should share the single data definition of x-categories. Values in x data set are not required evenly distributed over x-axis.
Hope ECharts could provide similar function as soon.

For 'category', if the data is ordered numbers, is it possible to expose an option to tell echart to treat these categories as x values, so the line points could be rendered at expected x-locations?

@carlshe I don't understand. If you have such requirement, why don't using category x axis?

Guesswork continues... here is an Updated Demo with "sampling categorization", for lack of a better word.
Idea is to split xAxis into smaller units and generate more points on the original lines (sampling). Sampling step smp could be adjusted from rough (2+) to fine (<=1). Then we have enough categories (vertical reference points) to stack the lines on top of each other.
Use Switch button to toggle between lines and stacked lines.
@carlshe, @maciej-flexcompute, will this work for you?

Sorry, I didn't know that the problem was not sufficiently described. Sorry for the confusion. Sure, here are examples:
xaxis
x = [0.01, 0.50, 1.01, 1.50, 2.01, 2.51, 3.01, 3.50, 4.01, 4.50, 5.00, 5.51, 6.01, 6.50, 7.01, 7.51, 8.00, 8.50, 9.01, 9.50, 10.01, 10.51, 11.00, 11.51, 12.01, 12.51, 13.01, 13.50, 14.00, 14.20, 14.41, 14.60, 14.80, 15.01, 15.20, 15.41, 15.60, 15.80, 16.01, 16.21, 16.40, 16.60, 16.80, 17.00, 17.20, 17.41, 17.61, 17.81, 18.01, 18.20, 18.41, 18.60, 18.81, 19.00, 19.21, 19.40, 19.60, 19.81]
series 1 (y1)
y1 = [1.00, 1.03, 1.05, 1.08, 1.10, 1.13, 1.15, 1.17, 1.20, 1.22, 1.25, 1.27, 1.30, 1.32, 1.34, 1.37, 1.39, 1.41, 1.44, 1.46, 1.48, 1.50, 1.52, 1.54, 1.57, 1.59, 1.61, 1.62, 1.64, 1.65, 1.66, 1.67, 1.67, 1.68, 1.69, 1.70, 1.70, 1.71, 1.72, 1.72, 1.73, 1.74, 1.74, 1.75, 1.76, 1.76, 1.77, 1.78, 1.78, 1.79, 1.80, 1.80, 1.81, 1.81, 1.82, 1.82, 1.83, 1.84]
series 1 (y1)
y2 = [0.01, 0.71, 1.00, 0.70, -0.01, -0.72, -1.00, -0.71, 0.01, 0.71, 1.00, 0.70, -0.02, -0.71, -1.00, -0.70, 0.01, 0.71, 1.00, 0.70, -0.01, -0.71, -1.00, -0.70, 0.02, 0.72, 1.00, 0.71, -0.00, -0.31, -0.60, -0.81, -0.95, -1.00, -0.95, -0.80, -0.58, -0.31, 0.01, 0.32, 0.59, 0.81, 0.95, 1.00, 0.95, 0.80, 0.58, 0.30, -0.01, -0.31, -0.59, -0.81, -0.95, -1.00, -0.95, -0.81, -0.59, -0.30]
Plotted as category line:

Category Line Demo
Plotted as category stack:

Category Stack Demo
Plotted as value line:

Value Line Demo
Problem
You can clearly see that two line plots appear visually different. In the category plots, it seems that y1 is not a linear function as has a kink arround x=14. The y2 line has larger period in x in [14, 20] and shorter period in x in [0, 14].
In the value plot you can see that y1 is actually a linear function (the small off-linearity is due to roundoff error), and y2 is regular periodic function without any changes in periods.
This data plotted as 'category' creates confusion to the receiver, because datapoint lay in mathematically incorrect locations. User can incorrectly conclude that y1 is non-linear and y2 is not regularly periodic. In general any (x, y) data represented by real numbers should always be presented as 'values' and never as 'categories'. See plotting in Matlab, python, gnuplot, excel or any other plotting tools.
The solution you proposed in the last plot, in my example I would need to put sampling in 0.01 period, which is 2000 points instead of 58 (dataset size). But in general case (interval are real numbers) this is not possible to find such interval.
Finally, you can make a restriction that x values are common for both y1 and y2. I.e., there is not such x point of series 1 for which we don't have data from series 2 and vice versa.

Aah what a beauty, a model for problem/feature presentation, great job @maciej-flexcompute !
The new data helped me improve algorithm and code.

But in general case (interval are real numbers) this is not possible to find such interval.

Curious on the reason behind this statement. A constant step (interval) is crucial for the algorithm.
Please take a look at the couple lines where smp is generated. Could it not work with different data ?

Finally, you can make a restriction that x values are common for both y1 and y2

That may eventually help depending on what comes out of the previous question. But hope this limitation won't be necessary.
Latest Demo, essentially redrawing the lines with new X-values with a fixed step (smp).
Number of points for each series:   original value = 58,  new categorical = 100.

@helgasoft
Your example looks exactly like what I am looking for.

But in general case (interval are real numbers) this is not possible to find such interval.

Here is the proof:

lets take a, b real numbers which represents intervals between points. We are looking for such interval c that a = p * c and b = q * c where p, q are whole numbers and they have no common factors. I will show that such interval c does not exist if a=sqrt(2) and b=sqrt(3). We have respectively:
sqrt(2) = p * c and sqrt(3) = q * c. We have sqrt(2) * q = sqrt(3) * p. We can raise to power 2 both sides:
2 * q^2 = 3* p^2. We can see that left-hand side is even, thus, p^2 must be even, thus p must be even, right-hand side is divisible by 4, thus q^2 must be even, => q is even. We showed that both p and q are even which is in contradicts statement that p, q are whole numbers and they have no common factors.

Mathematically it is not possible, but numerically it is possible since we represent numbers double precision, so at least 1e-16 can work. And practically speaking human cannot visually distinguish ~1e-5.

authors: Ovilia carlshe helgasoft maciej-flexcompute xitice

repo: apache/echarts


text:
[热力图yAxis.type='value'时，单元格不能对齐]

Version
5.4.3
Link to Minimal Reproduction
https://echarts.apache.org/examples/zh/editor.html?c=heatmap-cartesian
Steps to Reproduce
我的option和相关数据如下：
// prettier-ignore
const hours = [
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23
];
// prettier-ignore
const days = [
1,2,3,4,5,6,7
];
// prettier-ignore
const data = [[0, 0, 5], [0, 1, 1], [0, 2, 0], [0, 3, 0], [0, 4, 0], [0, 5, 0], [0, 6, 0], [0, 7, 0], [0, 8, 0], [0, 9, 0], [0, 10, 0], [0, 11, 2], [0, 12, 4], [0, 13, 1], [0, 14, 1], [0, 15, 3], [0, 16, 4], [0, 17, 6], [0, 18, 4], [0, 19, 4], [0, 20, 3], [0, 21, 3], [0, 22, 2], [0, 23, 5], [1, 0, 7], [1, 1, 0], [1, 2, 0], [1, 3, 0], [1, 4, 0], [1, 5, 0], [1, 6, 0], [1, 7, 0], [1, 8, 0], [1, 9, 0], [1, 10, 5], [1, 11, 2], [1, 12, 2], [1, 13, 6], [1, 14, 9], [1, 15, 11], [1, 16, 6], [1, 17, 7], [1, 18, 8], [1, 19, 12], [1, 20, 5], [1, 21, 5], [1, 22, 7], [1, 23, 2], [2, 0, 1], [2, 1, 1], [2, 2, 0], [2, 3, 0], [2, 4, 0], [2, 5, 0], [2, 6, 0], [2, 7, 0], [2, 8, 0], [2, 9, 0], [2, 10, 3], [2, 11, 2], [2, 12, 1], [2, 13, 9], [2, 14, 8], [2, 15, 10], [2, 16, 6], [2, 17, 5], [2, 18, 5], [2, 19, 5], [2, 20, 7], [2, 21, 4], [2, 22, 2], [2, 23, 4], [3, 0, 7], [3, 1, 3], [3, 2, 0], [3, 3, 0], [3, 4, 0], [3, 5, 0], [3, 6, 0], [3, 7, 0], [3, 8, 1], [3, 9, 0], [3, 10, 5], [3, 11, 4], [3, 12, 7], [3, 13, 14], [3, 14, 13], [3, 15, 12], [3, 16, 9], [3, 17, 5], [3, 18, 5], [3, 19, 10], [3, 20, 6], [3, 21, 4], [3, 22, 4], [3, 23, 1], [4, 0, 1], [4, 1, 3], [4, 2, 0], [4, 3, 0], [4, 4, 0], [4, 5, 1], [4, 6, 0], [4, 7, 0], [4, 8, 0], [4, 9, 2], [4, 10, 4], [4, 11, 4], [4, 12, 2], [4, 13, 4], [4, 14, 4], [4, 15, 14], [4, 16, 12], [4, 17, 1], [4, 18, 8], [4, 19, 5], [4, 20, 3], [4, 21, 7], [4, 22, 3], [4, 23, 0], [5, 0, 2], [5, 1, 1], [5, 2, 0], [5, 3, 3], [5, 4, 0], [5, 5, 0], [5, 6, 0], [5, 7, 0], [5, 8, 2], [5, 9, 0], [5, 10, 4], [5, 11, 1], [5, 12, 5], [5, 13, 10], [5, 14, 5], [5, 15, 7], [5, 16, 11], [5, 17, 6], [5, 18, 0], [5, 19, 5], [5, 20, 3], [5, 21, 4], [5, 22, 2], [5, 23, 0], [6, 0, 1], [6, 1, 0], [6, 2, 0], [6, 3, 0], [6, 4, 0], [6, 5, 0], [6, 6, 0], [6, 7, 0], [6, 8, 0], [6, 9, 0], [6, 10, 1], [6, 11, 0], [6, 12, 2], [6, 13, 1], [6, 14, 3], [6, 15, 4], [6, 16, 0], [6, 17, 0], [6, 18, 0], [6, 19, 0], [6, 20, 1], [6, 21, 2], [6, 22, 2], [6, 23, 6]]
.map(function (item) {
return [item[1], item[0], item[2] || '-'];
});
option = {
tooltip: {
position: 'top'
},
grid: {
height: '50%',
top: '20%'
},
xAxis: {
type: 'category',
data: hours,
splitArea: {
show: true
}
},
yAxis: {
type: 'value',
data: days,
scale:true,
splitArea: {
show: true
}
},
visualMap: {
min: 0,
max: 10,
calculable: true,
orient: 'horizontal',
left: 'center',
bottom: '15%'
},
series: [
{
name: 'Punch Card',
type: 'heatmap',
data: data,
label: {
show: true
},
emphasis: {
itemStyle: {
shadowBlur: 10,
shadowColor: 'rgba(0, 0, 0, 0.5)'
}
}
}
]
};
Current Behavior
当yAxis.type='value'时，热力图单元格和Y轴没有对其

Expected Behavior
希望能对其
Environment
- OS:macOS
- Browser:Chrome
- Framework:JS

Any additional comments?
No response

maybe like that ?

maybe like that ?  NB: please close issue if problem solved.
Thanks !

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
[FLINK-35196] [Connector / Pulsar] Fix bouncycastle class not found

Purpose of the change

The bouncycastle dependency was removed in this PR  https://github.com/apache/flink-connector-pulsar/pull/9 , which was correct at the time, and there was no table api at that time

When we introduced table api, dependencies started to show errors because bouncycastle was referenced in table api

2024-04-22 14:45:48
java.lang.NoClassDefFoundError: org/apache/pulsar/shade/org/bouncycastle/util/Arrays
	at org.apache.flink.connector.pulsar.table.sink.PulsarWritableMetadata.readMetadata(PulsarWritableMetadata.java:67)
	at org.apache.flink.connector.pulsar.table.sink.PulsarWritableMetadata.applyWritableMetadataInMessage(PulsarWritableMetadata.java:55)
	at org.apache.flink.connector.pulsar.table.sink.PulsarTableSerializationSchema.serialize(PulsarTableSerializationSchema.java:106)
	at org.apache.flink.connector.pulsar.table.sink.PulsarTableSerializationSchema.serialize(PulsarTableSerializationSchema.java:40)
	at org.apache.flink.connector.pulsar.sink.writer.PulsarWriter.write(PulsarWriter.java:143)
	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:160)
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
	at java.lang.Thread.run(Thread.java:748)

@syhily @snuyanzin @tsreaper PTAL. Thanks.

Thanks for your commit. I think jdk15on should be dropped by in favor of the jdk18on.

Thanks for your commit. I think jdk15on should be dropped by in favor of the jdk18on.

@syhily Thanks review. I have upgraded jdk15on to jdk18on. PTAL.

The pulsar-client-all should be upgrade to the 3.0.4 in the mean time.

The pulsar-client-all should be upgrade to the 3.0.4 in the mean time.

@syhily

The pulsar-client-all should be upgrade to the 3.0.4 in the mean time.

@syhily After pulsar-client-all was updated to 3.0.4, the unit test failed. I took a brief look and found that publishTimeStopCursor seemed to be connected to the consumer and then seek position, so consumer will be disconnect.Then after reconnecting the consumer, an exception of Exclusive consumer is already connected occurred. I think this requires more detailed investigation shows that this may be due to some changes in the internal implementation of pulsar-client-all due to version upgrades, and troubleshooting is not that easy.
So I think it is worth using a separate PR to upgrade pulsar-client-all to version 3.0.4 and solve the problems caused by the related upgrade.
This PR only solves the problem that the table api cannot find the bouncycastle class.
Related error logs: apache/flink-connector-pulsar/actions/runs/8832359197/job/24249517876?pr=91
PTAL. Thanks.

Thanks for your works. I think we can divide the work into two steps. Bump pulsar-client-all do require a lot of works. But since the pulsar-client-all depends on the legacy 15on. I think we can exclude these dependencies first in this PR and use 18on.

@syhily ignore this problem. Thanks.

OK, feel free to reopen this PR if you need.

source: gharchive/pull-request

authors: syhily wenbingshen

repo: apache/flink-connector-pulsar


text:
[FLINK-16219][runtime] Made AsyncWaitOperator chainable to non-sources.

What is the purpose of the change
Made AsyncWaitOperator chainable to non-sources.
Brief change log

Adding HEAD_AFTER_LEGACY_SOURCE ChainingStrategy.  Operators will be eagerly chained whenever possible, except after legacy sources.
Made AsyncWaitOperator chainable to non-sources.

Verifying this change

Added AsyncWaitOperatorTest#testAsyncOperatorIsNotChainedToSources

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Addressed all comments; let's wait for travis.

@pnowojski green

repo: apache/flink


text:
[FLINK-17112][table] Support DESCRIBE statement in Flink SQL

What is the purpose of the change
FLINK-14688 introduces DESCRIBE statement in sql parser, but doesn't implement it in planner side because the TableEnvironment.sqlUpdate returns nothing. Since FLINK-16366 introduces TableEnvironment.executeSql and returns TableResult, we can implement DESCRIBE statement in planner now.
Brief change log

84b4681 supports DESCRIBE statement in both planners

Verifying this change
This change added tests
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

 +------+--------------------------------+-------+--------+-------------------+----------------------------+ | name |                           type |  null |    key |    compute column |                  watermark | +------+--------------------------------+-------+--------+-------------------+----------------------------+ |    a |                            INT | false | (NULL) |            (NULL) |                     (NULL) | |    b |                         STRING |  true | (NULL) |            (NULL) |                     (NULL) | |    c |  ROW<f0INT NOT NULL,f1... | false | (NULL) |            (NULL) |                     (NULL) |
|   ts |                   TIMESTAMP(3) |  true | (NULL) | TO_TIMESTAMP(b) | ts - INTERVAL '1' SECOND |
+------+--------------------------------+-------+--------+-------------------+----------------------------+
`

Thanks for the update! I tried this command locally with your test, it seems the output will be truncated with TableResult.print().
For DDL command, I think it doesn't make sense to truncate the output. The truncation was mostly introduced for select query. Could you relax the column width limit for all DDL queries?
One possible solution might be relax the limit by default and only set the limitation if needed, like select query. cc @godfreyhe

[ERROR] src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java:[116] (imports) ImportOrder: Import org.apache.flink.table.types.logical.LogicalType appears after other imports that it should precede

[ERROR] src/main/java/org/apache/flink/table/sqlexec/SqlToOperationConverter.java:[66] (imports) ImportOrder: Import org.apache.flink.table.operations.DescribeTableOperation appears after other imports that it should precede

authors: KurtYoung docete


text:
[FLINK-17646][python] Reduce the package size of PyFlink.

What is the purpose of the change
This pull request reduces the the python package size of PyFlink.
Brief change log

Remove the jars in the deps/opt folder of the PyFlink python package file except flink-python.jar and flink-sql-client.jar.
Regenerate the licenses directory and NOTICE file of the PyFlink python package file.

Verifying this change
This change added tests and can be verified as follows:

run mvn clean install -DskipTests on the flink source root directory.
run cd flink-python;python setup.py sdist to pack a python package.
check the package size in flink-python/dist, make sure it is less than 300MB.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@dianfu Thanks for your review! I have addressed your comments.

The test failure is not related to PR. Merging...


text:
[FLINK-21815][table-planner-blink] Support json ser/de for StreamExecUnion

What is the purpose of the change
Support json ser/de for StreamExecUnion
Brief change log
Support json ser/de for StreamExecUnion
Verifying this change
This change added tests and can be verified as follows:
(example:)

Added UnionJsonPlanITCase and UnionJsonPlanTest to validate the json ser/de for StreamExecUnion

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

@flinkbot run azure

The failed test has nothing to do with this change, I will merge it


text:
[hotfix] Fix typo in comment of WindowBuffer

Fix an embarrassing typo in the comment of org.apache.flink.table.runtime.operators.aggregate.window.buffers.WindowBuffer class.

Hello!  For completeness, we might want to consider merging https://github.com/apache/flink/pull/18722.

authors: RyanSkraba kylemeow


text:
[FLINK-28883][hive] Fix HiveTableSink failed to report statistic to hive metastore in batch mode

What is the purpose of the change
To fix the issue that HiveTableSInk fails to report statistic to Hive metastore.
Brief change log

When commit non-partition table/partition, try to collect statistic from files in non-partition table/partition, and then put these statistic to metastore.
When collect statistic from files, try to get RecordReader according to table's input format. If the RecordReader is intance of StatsProvidingRecordReader, which means it's orc or parquet format,  we collect the statistic numRows, fileSize, numFiles, rawDataSize.  If not, we only collect fileSize and numFiles since other statistic is hard to collect efficiently.
While commit partition, put these statistic collected to partition properties.

Verifying this change
Added UT.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure


text:
[FLINK-10085][tests] Update AbstractOperatorRestoreTestBase for 1.5

What is the purpose of the change
This PR extends the AbstractOperatorRestoreTestBase sub-classes to also cover migration from 1.5.
Brief change log

add 1.5 savepoints
add 1.5 to migration versions

Do we have any reasonable way to verify those kinds of PRs? Or shall we just look for typos while reviewing it?
If the latter than +1.

the added migration versions can be verified by running the test locally. Verifying the savepoint is a bit trickier as they don't include the actual flink version. For this test, where the content of the savepoint is deterministic, you could generate a savepoint locally and compare the contents.

Are the savepoints binary deterministic? I've tried to generate savepoints twice and each time I got different contents (also compared with savepoint in this PR). If it should be deterministic I will try to repeat it once again. Maybe I've done sth completely wrong.

I'm not sure about the binary contents being deterministic, but the state contents most definitely should be. I'm not aware of any truly random savepoint components, as the operator IDs should also be deterministic.

authors: dawidwys zentol


text:
[FLINK-11343] Omit actions in memory manager after shutdown instead o…

…f IllegalStateException
What is the purpose of the change

org.apache.flink.runtime.util.TestingFatalErrorHandler$TestingException: java.lang.IllegalStateException: Memory manager has been shut down.
at org.apache.flink.runtime.util.TestingFatalErrorHandler.rethrowError(TestingFatalErrorHandler.java:51)
at org.apache.flink.runtime.taskexecutor.TaskExecutorTest.teardown(TaskExecutorTest.java:223)
Caused by: java.lang.IllegalStateException: Memory manager has been shut down.
at org.apache.flink.runtime.memory.MemoryManager.releaseAll(MemoryManager.java:480)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:821)
at java.lang.Thread.run(Thread.java:748)

After an investigation, I notice that it occurred when

Task#run set task state as FINISHED
Test case get the FINISHED future, call taskExecutor#shutdown, so the MemoryManager shutdown.
In finally block of Task#run, call MemoryManager#releaseAll(owner)
cause IllegalStateException

Here, I find that after MemoryManager#shutdown, allocatedSegments and memoryPool are released properly. Thus a following #releaseAll can be safely omitted and an IllegalStateException is no-need.
UPDATE basically it is an issue about execute order, I add a follow-up commit to ensure that the TaskExecutor shutdown after Task exit. We can revert the first commit but as pointed out above, maybe it is a better choice to omit a #releaseAll following #shutdown instead of causing a fatal error.
Brief change log
Omit actions in memory manager after shutdown instead of throwing IllegalStateException
Verifying this change
This change is already covered by existing tests, such as TaskExecutorTest.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

cc @tillrohrmann @NicoK @zhijiangW

Hi @TisonKun
Thank you for the contribution. I think the changes in test make sense. I am against though adjusting production code just to make test pass. I think those exception make sense and I would prefer to revert those changes.

authors: TisonKun dawidwys


text:
[FLINK-13121] [table-planner-blink] Set batch properties to runtime in blink batch executor

What is the purpose of the change

set batch properties to StreamGraph
set managedMemory size to ResourceSpec
set shuffleMode to PartitionTransformations.

Brief change log

set batch properties to StreamGraph
set managedMemory size to ResourceSpec
set shuffleMode to PartitionTransformations.

Verifying this change
UT
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: ( no)
The runtime per-record code paths (performance sensitive): (no )
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

travis failed on irrelevant case: https://travis-ci.org/KurtYoung/flink/builds/557582152

authors: KurtYoung XuPingyong


text:
Update defaultSystem/geode.properties

@davebarnes97 and I stumbled upon these when looking at some documentation

@moleske please use the GEODE number in the PR title and at least the first commit message.

My bad, was on GitHub when we saw it and forgot to check Jira before blindly updating comments

authors: moleske pivotal-jbarrett

repo: apache/geode-native


text:
GEODE-5373: Adjust timing for GMSHealthMonitorJUnitTest.testRemoveMem…

…berCalled

This test fails on Windows when run in conjunction with other tests - on its
own it always passes. There appears to be a thread scheduling difference in
Windows that is resulting in the Failure Detection Scheduler thread running
after the sleep interval.

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[ ] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[ ] Is your initial contribution a single, squashed commit?

[ ] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

precheckin is green

repo: apache/geode


text:
[GOBBLIN-1637] Add writer, operation, and partition info to failed metadata writer events

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1637

Description

[x] Here are some details about my PR, including screenshots (if applicable):

Add info of which metadata writer failed in the failure events

Add which operation type failed. To do this, split up the events so that there's a separate event if opType changes, otherwise the event offset ranges could contain multiple different offsets

Add which partition values were intended to be added/dropped as part of these GMCE. To do this added a HiveMetadataWriterWithPartitionInfoException.java only thrown from hive writer in order to pass partition info back to MCE Writer

Fix bug in hive metadata writer where failure in flush would throw RuntimeException, but catch block only catches IOException so fault tolerant behaviour would never work for flush failures

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Tested in test pipeline

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@ZihanLi58 please review

Codecov Report

Merging #3498 (9bf7629) into master (3e87795) will decrease coverage by 3.22%.
The diff coverage is 76.36%.

@@             Coverage Diff              @@
##             master    #3498      +/-   ##
============================================
- Coverage     46.68%   43.46%   -3.23%
+ Complexity    10406     2038    -8368
============================================
  Files          2078      405    -1673
  Lines         81283    17482   -63801
  Branches       9077     2137    -6940
============================================
- Hits          37946     7598   -30348
+ Misses        39839     9034   -30805
+ Partials       3498      850    -2648

Impacted Files
Coverage Δ

...pache/gobblin/iceberg/writer/GobblinMCEWriter.java
76.10% <74.00%> (-1.24%)
:arrow_down:

...bblin/iceberg/writer/GobblinMetadataException.java
100.00% <100.00%> (ø)

...a/org/apache/gobblin/util/limiter/NoopLimiter.java
40.00% <0.00%> (-20.00%)
:arrow_down:

...lin/util/filesystem/FileSystemInstrumentation.java
85.71% <0.00%> (-14.29%)
:arrow_down:

.../java/org/apache/gobblin/cluster/SleepingTask.java
39.39% <0.00%> (-6.07%)
:arrow_down:

...in/java/org/apache/gobblin/cluster/HelixUtils.java
31.70% <0.00%> (-4.27%)
:arrow_down:

...apache/gobblin/hive/writer/HiveMetadataWriter.java

...in/broker/gobblin_scopes/GobblinScopeInstance.java

...paction/action/CompactionGMCEPublishingAction.java

... and 1672 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e87795...9bf7629. Read the comment docs.

authors: codecov-commenter jack-moseley

repo: apache/gobblin


text:
[GOBBLIN-1695] Fix: Failure to add spec executors doesn't block deployment

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1695

Description

[ ] Here are some details about my PR, including screenshots (if applicable):

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3551 (c24a7d1) into master (e0d3c78) will increase coverage by 3.81%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3551      +/-   ##
============================================
+ Coverage     46.62%   50.44%   +3.81%
+ Complexity    10456     5693    -4763
============================================
  Files          2084     1043    -1041
  Lines         81620    39797   -41823
  Branches       9103     4471    -4632
============================================
- Hits          38058    20076   -17982
+ Misses        40043    17968   -22075
+ Partials       3519     1753    -1766

Impacted Files
Coverage Δ

...t/version/TimestampedDatasetStateStoreVersion.java
27.77% <0.00%> (-5.56%)
:arrow_down:

.../apache/gobblin/runtime/api/JobExecutionState.java
79.43% <0.00%> (-0.94%)
:arrow_down:

.../java/org/apache/gobblin/runtime/api/FlowSpec.java
40.10% <0.00%> (-0.84%)
:arrow_down:

...apache/gobblin/hive/writer/HiveMetadataWriter.java
0.00% <0.00%> (ø)

...anagement/copy/replication/ConfigBasedDataset.java
68.87% <0.00%> (ø)

...n/service/modules/orchestration/AzkabanClient.java

...ice/modules/orchestration/AzkabanSpecProducer.java

...vice/modules/core/GobblinServiceConfiguration.java

...oogle/webmaster/GoogleWebmasterDayPartitioner.java

.../apache/gobblin/config/client/ConfigClientCLI.java

... and 1039 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


text:
HDDS-3826. Split Ozone FS acceptance tests

What changes were proposed in this pull request?

Split monolithic Ozone FS acceptance test case into several steps.  This improves feedback about test progress, and makes easier to find out what's wrong in case of a failure.  (Unfortunately most subsequent steps still depend on earlier ones.  Improving that would be nice, but would make the test slower due to the necessary extra setup steps.)
Run test setup in proper Setup instead of a test case.
Split test setup into reusable keywords.
Add functions for formatting Ozone FS URLs with both o3fs and ofs flavor.  Includes its own test cases.
Test execution for ofs and o3fs is controlled outside of the test script.  This is necessary to avoid duplication of test cases and/or keywords.
Pass SECURITY_ENABLED and similar variables as Robot variables (-v), not environment ones (-e).  This allows setting default value in Robot tests, which makes it easier to run the test from command-line.

Most of this is in preparation for bucket links, as the same test should be executed for regular buckets and links.  Submitting as a separate PR for easier review.
Note that this PR is based on top of the change for #1091, a minor improvement which is necessary to save output of multiple executions of the same test suite (ozonefs.robot in this case).
https://issues.apache.org/jira/browse/HDDS-3826
How was this patch tested?
Ran tests for lib locally (no docker-compose):
$ robot lib
...
Lib.Fs Tests                                                          | PASS |
...
Lib.String Tests                                                      | PASS |
...

Ran Ozone FS tests inside docker container bypassing test.sh:
$ docker-compose exec scm robot smoketest/ozonefs/ozonefs.robot
...
Ozonefs :: Ozone FS tests                                             | PASS |
20 critical tests, 20 passed, 0 failed

$ docker-compose exec scm robot -v SCHEME:o3fs smoketest/ozonefs/ozonefs.robot
...
Ozonefs :: Ozone FS tests                                             | PASS |
20 critical tests, 20 passed, 0 failed

Full acceptance test suite:
https://github.com/adoroszlai/hadoop-ozone/runs/783679213
https://github.com/adoroszlai/hadoop-ozone/runs/784247369

Pass SECURITY_ENABLED and similar variables as Robot variables (-v), not environment ones (-e). This allows setting default value in Robot tests, which makes it easier to run the test from command-line.

DANCING BANANA. Thank you very much,
The patch LGTM.
One comment: string_test.robot and fs_test.robot seems to be duplicate of robot.robot. (I guess you refactored the robot.robot to two separated files.) I think one of them can be removed.

One comment: string_test.robot and fs_test.robot seems to be duplicate of robot.robot.  I think one of them can be removed.

Nice catch.

(I guess you refactored the robot.robot to two separated files.)

Exactly.

Thanks @elek for reviewing and committing it.

authors: adoroszlai elek

repo: apache/hadoop-ozone


text:
HADOOP-16380 S3Guard to determine empty directory status for all non-root directories

Gabor's PR #1106 merged with the test of mine in #1079 with that test modified to show that yes, the tombstone problem goes all the way down
Fix: pass down the needEmptyDirectory flag all the way down from S3AFileSystem to metastore.get

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
82
Docker mode activated.

_ Prechecks _

+1
dupname
1
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

0
mvndep
89
Maven dependency ordering for branch

+1
mvninstall
1257
trunk passed

+1
compile
1222
trunk passed

+1
checkstyle
143
trunk passed

+1
mvnsite
127
trunk passed

+1
shadedclient
983
branch has no errors when building and testing our client artifacts.

+1
javadoc
97
trunk passed

0
spotbugs
65
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
185
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
78
the patch passed

+1
compile
1266
the patch passed

+1
javac
1266
the patch passed

+1
checkstyle
140
the patch passed

+1
mvnsite
122
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
717
patch has no errors when building and testing our client artifacts.

+1
javadoc
99
the patch passed

+1
findbugs
199
the patch passed

_ Other Tests _

+1
unit
528
hadoop-common in the patch passed.

+1
unit
289
hadoop-aws in the patch passed.

+1
asflicense
47
The patch does not generate ASF License warnings.

7684

Subsystem
Report/Notes

Docker
Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1123

Optional Tests
dupname asflicense mvnsite compile javac javadoc mvninstall unit shadedclient findbugs checkstyle

uname
Linux 6b218744fdc8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 9838a47

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/1/testReport/

Max. process+thread count
1387 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
92
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ trunk Compile Tests _

0
mvndep
51
Maven dependency ordering for branch

+1
mvninstall
1228
trunk passed

+1
compile
1122
trunk passed

+1
checkstyle
161
trunk passed

+1
mvnsite
131
trunk passed

+1
shadedclient
1092
branch has no errors when building and testing our client artifacts.

+1
javadoc
119
trunk passed

0
spotbugs
87
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
246
trunk passed

_ Patch Compile Tests _

0
mvndep
31
Maven dependency ordering for patch

+1
mvninstall
109
the patch passed

+1
compile
1345
the patch passed

+1
javac
1345
the patch passed

+1
checkstyle
195
the patch passed

+1
mvnsite
171
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
890
patch has no errors when building and testing our client artifacts.

+1
javadoc
107
the patch passed

+1
findbugs
262
the patch passed

_ Other Tests _

+1
unit
655
hadoop-common in the patch passed.

+1
unit
300
hadoop-aws in the patch passed.

+1
asflicense
64
The patch does not generate ASF License warnings.

8391

Subsystem
Report/Notes

Docker
Client=18.09.7 Server=18.09.7 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1123

Optional Tests
dupname asflicense mvnsite compile javac javadoc mvninstall unit shadedclient findbugs checkstyle

uname
Linux 33e15d90e29a 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 4e66cb9

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/2/testReport/

Max. process+thread count
1737 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Updated patch fixes the invocation chain so needEmptyDirectoryFlag gets passed through. New test now works happily.
Tested: s3 ireland with s3guard and dynamo. I've not done the -scale option; kicking it off now

Patch tested at scale, S3 ireland -Ds3guard -Ddynamo -Dscale; no failures.
Latest patch

wires up the needEmptyDirectoryFlag again
fixed up the mocking tests to handle the relevant changes
added a special para about object store root directory deletion in filesystem.md

Latest patch: full test run @ scale with Ireland  + S3Guard + DDB; ran the empty dirs test with local: -Dit.test=ITestS3GuardEmptyDirs -Ds3guard -Dlocal to be sure the metrics checks are skipped when the store != DDB

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
35
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 5 new or modified test files.

_ trunk Compile Tests _

0
mvndep
24
Maven dependency ordering for branch

+1
mvninstall
1061
trunk passed

+1
compile
1075
trunk passed

+1
checkstyle
135
trunk passed

+1
mvnsite
118
trunk passed

+1
shadedclient
935
branch has no errors when building and testing our client artifacts.

+1
javadoc
87
trunk passed

0
spotbugs
66
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
180
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
78
the patch passed

+1
compile
1022
the patch passed

+1
javac
1022
the patch passed

+1
checkstyle
132
the patch passed

+1
mvnsite
114
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
648
patch has no errors when building and testing our client artifacts.

+1
javadoc
86
the patch passed

+1
findbugs
198
the patch passed

_ Other Tests _

+1
unit
525
hadoop-common in the patch passed.

+1
unit
275
hadoop-aws in the patch passed.

+1
asflicense
40
The patch does not generate ASF License warnings.

6802

Subsystem
Report/Notes

Docker
Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1123

Optional Tests
dupname asflicense mvnsite compile javac javadoc mvninstall unit shadedclient findbugs checkstyle

uname
Linux 28a7262667ce 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 7f1b76c

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/3/testReport/

Max. process+thread count
1380 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
38
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 5 new or modified test files.

_ trunk Compile Tests _

0
mvndep
24
Maven dependency ordering for branch

+1
mvninstall
1078
trunk passed

+1
compile
1094
trunk passed

+1
checkstyle
148
trunk passed

+1
mvnsite
123
trunk passed

+1
shadedclient
961
branch has no errors when building and testing our client artifacts.

+1
javadoc
87
trunk passed

0
spotbugs
63
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
181
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
79
the patch passed

+1
compile
1002
the patch passed

+1
javac
1002
the patch passed

+1
checkstyle
133
the patch passed

+1
mvnsite
117
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
687
patch has no errors when building and testing our client artifacts.

+1
javadoc
89
the patch passed

+1
findbugs
187
the patch passed

_ Other Tests _

+1
unit
518
hadoop-common in the patch passed.

+1
unit
292
hadoop-aws in the patch passed.

+1
asflicense
37
The patch does not generate ASF License warnings.

6900

Subsystem
Report/Notes

Docker
Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1123

Optional Tests
dupname asflicense mvnsite compile javac javadoc mvninstall unit shadedclient findbugs checkstyle

uname
Linux 9da2a211580f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 7f1b76c

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/4/testReport/

Max. process+thread count
1497 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
42
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 5 new or modified test files.

_ trunk Compile Tests _

0
mvndep
26
Maven dependency ordering for branch

+1
mvninstall
1095
trunk passed

+1
compile
1054
trunk passed

+1
checkstyle
137
trunk passed

+1
mvnsite
121
trunk passed

+1
shadedclient
947
branch has no errors when building and testing our client artifacts.

+1
javadoc
89
trunk passed

0
spotbugs
64
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
175
trunk passed

_ Patch Compile Tests _

0
mvndep
24
Maven dependency ordering for patch

+1
mvninstall
77
the patch passed

+1
compile
991
the patch passed

+1
javac
991
the patch passed

+1
checkstyle
135
the patch passed

+1
mvnsite
124
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
664
patch has no errors when building and testing our client artifacts.

+1
javadoc
91
the patch passed

+1
findbugs
193
the patch passed

_ Other Tests _

+1
unit
513
hadoop-common in the patch passed.

+1
unit
285
hadoop-aws in the patch passed.

+1
asflicense
42
The patch does not generate ASF License warnings.

6835

Subsystem
Report/Notes

Docker
Client=18.09.8 Server=18.09.8 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1123

Optional Tests
dupname asflicense mvnsite compile javac javadoc mvninstall unit shadedclient findbugs checkstyle

uname
Linux e4084db3a57c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 7f1b76c

Default Java
1.8.0_212

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/5/testReport/

Max. process+thread count
1380 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

tested against ireland with dynamo: known testMRJob failures
scale tests are running with dynamo
will run localms tests as well and report back with the results.

thanks. PR #1115 is working on the testMRJob failures; the initial patch will let us collect failures in the local FS; the test reports will also collect and print the result from the AM. and by working in the parallel phase again (yet still executing each MR job in an isolated sequence), we get a speedup of a minute or two.

@bgaborg , did that "1 approval"  mean a +1? If so, can you add it explicitly for the record. Thanks

-Dscale against ireland with dynamo: some test failed, so after the rerun:
[INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[ERROR] Tests run: 9, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 224.725 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir
[ERROR] testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 180.018 s  <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 180000 milliseconds

[ERROR] testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 31.867 s  <<< FAILURE!
java.lang.AssertionError:
After 12 attempts: listing after rm /* not empty
final [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null

deleted [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null

original [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null
[01] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null

[ERROR] testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)  Time elapsed: 2.905 s  <<< ERROR!
java.io.FileNotFoundException: about to be deleted file: not found s3a://gabota-versioned-bucket-ireland/tests3ascale in s3a://gabota-versioned-bucket-ireland/
	at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:85)
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://gabota-versioned-bucket-ireland/tests3ascale
	at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:85)

[INFO] Running org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter
[ERROR] Tests run: 7, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 67.798 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter
[ERROR] test_110_teragen(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter)  Time elapsed: 16.361 s  <<< FAILURE!
java.lang.AssertionError: Teragen(1000, s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin) failed expected:<0> but was:<1>

[ERROR] test_120_terasort(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter)  Time elapsed: 2.044 s  <<< ERROR!
java.io.FileNotFoundException: Output directory s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin from previous teragen stage not found: Job may not have executed: not found s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin in s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin

[ERROR] test_130_teravalidate(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter)  Time elapsed: 2.033 s  <<< ERROR!
java.io.FileNotFoundException: Output directory s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortout from previous terasort stage not found: Job may not have executed: not found s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortout in s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortout

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:97->Assert.fail:88 After 12 attempts: listing after rm /* not empty
final [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null

deleted [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null

original [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null
[01] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null

[ERROR]   ITestTerasortMagicCommitter>AbstractCommitTerasortIT.test_110_teragen:173->AbstractCommitTerasortIT.executeStage:148->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Teragen(1000, s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin) failed expected:<0> but was:<1>
[ERROR] Errors:
[ERROR]   ITestS3AContractRootDir.testListEmptyRootDirectory:85->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:192 ? FileNotFound
[ERROR]   ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:254 ? TestTimedOut
[ERROR]   ITestTerasortMagicCommitter>AbstractCommitTerasortIT.test_120_terasort:185->AbstractCommitITest.loadSuccessFile:473 ? FileNotFound
[ERROR]   ITestTerasortMagicCommitter>AbstractCommitTerasortIT.test_130_teravalidate:201->AbstractCommitITest.loadSuccessFile:473 ? FileNotFound
[INFO]
[ERROR] Tests run: 16, Failures: 2, Errors: 4, Skipped: 0

Ireland -Dlocal:
no unknown test failures other than testInitNegativeRead and testMRJob
Summary: +1 pending on fixing org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir#testRmEmptyRootDirNonRecursive, testListEmptyRootDirectory failures
I don't think that org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter#test_110_teragen,test_120_terasort,test_130_teravalidate would be related to this change.

I'm going to say the terasort failures are unrelated: somehow teragen failed and the successors refuse to work at that point.
Regarding the other failure, it looks potentially like a bug in the initial delete code which tried to delete a file which was already missing when it looked; that's the one rethrown as we only throw the first failure on the loop. not the final one. I'll deal with those for some better checks and reporting

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
92
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 7 new or modified test files.

_ trunk Compile Tests _

0
mvndep
70
Maven dependency ordering for branch

+1
mvninstall
1213
trunk passed

+1
compile
1007
trunk passed

+1
checkstyle
147
trunk passed

+1
mvnsite
126
trunk passed

+1
shadedclient
1042
branch has no errors when building and testing our client artifacts.

+1
javadoc
110
trunk passed

0
spotbugs
71
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
203
trunk passed

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

+1
mvninstall
84
the patch passed

+1
compile
1057
the patch passed

+1
javac
1057
the patch passed

-0
checkstyle
149
root: The patch generated 1 new + 36 unchanged - 0 fixed = 37 total (was 36)

+1
mvnsite
124
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
736
patch has no errors when building and testing our client artifacts.

+1
javadoc
97
the patch passed

+1
findbugs
215
the patch passed

_ Other Tests _

+1
unit
562
hadoop-common in the patch passed.

+1
unit
308
hadoop-aws in the patch passed.

+1
asflicense
57
The patch does not generate ASF License warnings.

7430

Subsystem
Report/Notes

Docker
Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1123

Optional Tests
dupname asflicense mvnsite compile javac javadoc mvninstall unit shadedclient findbugs checkstyle

uname
Linux 784008921b11 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 340bbaf

Default Java
1.8.0_212

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/6/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/6/testReport/

Max. process+thread count
1346 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1123/6/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

+1 - retested with ireland

thanks; I'll fix the checkstyle on the commit
./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java:192:      ContractTestUtils.assertDeleted(fs, status.getPath(), false,true, false);:66: ',' is not followed by whitespace. [WhitespaceAfter]

committed to trunk.

authors: bgaborg hadoop-yetus steveloughran

repo: apache/hadoop


text:
HADOOP-17576. ABFS: Disable throttling update for auth failures

Contributed by Sumangala Patki
(cherry picked from commit 6f640abbaf14efa98d6c599e5fff95647730ad42)

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
24m 16s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ branch-3.3 Compile Tests _

+1 :green_heart:
mvninstall
35m 52s

branch-3.3 passed

+1 :green_heart:
compile
0m 29s

branch-3.3 passed

+1 :green_heart:
checkstyle
0m 22s

branch-3.3 passed

+1 :green_heart:
mvnsite
0m 33s

branch-3.3 passed

+1 :green_heart:
javadoc
0m 26s

branch-3.3 passed

+1 :green_heart:
spotbugs
0m 57s

branch-3.3 passed

+1 :green_heart:
shadedclient
18m  3s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 28s

the patch passed

+1 :green_heart:
compile
0m 24s

the patch passed

+1 :green_heart:
javac
0m 24s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 16s

the patch passed

+1 :green_heart:
mvnsite
0m 26s

the patch passed

+1 :green_heart:
javadoc
0m 18s

the patch passed

+1 :green_heart:
spotbugs
1m  1s

the patch passed

+1 :green_heart:
shadedclient
17m 48s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
1m 56s

hadoop-azure in the patch passed.

+1 :green_heart:
asflicense
0m 28s

The patch does not generate ASF License warnings.

104m 47s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2885/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2885

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 7622969a9cac 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.3 / 8f215d397e702e6dc5752d0764003556a34a1449

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~18.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2885/1/testReport/

Max. process+thread count
513 (vs. ulimit of 5500)

modules
C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2885/1/console

versions
git=2.17.1 maven=3.6.0 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

TEST RESULTS
HNS Account Location: East US 2
NonHNS Account Location: East US 2, Central US
HNS-OAuth

[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 512, Failures: 0, Errors: 0, Skipped: 68
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:631 » TestTimedOut
[ERROR] Tests run: 261, Failures: 0, Errors: 1, Skipped: 50

HNS-SharedKey

[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[ERROR] Tests run: 512, Failures: 0, Errors: 0, Skipped: 24
[ERROR]   ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:62->testReadWriteAndSeek:78 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:631 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:631 » TestTimedOut
[ERROR] Tests run: 261, Failures: 0, Errors: 3, Skipped: 40

NonHNS-SharedKey

[INFO] Tests run: 93, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 512, Failures: 0, Errors: 0, Skipped: 247
[ERROR]   ITestAbfsReadWriteAndSeek.testReadAndWriteWithDifferentBufferSizesAndSeek:62->testReadWriteAndSeek:78 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractDistCp>AbstractContractDistCpTest.testDistCpWithIterator:631 » TestTimedOut
[ERROR]   ITestAbfsFileSystemContractSecureDistCp>AbstractContractDistCpTest.testDistCpWithIterator:631 » TestTimedOut
[ERROR] Tests run: 261, Failures: 0, Errors: 3, Skipped: 40

Note: The timeouts in some of the above tests are commonly observed due to network issues and are not related to this PR.
JIRAs for the timeouts: AbstractContractDistCpTest, ITestAbfsReadWriteAndSeek

authors: hadoop-yetus sumangala-patki


text:
HADOOP-16532: Fix TestViewFsTrash to use the correct homeDir.

Fixed the homeDir issue reported in HADOOP-16532. We are finally able to pass this test for ViewFS.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
16m 55s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
34m 19s

trunk passed

+1 :green_heart:
compile
22m 57s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
19m 33s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m  0s

trunk passed

+1 :green_heart:
mvnsite
1m 33s

trunk passed

+1 :green_heart:
javadoc
1m  4s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 34s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 25s

trunk passed

+1 :green_heart:
shadedclient
23m 48s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 55s

the patch passed

+1 :green_heart:
compile
22m  9s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
22m  9s

the patch passed

+1 :green_heart:
compile
19m 32s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
19m 32s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 58s
/results-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 2 new + 36 unchanged - 1 fixed = 38 total (was 37)

+1 :green_heart:
mvnsite
1m 30s

the patch passed

+1 :green_heart:
javadoc
1m  0s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 34s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 36s

the patch passed

+1 :green_heart:
shadedclient
24m 11s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m 16s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

217m 28s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3514

JIRA Issue
HADOOP-16532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 69e91936341f 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / abd4d0dc46ec18d8e9d511042c46d01166e07971

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/1/testReport/

Max. process+thread count
1353 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 49s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
36m 16s

trunk passed

+1 :green_heart:
compile
27m  6s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
23m  6s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m  4s

trunk passed

+1 :green_heart:
mvnsite
1m 41s

trunk passed

+1 :green_heart:
javadoc
1m 10s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 39s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 42s

trunk passed

+1 :green_heart:
shadedclient
24m 34s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  1s

the patch passed

+1 :green_heart:
compile
26m 13s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
26m 13s

the patch passed

+1 :green_heart:
compile
23m  0s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
23m  0s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  0s
/results-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 1 new + 36 unchanged - 1 fixed = 37 total (was 37)

+1 :green_heart:
mvnsite
1m 45s

the patch passed

+1 :green_heart:
javadoc
1m  6s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 39s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 48s

the patch passed

+1 :green_heart:
shadedclient
24m 28s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m 59s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 52s

The patch does not generate ASF License warnings.

221m 22s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3514

JIRA Issue
HADOOP-16532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux b399bc3d8133 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / af3c01b0d331d8a72ebbe6322b448743d4ff6015

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/2/testReport/

Max. process+thread count
2939 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/2/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 54s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
36m 14s

trunk passed

+1 :green_heart:
compile
27m 33s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
22m 40s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m  4s

trunk passed

+1 :green_heart:
mvnsite
1m 45s

trunk passed

+1 :green_heart:
javadoc
1m  4s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 41s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 34s

trunk passed

+1 :green_heart:
shadedclient
24m 51s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 17s

the patch passed

+1 :green_heart:
compile
27m  4s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
27m  4s

the patch passed

+1 :green_heart:
compile
22m 24s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
22m 24s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m  3s
/results-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 1 new + 36 unchanged - 1 fixed = 37 total (was 37)

+1 :green_heart:
mvnsite
1m 36s

the patch passed

+1 :green_heart:
javadoc
1m  2s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 33s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 41s

the patch passed

+1 :green_heart:
shadedclient
24m 44s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m 51s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

222m  6s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3514

JIRA Issue
HADOOP-16532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 38d06b833b1e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 86343d9c021ec77c1df3c1a7ef8e573b9a011b8d

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/3/testReport/

Max. process+thread count
3137 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/3/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 50s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
34m 16s

trunk passed

+1 :green_heart:
compile
22m 44s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
19m 33s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m  1s

trunk passed

+1 :green_heart:
mvnsite
1m 30s

trunk passed

+1 :green_heart:
javadoc
1m  4s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 33s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 24s

trunk passed

+1 :green_heart:
shadedclient
23m 48s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 56s

the patch passed

+1 :green_heart:
compile
22m 10s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
22m 10s

the patch passed

+1 :green_heart:
compile
19m 32s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
19m 32s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 58s

hadoop-common-project/hadoop-common: The patch generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37)

+1 :green_heart:
mvnsite
1m 29s

the patch passed

+1 :green_heart:
javadoc
0m 59s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 35s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 34s

the patch passed

+1 :green_heart:
shadedclient
23m 42s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m 23s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 50s

The patch does not generate ASF License warnings.

200m 49s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3514

JIRA Issue
HADOOP-16532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux f69c125a35fd 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / f7facb859522b8bb7709dea74cd57af33c0ccf9f

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/4/testReport/

Max. process+thread count
1252 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 24s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
36m 26s

trunk passed

+1 :green_heart:
compile
28m 50s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
24m 23s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m  0s

trunk passed

+1 :green_heart:
mvnsite
1m 39s

trunk passed

+1 :green_heart:
javadoc
1m  3s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 34s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 40s

trunk passed

+1 :green_heart:
shadedclient
24m 14s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  2s

the patch passed

+1 :green_heart:
compile
27m 35s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
27m 35s

the patch passed

+1 :green_heart:
compile
23m 49s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
23m 49s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 59s

hadoop-common-project/hadoop-common: The patch generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37)

+1 :green_heart:
mvnsite
1m 38s

the patch passed

+1 :green_heart:
javadoc
1m  3s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 36s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 53s

the patch passed

+1 :green_heart:
shadedclient
24m 17s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
17m 54s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 48s

The patch does not generate ASF License warnings.

226m  4s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/6/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3514

JIRA Issue
HADOOP-16532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux f006ef8bc7cf 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e747b47b1edb2196536c9c694cb12580a4fbe39d

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/6/testReport/

Max. process+thread count
3136 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/6/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 57s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  2s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 58s

trunk passed

+1 :green_heart:
compile
29m 16s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
24m 51s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m 12s

trunk passed

+1 :green_heart:
mvnsite
1m 52s

trunk passed

+1 :green_heart:
javadoc
1m 19s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 47s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 59s

trunk passed

+1 :green_heart:
shadedclient
26m 18s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m 17s

the patch passed

+1 :green_heart:
compile
28m 38s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
28m 38s

the patch passed

+1 :green_heart:
compile
24m 36s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
24m 36s

the patch passed

+1 :green_heart:
blanks
0m  1s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
1m  8s

hadoop-common-project/hadoop-common: The patch generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37)

+1 :green_heart:
mvnsite
1m 48s

the patch passed

+1 :green_heart:
javadoc
1m 11s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 49s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
3m 16s

the patch passed

+1 :green_heart:
shadedclient
26m 38s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
19m 51s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 57s

The patch does not generate ASF License warnings.

239m  2s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3514

JIRA Issue
HADOOP-16532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux ad8365a38b20 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e747b47b1edb2196536c9c694cb12580a4fbe39d

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/5/testReport/

Max. process+thread count
2077 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m  8s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
38m 20s

trunk passed

+1 :green_heart:
compile
26m 53s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
compile
23m 32s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
checkstyle
1m  9s

trunk passed

+1 :green_heart:
mvnsite
1m 45s

trunk passed

+1 :green_heart:
javadoc
1m 17s

trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 51s

trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 47s

trunk passed

+1 :green_heart:
shadedclient
27m 38s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  5s

the patch passed

+1 :green_heart:
compile
27m 10s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javac
27m 10s

the patch passed

+1 :green_heart:
compile
24m  5s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
javac
24m  5s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
1m 13s
/results-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 3 new + 35 unchanged - 2 fixed = 38 total (was 37)

+1 :green_heart:
mvnsite
1m 48s

the patch passed

+1 :green_heart:
javadoc
1m 13s

the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04

+1 :green_heart:
javadoc
1m 43s

the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

+1 :green_heart:
spotbugs
2m 51s

the patch passed

+1 :green_heart:
shadedclient
25m 58s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
18m 19s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 49s

The patch does not generate ASF License warnings.

232m 26s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/8/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3514

JIRA Issue
HADOOP-16532

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux ad53a9eb33ca 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 207c4f2cc75222d8fd6df6a9e86d455e39559bd5

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/8/testReport/

Max. process+thread count
3136 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3514/8/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

The patch has been merged. Closing.

authors: aajisaka hadoop-yetus xinglin