text:
[FLINK-3477] [runtime] Add hash-based combine strategy for ReduceFunction.

This adds ReduceHashTable, and modifies ReduceCombineDriver to use it instead of sorting. When an input element comes in, the driver probes the table for the key, and either inserts the element if it is not present, or updates it with one reduce step. When memory runs out, all the current elements (partial aggregates) in the table are sent to the output.
I haven't yet deleted the code of the sort based solution from the driver, but it seems that the hash-based solution is always faster (sometimes by a factor of 3), so there isn't a need for dynamically choosing between them. I will do some more performance tests to confirm this, and then probably delete the sort based code. The hash-based approach also has the advantage that the memory used is proportional to the number of different keys, instead of the number of input elements.
The interface of ReduceHashTable is a subset of the interface of CompactingHashTable, so in theory, it could replace CompactingHashTable everywhere (this is only JoinWithSolutionSet and coGroupWithSolutionSet, if I'm not mistaken). I haven't actually tried this yet though, but if you think that it worth to give it a go, then I will try it, and run some real performance tests. I've already added ReduceHashTable to HashTablePerformanceComparison, and when there is plenty of memory available, ReduceHashTable is about 5-10% faster than CompactingHashTable. It also uses ~15% less memory, and it also has the advantage that if the sizes of the records don't change during the updates, then no compactings are needed (because records are updated in-place), so performance doesn't degrade when we are close to occupying all the memory. (In contrast to CompactingHashTable, where updates start getting slow, when there is little free memory left.)
For details about the operation of ReduceHashTable, see the comment at the beginning of the file.
I've changed the ctor of RandomAccessInputView to not do a seek. (This behaviour is more practical for me, because I want to create an instance before allocating memory segments to it.) I verified at most places of usage that the code does an explicit seek after the ctor, but I couldn't see this in AbstractBlockResettableIterator, so I inserted an explicit seek there.
I fixed a small bug in HashTablePerformanceComparison: the *-1 was not doing anything, because that value was always 0 there.
The predefined aggregations (eg. DataSet.sum()) currently use groupReduce. I'm not sure what was the reason for this originally, but in light of this speedup of reduce now, maybe this could be reconsidered. (And the same stands for DataSet.distinct().)
Additionally, I have plans for making a different hash table for handling the case when the serializer can tell the size of the records. This would allow open addressing to be used with linear probing, which would avoid a few cache misses per element access, so I'm expecting maybe a 2x speedup from this. (This open addressing hash table would actually be simpler than ReduceHashTable.) The difficulty with this is that currently the serializers are not very smart in this respect (I mean knowing the size), so some improvements would be needed for them, but I have some ideas for this.

Hi @ggevay, thanks a lot for this PR! A hash-based combiner is a long desired feature on our list!
I haven't looked at the code yet, but I have a comments already.
You said, the combiner flushes the full hash table if it runs out of memory. Do you think it would be possible to track the update frequency of buckets and only flush the bucket with the least updates (or n buckets with least updates)? This might improve the performance for skewed input data.
Regarding your future plans:

I agree that the sort-based strategy for reduce combiners can be removed eventually, when the hash-based strategy proves to work well. I would like to give it a bit more exposure though, before we drop the code.
Porting the built-in aggregation functions, distinct, etc. from GroupReduceFunctions to ReduceFunctions sounds good. I think the reason for this design decision was that for the sort-based strategies ReduceFunctions did not had a benefit over GroupReduceFunctions. Instead they caused more method invocations (once for each input record) compared to once per key.
It would be great if you could open a JIRA and add a short design document / API proposal for the changes on the serializers that you talked about. This would allow the community to review and discuss your proposal.

I will shepherd this PR and hope to review it soon.
Thanks again, Fabian

Hello @fhueske, thanks for the comment and shepherding the PR!

You said, the combiner flushes the full hash table if it runs out of memory. Do you think it would be possible to track the update frequency of buckets and only flush the bucket with the least updates (or n buckets with least updates)? This might improve the performance for skewed input data.

If we flush only some of the elements when the memory is full, then the append position of the record area doesn't change, but only some holes appear in the record area, so we immediately need to do a compaction. Since the compaction traverses the entire table, this idea could only work if we flush a substantial amount of the elements (like half of them).
This is an interesting idea, but I'm finding it quite hard to form even just an intuitive understanding about its performance effects. I mean doing this would have some overhead, and at the moment I totally can't see how much skew in the data would make this worth it (or what percentage of the data should be flushed at a time, etc.).
I'll think about this some more, but getting right the trade-offs here would probably require quite a lot of work: experimentation with different variants of the algorithm, with differently skewed data, with different distinct keys / total number of input elements ratios, so we should probably postpone this to some later time, after this basic version is merged.

I agree that the sort-based strategy for reduce combiners can be removed eventually, when the hash-based strategy proves to work well. I would like to give it a bit more exposure though, before we drop the code.

OK.

Porting the built-in aggregation functions, distinct, etc. from GroupReduceFunctions to ReduceFunctions sounds good. I think the reason for this design decision was that for the sort-based strategies ReduceFunctions did not had a benefit over GroupReduceFunctions. Instead they caused more method invocations (once for each input record) compared to once per key.

I see, thanks! I'll keep this in mind for now, and open a JIRA when the dust settles around the hash-based combiner, and we see the performance differences more clearly.

It would be great if you could open a JIRA and add a short design document / API proposal for the changes on the serializers that you talked about. This would allow the community to review and discuss your proposal.

OK, I will.
Best,
Gábor

I've added some performance testing code and comments.

I've fixed a minor issue related to closing the table, and improved some comments.

Quick question: Is there a reason why there is no chained version of ReduceCombineDriver? (GroupReduceDriver and AllReduceDriver both have chained versions: SynchronousChainedCombineDriver and ChainedAllReduceDriver.) If it is just because no one got around to it yet, then I'll open a JIRA and implement it quickly.

No one got around to do it in an efficient/safe manner. There is a JIRA for it afaik.

Thanks! I've found the JIRA: https://issues.apache.org/jira/browse/FLINK-2246

I have added a commit to this PR that adds the chained version of ReduceCombineDriver. (It doesn't have the sort-based code, only the hash-based.)

I have changed the hash function to include that bit-magic that is done in the other hash tables. (Originally, I thought that it is not needed here, because I haven't realized that it is not only for diffusing clusters, but also to have the high bits affect the low bits of the hashes.)

I think exposing the feature through the API similarly to the way join strategies are exposed is the way to properly make this feature available for users. I would leave the well-tested sorting version the default for now.

I agree with @mbalassi. Adding a strategy hint to the API, similar to the join hints for inner and outer joins is a good idea.

I have added the strategy hint. At the moment, it defaults to sorting, but I couldn't actually find any case where hashing is slower, and it is often about 2 times faster.

What I'm not sure about is the closeUserCode call in ChainedReduceCombineDriver.closeTask. Those other chained drivers that have a running flag for indicating canceling, make this call only when the driver was not canceled. But those other chained drivers where there is no running flag seem to make this call also when they were canceled. What is the reasoning behind this situation?

Hi @ggevay, sorry it took me very long to review your PR.
As I said before, this is a very desirable feature and a solid implementation.
I think a few things can be improved. Especially the full stop to resize / rebuild / emit records might take some time, depending on the size of the table.
I have the following suggestions / ideas:

Split the single record into multiple partitions (i.e., use multiple RecordAreas). Each partition holds the data of multiple buckets. This allows to restrict rebuilding, compaction, etc. to only a part of the whole table. In fact this is what I meant in my initial comment about tracking the updates of buckets (which I confused with partitions...). Partitioning is also used in the other hash tables in Flink. It can also help to make your implementation more suitable for the final reduce case, because it allows to spill individual partitions to disk. In CompactingHashTable the max number of partitions is set to 32.
Should we think about linear hashing for resizing the table. This technique grows the table by splitting individual buckets without the need to reorganize the whole table.
Do you think it is possible to extract the ReduceFunction from the table? IMO this would be a cleaner design if we want to use the table instead of the CompactingHashTable.

What do you think?
In any case, we need to update the documentation for the added CombineHint. I would also be good to extend ReduceITCase with a few tests that use CombineHint.HASH.

In any case, we need to update the documentation for the added CombineHint. I would also be good to extend ReduceITCase with a few tests that use CombineHint.HASH.

OK, I'll do these.

the full stop to resize / rebuild / emit records might take some time, depending on the size of the table.

Emit is very fast, because EntryIterator is reading the record area sequentially. I don't think there is much room for improvement there. (Except perhaps your idea of storing the lengths as negative values, and skipping over abandoned records :))
Rebuilds because of compactions:
Typically, the size of the records don't change, in which case no compactions happen.
If the records change size, that's a mess. But even in this case, performance should be reduced by less than 2x, because the number of insertions done in rebuilds should be proportional to the number of normal update/insert operations, and one insert during rebuild is faster then one usual insert/update, because the linked lists are not traversed (only the bucket segments are accessed randomly; the elements are inserted to the beginning of the lists).
Rebuilds because of resizing:
If the serializer knows the length of the records in advance, then no resizings will happen, because we just calculate the final number of bucket segments at the beginning (see calcInitialNumBucketSegments). Unfortunately, currently the serializers almost never know the length, but I hope this situation will improve in the future (for example, see FLINK-3321).
If the serializer doesn't know the length, then how much time is spent with resizings is greatly affected by the average number of records per key. This is because the time spent with doing resizes is proportional to the number of elements inserted, and if there are lots of elements with matching keys, then most operations will be updates (rather than insertions). I will probably do some performance tests to confirm this, and see how much time is spent in resizes in the worst case (when every key has only one element). (Also, the "performance should be reduced by less than 2x" argument from above should be applicable in this case as well.)

Do you think it is possible to extract the ReduceFunction from the table?

This should be doable, I'll try.

Should we think about linear hashing for resizing the table.

Thanks, I didn't know about this algorithm. I'll think about this.

Split the single record into multiple partitions (i.e., use multiple RecordAreas). Each partition holds the data of multiple buckets. This allows to restrict rebuilding, compaction, etc. to only a part of the whole table.

Do you think that this would have a performance benefit? The time of a rebuild is linear with the size, so I don't see how doing it in chunks would improve performance. (I mean, the (frequency of rebuilds of a partition) / (frequency of rebuilds of the full table) is probably the same ratio as the (time to rebuild a partition) / (time to rebuild the full table).)

It can also help to make your implementation more suitable for the final reduce case

Yes, some partitioning will be needed for that. However, I would like to get this in first, and then I'll design the implementation of that case.

The benefit of having multiple partitions is that you do not need to go over the complete RecordArea but only over a partition. After the partition is compacted or was emitted, it's MemorySegments can be used by other Partitions as well. Depending on the strategy to choose the Partition to emit or compact (#segments in partition, update count on partition, #unique values in partition), this should improve the combine rate (less emitted records from the table) and also improve the pipelined behavior of the combiner, because you'll get more but much shorter pauses.

I'm thinking about partitioning, but again, I'm finding it too difficult to imagine how this algorithm behaves, when we emit/rebuild only some part of the table at a time. In the partitioned case, there seem to be a lot of potential for messing things up in complicated and hard to foresee ways.
For example, if I always emit the partition with the least number of segments, then what will actually happen is that I will kinda always emit the same partition (because after an emit, it starts from a "disadvantage" compared to the others), and the others just keep growing until they eat all the memory, so this degenerates into the situation where there isn't a single memory segment left for the smallest one and bad things happen.
In contrast, an advantage of the current design is that its performance characteristics are not that hard to grok, and it is relatively straightforward to see how the sizes of the 3 memory regions (bucket segments, record segments, staging segments) relate to each other.
Btw. I think I figured out a simple way to do even the final reduce with the spilling without partitioning the memory: when memory is full I write everything to disk, and I partition only at this time, by writing to different files (the same files across multiple spillings, so eg. the second spilling appends to the same files as the first one created), and then when the input ends, I recurse on processing the files one by one (and I start a new level of partitioning).

I think I've addressed all the comments, except documenting CombineHint (I will do that tomorrow). Thanks again for reviewing the PR!

I've added some more documentation.

Hi Gabor,
thanks for the updates and additions you made to the PR. If didn't go through them yet, but would like to follow up on the discussion and propose a few potential improvements.
I agree that using multiple partitions would make the implementation more complex and the reasoning about the behavior less straightforward. The motivation for partitions is twofold. First, it should helps to maintain a continuous data flow without backpressure effects. Second, it should improve the effectiveness of the combiner by avoiding to evict hot elements which are frequently updated which would happen for each full eviction of the table. Emitting the partition with the least segments wouldn't be a good strategy for the reasons you mentioned. However, a strategy could be to remove the partition with the smallest updates / keys ratio. However, this would require more thoughts…
Independent of the partitions discussion, I have a few ideas to make the compaction more effective. However, these would break the compatibility with the AbstractMutableHashTable interface. Maybe we can provide a common base class and special implementations for the SolutionSet case and the Combiner case?

Remember a pointer to the hole which is closest to the start of the RecordArea (or partition).
When compacting, emit all elements which are before the first hole. These elements are "cold" and have not been updated since the last compaction (assuming non-in-place updates, but in-place updates should not be compacted anyways). Compact all elements after the first hole as usual. The benefit of this is to free more segments by compaction which delays (or even completely avoids) full eviction, and thus improving the compaction rate for hot elements.
Do only compact if more than (for instance) 50% of space can be freed and do a full eviction otherwise. If there are too few holes in the table, most elements have not been touched (in case of non-in-place updates). Hence, we should just emit everything.

We can also think about adding a by-pass switch if the HT observes that the combiner does not help to significantly reduce the data (too many unique keys for not enough memory) but only adds additional overhead. For instance if the number of records is not reduced by 2x, we could simply forward the records to the collector (In case of partitions, this decision could be done on partition level).

Hello Fabian,
I did 3. (https://github.com/apache/flink/commit/ef644821d5417b992cfda366225bd5622faa9b2b), because the machinery for that was already in place (see the condition in compactOrThrow). I chose the threshold to be 5%. (This can probably be the same with the solution set case, because if lengths change a lot then we get very slow as memory load gets near the total memory, so it is probably better to indicate the memory problem to the user with an exception than to silently be very slow.)
I also did some changes to the tests.
For 2., the situation doesn't seem straightforward to me. For example, if there are not many length changes, then exactly the opposite should be done: we should emit from the end of the record area (rather than the beginning), because if there is skew in the data, then the more common keys will appear sooner, so they tend to appear near the beginning of the record area.
The other ideas are also interesting, and I would love to experiment with them, but unfortunately I don't really have that much time for this at the moment. So I would suggest to merge the non-partitioned version, and then the partitioned version can be implemented later when I or someone else has a lot of free time on their hands.
(Btw., it would be very interesting to try machine learning techniques for dynamically making these decisions that involve complicated trade-offs, based on the actual data:

Have some switches which control these things like

what part of the record area to emit (begin or end; how much)
at how much fragmentation should we do compacting instead of emitting
what load factor should trigger a resize
size of bucket area
how to choose which partition to emit
maybe even do spilling also in the combiner
whether to insert prefetch instructions for the random memory accesses that will probably involve a CPU cache miss (the trade-off here is that then you have to work with multiple consecutive input records at the same time, so you have to do extra copies if object reuse is enabled, which might cost a lot) (I have actually experimented with this a little, and there were 20-35% speedups, if copies are cheap)
... (it's easy to come up with many more)

Gather some statistics about what is happening, and turn them into features

avg. record size
#keys / #elements ratio
skew
time it takes to serialize a record
time it takes to run the ReduceFunction
ratio of updates that involve size changes
size is changing up or down on average
backpressure

that we are generating
that we get from our outputs (if this is large (eg. because of a saturated network), then we should set the switches to do more aggressive combining)

how many CPU cache misses occur while looking up keys (eg. for recognizing the situation where records with matching keys are often close to each other for some reason)
hash collisions (so that we can start with a more simple hash function (few percent speedup), and change it, if it is bad)
... (it's easy to come up with many more)

Train some machine learning model which will figure out how to set the switches based on the features

I think a pretty good speedup could result from tuning all these things to the actual data at hand.
Maybe in a few years, when data flow systems get more mature, then this can become a reality.)

OK, then lets keep the data in one partition for now. In case of var-length updates, this can default to a memory usage / combine behavior which is somewhat similar to the sort-based strategy: Filling the memory with records and emitting it (putting compaction aside).
I'll review the PR once more will run a few end-to-end benchmarks as well.
What kind of benchmarks have you done so far?

Did you check the combine rate (input / output ratio) compared to the sort-based strategy?
How much memory did you use for tests (upper bound)? Did you vary the memory?
Have you checked heap memory consumption / GC activity compared to the sort-based strategy?

It might take a few more days before I actually get to this, but it is on my list.
Thanks,
Fabian

@ggevay I would be interesting in helping you prepare a peel-flink-bundle for the benchmarks @fhueske mentioned.
It will make for a perfect use-case for what Peel is intended and a nice first contribution to the bundles repository.

OK, thanks @aalexandrov ! I wanted to try Peel anyway, this is a good opportunity.

@fhueske:
So far, I have only run benchmarks on my work laptop and home laptop.
I have used the newly created ReducePerformance, with varying the (number of elements / number of different keys) ratio, and also the memory usage (between 250 MB - 6 GB). The hash-based strategy was always faster, up to 3 times, and mostly by a factor of 1.5 - 2.5.
I have also used ReduceHashTableTest.testWithIntPair. To make this a benchmark test, the lines that deal with the reference hash table for checking the correct output should be commented out, and the time measurement commented in. (This only tests the hash table, not end-to-end; and it can't be directly compared with the sort based reduce.)
I have also used HashTablePerformanceComparison to compare with the other hash tables. This has the disadvantage that the workload here is not characteristic of a reduce (eg. no processRecordWithReduce calls, because the other hash tables don't have that).
I have also benchmarked using my game theory stuff (which originally prompted me to start working on this), and the hash-based combiner was faster here as well.

Did you check the combine rate (input / output ratio) compared to the sort-based strategy?

I didn't measure it directly, but ReducePerformance shows the difference nicely when the number of input elements is large enough that the sort-based strategy has to emit often, but the number of different keys is small enough that the hash-based strategy can fit all the keys in memory. In this case, the speedup of the final reduce phase is striking.

Have you checked heap memory consumption / GC activity compared to the sort-based strategy?

I haven't actually checked it, but it should be the same or better as with the sort-based strategy:

enabled object reuse: no allocations at all
disabled object reuse: one allocation per input element, and one allocation per reduce step

The sort-based strategy might additionally have allocations inside maybe the sorter and/or MergeIterator.

Great, thanks for the detailed response @ggevay!

@fhueske We will propose three experiments based on your suggestions in a Google Doc on Monday. Once we have fixed the setup we will prepare a Peel bundle an run them on one of the clusters in the lab.
I would be also happy to promote Peel by leading a joint blog post for the Peel website together with @ggevay and you if you are interested. I think that the hash-table makes for a perfect use-case.

I've added a Google Doc where we can collaborate on the design of the experiments.
Once we're fixed on that, we will proceed by implementing them. The code will be available in the flink-hashagg-experiments repository.

Thanks @aalexandrov. I added a few comments to the document.
Looking forward to the results :-)

I created a few sub-issues for the rather generic issue FLINK-2237.
FLINK-3477 reflects the changes of this PR. Can you please update the commit message to FLINK-3477. Thanks, Fabian

Done

I'll do the Peel thing after March 15, because I have to work on my PhD thesis proposal until then.

Hi @ggevay, how are things going?
I would like to add this feature soon and will do another review next week.
Have you started with the benchmarks or time to do them in the next 1 to 2 weeks?

Hi @fhueske,
I'm planning to do the benchmarks next week Monday-Wednesday.

Very good. Let's try to merge this soon. Looking forward to the numbers :-)

@fhueske We have used the Easter break to conduct the experiments. A preliminary writeup is in the Google Doc. @ggevay will provide the results analysis later today. Cheers!

Overall it seems that the hash-based combiner works better than the sort-based one for (a) uniform, or normal key distribution, and (b) fixed-length records.
For skewed key distribution (like Zipf) the two strategies are practically equal, and for variable-length record the extra effort in compacting the record offsets the advanges of the hash-based aggregation approach.

Thanks for doing these experiments! The results are quite convincing. I'm currently on vacation and will be back in about a week.

We've summarized the use-case around the hash aggregation experiments in a blog post on the Peel webpage.
If you follow the instructions from the Repeatability section you should be able to reproduce the results on other environments without too much hastle.
I hope that this will be the first of many Flink-related public Peel bundles.

Thanks @ggevay and @aalexandrov! I left a few comments in GDoc. I am not 100% sure about the results for changing record sizes and the overhead of compaction. Otherwise the results look good. 25% overall job improvement sounds very promising. :-)
I'll do another pass over the code in the next days. Thanks, Fabian

What is the status of this PR? Can we get this in and tested for the 1.1 release?

We talked about this yesterday with @fhueske at the Flink meetup, and the conclusion was that we will get it into 1.1. @fhueske has several comments that he will write here in a few days, and then I will make the (hopefully) final adjustments.

And @th0br0 is working on CombineHint.NONE.

Hi @ggevay as promised yesterday, I added my comments and suggestions. Let me know what you think.
@greghogan, would you like to take a look as well? Another pair of eyes are always a good idea.

This PR does also fix issue FLINK-2246 (add ChainedReduceCombineDriver). Would be nice to have that change in a separate commit before we merge.

This PR does also fix issue FLINK-2246 (add ChainedReduceCombineDriver). Would be nice to have that change in a separate commit before we merge.

It is in c722f58baef0803a60b3232ada2e50f091d8b917. (second commit of the PR)

Hi @fhueske,
Thanks for the comments, I think I have addressed all of them.

I have rebased to the current master.

@ggevay :+1:

+1 with just a few superficial comments.
Reading back through the discussion I see that there are many ideas for future performance enhancements. If not already suggested I'd like to consider skipping staging for fixed length records.
I'm missing why we can't update in place with smaller records. The deserializer is responsible for detecting the end of the record and we wouldn't need to change the pointer value when replacing with a smaller record.
CombineHint.NONE can be implemented in a new PR since this looks to be ready as-is.

Reading back through the discussion I see that there are many ideas for future performance enhancements. If not already suggested I'd like to consider skipping staging for fixed length records.

Thanks, I've added this to my notes.

I'm missing why we can't update in place with smaller records. The deserializer is responsible for detecting the end of the record and we wouldn't need to change the pointer value when replacing with a smaller record.

A problem would arise in EntryIterator: after reading a record, we wouldn't know where the next record starts. (As it is now, it always starts right after the previous.)
Thanks @greghogan for pushing this forward. I think I have addressed all your comments.

CI tests are passing. I've been testing Gelly algorithms with this without error. I will merge this ...

authors: aalexandrov fhueske ggevay greghogan mbalassi zentol


text:
[FLINK-27807][Connectors/JDBC] The improvement of addBatch is empty w…

What is the purpose of the change
The improvement of addBatch is empty when jdbc batch submit
Brief change log
Increase execution flags to ensure empty batches are not executed
Verifying this change
This change is already covered by existing tests
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? no

@wuchong Can you help review this little PR, tks!

We've now moved the code from the JDBC connector to its own dedicated repository; please re-route this PR to https://github.com/apache/flink-connector-jdbc

source: gharchive/pull-request

authors: MartijnVisser shmilygkd

repo: apache/flink

license_type: permissive

license_source: github-api


text:
[FLINK-4997] [streaming] Introduce ProcessWindowFunction

This is an updated/enhanced version of #2756.
I did roughly these changes:

Add support for the AggregatingFunction/ProcessWindowFunction combination
Mark the new methods/interfaces as @PublicEvolving
Change how the Scala wrapper functions work because we recently moved all of them to use WrappingFunction. This is not complete yet and I created issues for fixing that: https://issues.apache.org/jira/browse/FLINK-5740 and https://issues.apache.org/jira/browse/FLINK-5741
Add tests in WindowTranslationTest.java and WindowTranslationTest.scala.

@VenturaDelMonte and @manuzhang, would you like to have a look at this. I took quite a while because there was some upheaval in the internals of that part of the code and the tests. I'm sorry for that.

I checked the code and it looks fine according to me. I see you are dealing with RichFunction inheritance in wrapper classes. It is something that bothered me too when I was working on the previous PR. Actually there could be a similar minor issue in RichProcessWindowFunction due to duplicate code, however I cannot see any easy workaround for it.
Regarding the overall feature, are you planning to do anything similar for #2946 or should I reflect these changes also there?

LGTM. Really excited to see we move further in this direction, and faster if possible.

@VenturaDelMonte I opened this: https://issues.apache.org/jira/browse/FLINK-5740. I want to make WrappingFunction an interface because, as you noted, RichProcessWindowFunction does currently not work.
I would merge this then. If you want you can pick up the work on the all-window case again and carry it to an end in the same way. What do you think?

Sure, I can take care of the all-window case!

authors: VenturaDelMonte aljoscha manuzhang


text:
[FLINK-6916][blob] add API to allow job-related BLOBs to be stored

This PR is the fourth in a series for FLIP-19, based upon #4174. It allows (CONTENT_ADDRESSABLE) BLOBs to not only be stored job-independently but also based on a jobId which will become the future default for JARs, TaskDeploymentDescriptor data, and DistributedCache data.
Please note that we will rework/reorganise the whole BLOB store APIs completely in a future request. This PR focusses on the added functionality.

superseeded by #4237


text:
[docs] Fix typo in Python API docs

Rename Tockenizer to Tokenizer in python docs

@zentol I found this browsing through the docs

merging.


text:
[FLINK-10853] Provide end-to-end test for Kafka 0.9 connector

What is the purpose of the change
This pull request provides end-to-end test for Kafka 0.9 connector
Brief change log

Provide end-to-end test for Kafka 0.9 connector

Verifying this change
This change is already covered by existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

cc @pnowojski

@pnowojski Yes, I found some testing problems, this PR is not ready yet, so I didn't ping you more. Some of the issues you mentioned, I have found out, but not only that, I found that the shell method of sending and reading Kafka messages is not compatible with higher Kafka versions (parameters). I haven't had time to fix it yet, I will ping you when I am ready.

Ok, let me know once that is ready. Keep in mind that if it is not easy to fix, we can live without 0.9 e2e test.

authors: pnowojski yanghua


text:
[FLINK-12708][table] Introduce new source and sink interfaces to make Blink runner work

What is the purpose of the change
In order to support Blink runner (especially Blink batch source/sink and lookup source), we need some new source&sink interfaces and changes about TableSchema.

AsyncTableFunction: used as async lookup function, to support async temporal table join (i.e. dimension table join.).
LookupableTableSource: The LookupableTableSource interface adds support for the table to be accessed via key column(s) in a lookup fashion in order to support temporal table join.
BoundedTableSource: used for batch table source, returns a bounded DataStream, not a InputFormat, because we also want to reuse existing streaming source implementation to support bounded source, for example: bounded Kafka source.
BoundedTableSink: used for batch table sink, emit a bounded DataStream, not a OutputFormat. The reason is the same as above.

Brief change log
This contribution contains 4 commits:

https://github.com/apache/flink/pull/8678/commits/34a36ecc330149da5788799f8b678e427539e348 Introduce BoundedTableSource

add isBounded interface to StreamTableSource
BoundedTableSource extends StreamTableSource and expose getInputFormat
removes BatchTableSource and StreamTableSource in blink planner
support it in blink and flink planner

https://github.com/apache/flink/pull/8678/commits/0ed8b9315cefa413eac8abf604126d30c1dd5af0 Introduce BoundedTableSink

BoundedTableSink extends StreamTableSink expose getInputFormat
removes BatchTableSink in blink planner
support it in blink and flink planner

https://github.com/apache/flink/pull/8678/commits/1a978ad6c2beb5493f31ce88999d54cd5037ab05 Introduce LookupableTableSource

removes LookupableTableSource and LookupConfig in blink planner
support it only in blink planner

https://github.com/apache/flink/pull/8678/commits/d1caae0635fcafae602744d2efee9005438d1484 Expose getTableStats in TableSource

support it in blink and flink planner

Verifying this change
Add some unit tests and integration tests
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

Hi @twalthr  @aljoscha ,  it would be great if you have time to review it.

Comments addressed and conflicts solved.

Hi @aljoscha , Thanks very much for the suggestions.
Regarding to the TableStats. The TableStats is the statistics of a table (e.g. row count, ndv, max/min...). It is important information during optimization and is important for the performance improvement of TP-CDS. The TableStats was already there but didn't work with TableSource. So we expose the TableStats in TableSource, to support reporting statistics for a table source in this PR.
Regarding to the name of LookupableTableSource, do you think the lookupable is not appropriate? We use the lookup terminology in Blink runner including lookup join, lookup function, lookup tablesource.
If we use QueryableTableSource it might be not align with other terminologies. And we add a able suffix to lookup to indicate it is a table source which has the ability to lookup, similar to FilterableTableSource.
Regarding to the only one getLookupFunction(), I'm afraid we can't, because AsyncTableFunction doesn't extend TableFunction. The same super interface is UserDefinedFunction. But IMO, returning a UserDefinedFunction is not appropriate here.
I have an idea to make the interface only have one method. That is introuding a LookupFunctionProvider. I have created a gist for this idea: https://gist.github.com/wuchong/78fa3fa40323543aac8297fbecae09d8. What do you think about this?
Regarding to how to implement a lookup function, yes, it is not clear in the Javadoc. Because lookup join uses TableFunction as a lookup function, the TableFunction is not just used as lookup. So we may can't add the lookup function implementation details to the Javadoc of TableFunction.
I will add some more instruction to the Javadoc of getLookupFunction() and also the documentation with example in the future.

Cool, thanks! I think this is good to go now. 👍
We are marking the new lookup and async interfaces as @Experimental so we have some leeway in the future when we need to do changes.
However merges this: please make sure to merge the fixups into the respective original commits.

Of course, also only merge when Travis is green. 😅

Cool! Thanks @aljoscha . I will fix the CI.

authors: aljoscha wuchong


text:
[FLINK-13230] [pubsub] Add retry mechanism to acknowledging pubsub me…

What is the purpose of the change
Follow up on PubSub connector PR. This adds a retry mechanism to acknowledging pubsub messages. Google's PubSub is unstable enough for this to be a must-have imo.
Brief change log
Verifying this change
Trivial change.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): Yes, unless we can add this to 1.9. The pubsub connector has not yet been released.
The serializers: no
The runtime per-record code paths (performance sensitive): Yes, but only for the pubsub connector, not the flink core code.
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature?  no
If yes, how is the feature documented? JavaDocs

@becketqin as you've looked at the pubsub code the most: Would you be so kind to have a look at this PR? We can then close https://issues.apache.org/jira/browse/FLINK-13083

The changes in this PR have been applied (with attribution to @Xeli!) to the new FLIP-27 Pub/Sub connector in https://github.com/apache/flink-connector-gcp-pubsub/pull/2, and this PR might be closeable.  What do you think?

thanks @RyanSkraba
yes, since pubsub connector is externalised it makes sense to close it

authors: RyanSkraba Xeli snuyanzin


text:
groovyConsole: show line numbers when launched with a filename

When using the groovyConsole filename invocation format,
only the first line was numbered.

+1

Merged, thanks.

Related Jira: GROOVY-9434

Thank you for doing all the necessary house keeping (Jira, merging to the right branch) 😊👍

authors: danielsun1106 paulk-asert vmj

repo: apache/groovy


text:
HADOOP-18327.Fix eval expression in hadoop-functions.sh

Description of PR
Need to fix the eval expression.

Prefix exec by eval in Hadoop bin scripts Prior to this change, if HADOOP_OPTS contains any arguments that include a space, the command is not parsed correctly. For example, if HADOOP_OPTS="... -XX:OnOutOfMemoryError="kill -9 %p" ...", the bin/hadoop script will fail with the error "Unrecognized option: -9". No amount of clever escaping of the quotes or spaces in the "kill -9 %p" command will fix this. The only alternative appears to be to use 'eval'. Switching to use 'eval' instead of 'exec' also works, but it results in an intermediate bash process being left alive throughout the entire lifetime of the Java proces being started. Using 'exec' prefixed by 'eval' as has been done in this commit gets the best of both worlds, in that options with spaces are parsed correctly, and you don't end up with an intermediate bash process as the parent of the Java process.

We can replace single quote with escape-char and a double quote.

JIRA - HADOOP-18327
For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 44s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
shelldocs
0m  0s

Shelldocs was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 41s

trunk passed

+1 :green_heart:
mvnsite
1m 44s

trunk passed

+1 :green_heart:
shadedclient
19m 17s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
1m  7s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
1m 25s

the patch passed

+1 :green_heart:
shellcheck
0m  3s

No new issues.

+1 :green_heart:
shadedclient
19m  3s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
2m  1s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 52s

The patch does not generate ASF License warnings.

86m 14s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4536/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4536

Optional Tests
dupname asflicense mvnsite unit codespell detsecrets shellcheck shelldocs

uname
Linux 5cbaa84870a5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 15d9590aa3b41cdafea4e36852fbe6071d4c3c8d

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4536/1/testReport/

Max. process+thread count
581 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4536/1/console

versions
git=2.25.1 maven=3.6.3 shellcheck=0.7.0

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Can you share the way to reproduce the issue? It looks like we can not get expected outcome regardless of the patch due to another reason.
$ export HADOOP_OPTS="-XX:OnOutOfMemoryError='kill -9 %p'"
$ echo $HADOOP_OPTS
-XX:OnOutOfMemoryError='kill -9 %p'

$ bin/hadoop version
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
/home/iwasakims/dist/hadoop-3.3.3/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
Unrecognized option: -9
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

$ bash -x -c 'export SHELLOPTS && bin/hadoop version' 2>&1 | grep OnOutOf
++ export 'HADOOP_OPTS=-XX:OnOutOfMemoryError='\''kill -9 %p'\'''
++ HADOOP_OPTS='-XX:OnOutOfMemoryError='\''kill -9 %p'\'''
++ hadoop_debug 'Initial HADOOP_OPTS=-XX:OnOutOfMemoryError='\''kill -9 %p'\'''
+ HADOOP_OPTS='-XX:OnOutOfMemoryError='\''kill -9 %p'\'' '
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ yarn.log.dir ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dyarn.log.dir=/home/iwasakims/dist/hadoop-3.3.3/logs'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ yarn.log.file ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dyarn.log.file=hadoop.log'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ yarn.home.dir ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dyarn.home.dir=/home/iwasakims/dist/hadoop-3.3.3'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ yarn.root.logger ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dyarn.root.logger=INFO,console'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ java.library.path ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Djava.library.path=/home/iwasakims/dist/hadoop-3.3.3/lib/native'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ hadoop.log.dir ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dhadoop.log.dir=/home/iwasakims/dist/hadoop-3.3.3/logs'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ hadoop.log.file ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dhadoop.log.file=hadoop.log'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ hadoop.home.dir ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dhadoop.home.dir=/home/iwasakims/dist/hadoop-3.3.3'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ hadoop.id.str ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dhadoop.id.str=iwasakims'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ hadoop.root.logger ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dhadoop.root.logger=INFO,console'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ hadoop.policy.file ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dhadoop.policy.file=hadoop-policy.xml'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ [[ ! -XX:OnOutOfMemoryError='kill -9 %p'  =~ hadoop.security.logger ]]
+ eval 'HADOOP_OPTS='\''-XX:OnOutOfMemoryError='\''kill -9 %p'\''  -Dhadoop.security.logger=INFO,NullAppender'\'''
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
+ hadoop_debug 'Final HADOOP_OPTS: -XX:OnOutOfMemoryError='\''kill -9 %p'\'' '
+ exec /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dproc_version '-XX:OnOutOfMemoryError='\''kill' -9 '%p'\''' org.apache.hadoop.util.VersionInfo

@iwasakims  - Can you please review the latest changes, Thanks.

Thanks @ashutoshcipher for the patch. WIthout this patch the below fails.
export HADOOP_OPTS="-XX:OnOutOfMemoryError='kill -9 %p'"
[root@ip-17-3-4-1 usr]# hadoop fs -ls /
Unrecognized option: -9

@iwasakims Do you have any review comments.

Hmm, hadoop_add_param did not worked if the value contains ". (HADOOP_OPTS='-XX:OnOutOfMemoryError="kill -9 %p"' instead of previous HADOOP_OPTS="-XX:OnOutOfMemoryError='kill -9 %p'") @ashutoshcipher
$ export HADOOP_OPTS='-XX:OnOutOfMemoryError="kill -9 %p"'
$ echo $HADOOP_OPTS
-XX:OnOutOfMemoryError="kill -9 %p"

$ bash -x -c 'export SHELLOPTS && bin/hadoop version' 2>&1 | less
...
+ hadoop_add_param HADOOP_OPTS yarn.log.dir -Dyarn.log.dir=/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/logs
+ [[ ! -XX:OnOutOfMemoryError="kill -9 %p"  =~ yarn.log.dir ]]
+ eval 'HADOOP_OPTS="-XX:OnOutOfMemoryError="kill -9 %p"  -Dyarn.log.dir=/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/logs"'
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
++ -9 '%p  -Dyarn.log.dir=/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/logs'
/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
...

It would be nice if you can add test cases to files under hadoop-common-project/hadoop-common/src/test/scripts in order to see if no regression.

You can run the tests by mvn test -Pshelltest -Dtest=x on hadoop-common-project/hadoop-common. (You need to install bats for this.)

Hmm, hadoop_add_param does not work if the value contains ". (HADOOP_OPTS='-XX:OnOutOfMemoryError="kill -9 %p"' instead of previous HADOOP_OPTS="-XX:OnOutOfMemoryError='kill -9 %p'") @ashutoshcipher
$ export HADOOP_OPTS='-XX:OnOutOfMemoryError="kill -9 %p"'
$ echo $HADOOP_OPTS
-XX:OnOutOfMemoryError="kill -9 %p"

$ bash -x -c 'export SHELLOPTS && bin/hadoop version' 2>&1 | less
...
+ hadoop_add_param HADOOP_OPTS yarn.log.dir -Dyarn.log.dir=/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/logs
+ [[ ! -XX:OnOutOfMemoryError="kill -9 %p"  =~ yarn.log.dir ]]
+ eval 'HADOOP_OPTS="-XX:OnOutOfMemoryError="kill -9 %p"  -Dyarn.log.dir=/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/logs"'
++ HADOOP_OPTS=-XX:OnOutOfMemoryError=kill
++ -9 '%p  -Dyarn.log.dir=/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/logs'
/home/rocky/dist/hadoop-3.4.0-SNAPSHOT/bin/../libexec/hadoop-functions.sh: line 1143: -9: command not found
...

@iwasakims
Prefix exec by eval in Hadoop bin scripts Prior to this change, if HADOOP_OPTS contains any arguments that include a space, the command is not parsed correctly. For example, if HADOOP_OPTS="... -XX:OnOutOfMemoryError="kill -9 %p" ...", the bin/hadoop script will fail with the error "Unrecognized option: -9". No amount of clever escaping of the quotes or spaces in the "kill -9 %p" command will fix this. The only alternative appears to be to use 'eval'. Switching to use 'eval' instead of 'exec' also works, but it results in an intermediate bash process being left alive throughout the entire lifetime of the Java proces being started. Using 'exec' prefixed by 'eval' as has been done in this commit gets the best of both worlds, in that options with spaces are parsed correctly, and you don't end up with an intermediate bash process as the parent of the Java process.

@ashutoshcipher If your suggestion is to add support spaces enclosed in single quote (') like -XX:OnOutOfMemoryError='kill -9 %p', you should add the statement to documentation and tests showing the spec.

add support spaces enclosed in single quote (')

and not to support enclosing vars with double quote (")

authors: PrabhuJoseph ashutoshcipher hadoop-yetus iwasakims

repo: apache/hadoop


text:
HADOOP-18962. Upgrade kafka to 3.4.0

Description of PR
Upgrade kafka to 3.4.0 to fix CVE
How was this patch tested?
Built on local. ran  unit tests of hadoop-tools/hadoop-kafka project
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 27s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
xmllint
0m  0s

xmllint was not available.

+0 :ok:
shelldocs
0m  0s

Shelldocs was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+0 :ok:
mvndep
13m 54s

Maven dependency ordering for branch

+1 :green_heart:
mvninstall
19m 25s

trunk passed

+1 :green_heart:
compile
9m 14s

trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
compile
8m 21s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
mvnsite
12m 33s

trunk passed

+1 :green_heart:
javadoc
5m 35s

trunk passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
4m 53s

trunk passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
shadedclient
30m  6s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 30s

Maven dependency ordering for patch

+1 :green_heart:
mvninstall
17m 28s

the patch passed

+1 :green_heart:
compile
9m  3s

the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javac
9m  3s

the patch passed

+1 :green_heart:
compile
8m 17s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
javac
8m 17s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
mvnsite
8m  6s

the patch passed

+1 :green_heart:
shellcheck
0m  0s

No new issues.

+1 :green_heart:
javadoc
5m 34s

the patch passed with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04

+1 :green_heart:
javadoc
4m 51s

the patch passed with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

+1 :green_heart:
shadedclient
31m 25s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
224m 32s
/patch-unit-root.txt
root in the patch failed.

+0 :ok:
asflicense
0m 31s

ASF License check generated no output?

397m 51s

Reason
Tests

Failed junit tests
hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer

hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics

hadoop.hdfs.server.datanode.TestDataNodeReconfiguration

hadoop.hdfs.server.datanode.TestBatchIbr

hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport

hadoop.hdfs.server.datanode.TestCachingStrategy

hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting

hadoop.hdfs.server.datanode.TestBlockScanner

hadoop.hdfs.server.datanode.TestHSync

hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer

hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6247/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6247

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs

uname
Linux 887bae761035 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ea5be17499f2d507f3e1e3fddc950d52cd00074b

Default Java
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6247/1/testReport/

Max. process+thread count
3552 (vs. ulimit of 5500)

modules
C: hadoop-project . U: .

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6247/1/console

versions
git=2.25.1 maven=3.6.3 shellcheck=0.7.0

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

authors: dmmkr hadoop-yetus


text:
Backport: HBASE-24159 [flakey test] regionserver.TestRegionMergeTransactionOnCl…

…uster (#1472)
Delete tables after each test cases to avoid polluting states.
Signed-off-by: stack stack@apache.org

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 44s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
3m 47s
branch-2 passed

+1 :green_heart:
checkstyle
1m 15s
branch-2 passed

+1 :green_heart:
spotbugs
2m  7s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 19s
the patch passed

-0 :warning:
checkstyle
1m 13s
hbase-server: The patch generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
11m 35s
Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2.

+1 :green_heart:
spotbugs
2m 12s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 12s
The patch does not generate ASF License warnings.

35m 42s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1474

Optional Tests
dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle

uname
Linux 05170c09c975 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 823cf97495

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt

Max. process+thread count
84 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 54s
Docker mode activated.

-0 :warning:
yetus
0m  6s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
4m 15s
branch-2 passed

+1 :green_heart:
compile
1m  9s
branch-2 passed

+1 :green_heart:
shadedjars
5m 44s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 45s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 10s
the patch passed

+1 :green_heart:
compile
1m  5s
the patch passed

+1 :green_heart:
javac
1m  5s
the patch passed

+1 :green_heart:
shadedjars
5m 39s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 42s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
130m  5s
hbase-server in the patch passed.

156m 41s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1474

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 8cc215b55c98 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 823cf97495

Default Java
1.8.0_232

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/testReport/

Max. process+thread count
3776 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 16s
Docker mode activated.

-0 :warning:
yetus
0m  6s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
4m 21s
branch-2 passed

+1 :green_heart:
compile
1m  7s
branch-2 passed

+1 :green_heart:
shadedjars
5m 27s
branch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 43s
hbase-server in branch-2 failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m  3s
the patch passed

+1 :green_heart:
compile
1m  9s
the patch passed

+1 :green_heart:
javac
1m  9s
the patch passed

+1 :green_heart:
shadedjars
5m 32s
patch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 40s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
196m 51s
hbase-server in the patch passed.

223m  6s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1474

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux ce121115794d 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 823cf97495

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/testReport/

Max. process+thread count
2270 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1474/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

authors: Apache-HBase huaxiangsun

repo: apache/hbase


text:
HBASE-26029 It is not reliable to use nodeDeleted event to track regi…

…on server's death

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 19s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 20s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 12s
master passed

+1 :green_heart:
compile
2m  8s
master passed

+1 :green_heart:
shadedjars
9m  5s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  5s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

-1 :x:
mvninstall
2m 22s
root in the patch failed.

+1 :green_heart:
compile
2m  8s
the patch passed

+1 :green_heart:
javac
2m  8s
the patch passed

-1 :x:
shadedjars
7m  9s
patch has 11 errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 36s
hbase-server generated 3 new + 21 unchanged - 0 fixed = 24 total (was 21)

_ Other Tests _

+1 :green_heart:
unit
0m 47s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 37s
hbase-replication in the patch passed.

-1 :x:
unit
1m  5s
hbase-server in the patch failed.

36m  6s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux 4135e2abcdaf 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / bffe8957c5

Default Java
AdoptOpenJDK-1.8.0_282-b08

mvninstall
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt

shadedjars
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt

javadoc
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/testReport/

Max. process+thread count
271 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
6m 58s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 20s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 54s
master passed

+1 :green_heart:
compile
2m 38s
master passed

+1 :green_heart:
shadedjars
9m  3s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 11s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for patch

-1 :x:
mvninstall
2m 52s
root in the patch failed.

+1 :green_heart:
compile
2m 39s
the patch passed

+1 :green_heart:
javac
2m 39s
the patch passed

-1 :x:
shadedjars
7m 15s
patch has 11 errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 42s
hbase-server generated 3 new + 86 unchanged - 0 fixed = 89 total (was 86)

_ Other Tests _

+1 :green_heart:
unit
1m  3s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 42s
hbase-replication in the patch passed.

-1 :x:
unit
1m 18s
hbase-server in the patch failed.

43m 52s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux f00a6aecc838 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / bffe8957c5

Default Java
AdoptOpenJDK-11.0.10+9

mvninstall
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt

shadedjars
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt

javadoc
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk11-hadoop3-check/output/diff-javadoc-javadoc-hbase-server.txt

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/testReport/

Max. process+thread count
294 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 33s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
prototool
0m  1s
prototool was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 22s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 50s
master passed

+1 :green_heart:
compile
5m  6s
master passed

+1 :green_heart:
checkstyle
1m 30s
master passed

+1 :green_heart:
spotbugs
6m  9s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

-1 :x:
mvninstall
2m  5s
root in the patch failed.

+1 :green_heart:
compile
4m 55s
the patch passed

+1 :green_heart:
cc
4m 55s
the patch passed

-0 :warning:
javac
3m  8s
hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193)

-0 :warning:
checkstyle
1m  1s
hbase-server: The patch generated 3 new + 65 unchanged - 0 fixed = 68 total (was 65)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

-1 :x:
hadoopcheck
2m 16s
The patch causes 11 errors with Hadoop v3.1.2.

-1 :x:
hadoopcheck
4m 41s
The patch causes 11 errors with Hadoop v3.2.1.

-1 :x:
hadoopcheck
7m  4s
The patch causes 11 errors with Hadoop v3.3.0.

-1 :x:
hbaseprotoc
0m 31s
hbase-server in the patch failed.

+1 :green_heart:
spotbugs
6m 59s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 32s
The patch does not generate ASF License warnings.

45m  4s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
dupname asflicense cc hbaseprotoc prototool javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux 8370a9cdbb52 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / bffe8957c5

Default Java
AdoptOpenJDK-1.8.0_282-b08

mvninstall
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt

javac
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt

checkstyle
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt

hadoopcheck
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/patch-javac-3.1.2.txt

hadoopcheck
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/patch-javac-3.2.1.txt

hadoopcheck
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/patch-javac-3.3.0.txt

hbaseprotoc
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/artifact/yetus-general-check/output/patch-hbaseprotoc-hbase-server.txt

Max. process+thread count
95 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/1/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 28s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
prototool
0m  0s
prototool was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 22s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 55s
master passed

+1 :green_heart:
compile
5m  6s
master passed

+1 :green_heart:
checkstyle
1m 27s
master passed

+1 :green_heart:
spotbugs
6m 18s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 38s
the patch passed

+1 :green_heart:
compile
5m  6s
the patch passed

+1 :green_heart:
cc
5m  6s
the patch passed

+1 :green_heart:
javac
5m  6s
the patch passed

+1 :green_heart:
checkstyle
1m 27s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
18m 10s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
hbaseprotoc
1m 58s
the patch passed

+1 :green_heart:
spotbugs
6m 48s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 39s
The patch does not generate ASF License warnings.

64m  4s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
dupname asflicense cc hbaseprotoc prototool javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux 8e48a1d188db 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / bffe8957c5

Default Java
AdoptOpenJDK-1.8.0_282-b08

Max. process+thread count
96 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  6s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 22s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 50s
master passed

+1 :green_heart:
compile
2m  7s
master passed

+1 :green_heart:
shadedjars
8m 13s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  7s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 42s
the patch passed

+1 :green_heart:
compile
2m  4s
the patch passed

+1 :green_heart:
javac
2m  4s
the patch passed

+1 :green_heart:
shadedjars
8m 15s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  6s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 46s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 40s
hbase-replication in the patch passed.

-1 :x:
unit
140m  4s
hbase-server in the patch failed.

176m 17s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux eb151e017665 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / bffe8957c5

Default Java
AdoptOpenJDK-1.8.0_282-b08

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/testReport/

Max. process+thread count
3725 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  8s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
5m 11s
master passed

+1 :green_heart:
compile
2m 46s
master passed

+1 :green_heart:
shadedjars
9m  7s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 11s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 43s
the patch passed

+1 :green_heart:
compile
2m 37s
the patch passed

+1 :green_heart:
javac
2m 37s
the patch passed

+1 :green_heart:
shadedjars
9m 10s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  9s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
1m  3s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 42s
hbase-replication in the patch passed.

-1 :x:
unit
187m  3s
hbase-server in the patch failed.

228m 40s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux 203c132ccf3d 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / bffe8957c5

Default Java
AdoptOpenJDK-11.0.10+9

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/testReport/

Max. process+thread count
2853 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/2/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
8m 57s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
prototool
0m  1s
prototool was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 23s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 54s
master passed

+1 :green_heart:
compile
5m 50s
master passed

+1 :green_heart:
checkstyle
1m 45s
master passed

+1 :green_heart:
spotbugs
7m 17s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 10s
the patch passed

+1 :green_heart:
compile
5m 25s
the patch passed

+1 :green_heart:
cc
5m 25s
the patch passed

+1 :green_heart:
javac
5m 25s
the patch passed

+1 :green_heart:
checkstyle
1m 35s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
20m 16s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
hbaseprotoc
2m  0s
the patch passed

+1 :green_heart:
spotbugs
7m  8s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 32s
The patch does not generate ASF License warnings.

79m 31s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
dupname asflicense cc hbaseprotoc prototool javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux 67ea78811a38 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 22ec681ad9

Default Java
AdoptOpenJDK-1.8.0_282-b08

Max. process+thread count
86 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
3m 48s
Docker mode activated.

-0 :warning:
yetus
0m  4s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 23s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m  3s
master passed

+1 :green_heart:
compile
2m 13s
master passed

+1 :green_heart:
shadedjars
8m 51s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  8s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 37s
the patch passed

+1 :green_heart:
compile
2m  1s
the patch passed

+1 :green_heart:
javac
2m  1s
the patch passed

+1 :green_heart:
shadedjars
8m 52s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  5s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 48s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 25s
hbase-replication in the patch passed.

+1 :green_heart:
unit
135m  0s
hbase-server in the patch passed.

174m 56s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux 3f2bacde6252 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 22ec681ad9

Default Java
AdoptOpenJDK-1.8.0_282-b08

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/testReport/

Max. process+thread count
3491 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 25s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 29s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 48s
master passed

+1 :green_heart:
compile
2m 45s
master passed

+1 :green_heart:
shadedjars
9m 11s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 11s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 51s
the patch passed

+1 :green_heart:
compile
2m 42s
the patch passed

+1 :green_heart:
javac
2m 42s
the patch passed

+1 :green_heart:
shadedjars
9m 16s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 11s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
1m  5s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 41s
hbase-replication in the patch passed.

+1 :green_heart:
unit
183m  4s
hbase-server in the patch passed.

225m  1s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux 1f47d477b1fc 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 22ec681ad9

Default Java
AdoptOpenJDK-11.0.10+9

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/testReport/

Max. process+thread count
2683 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/3/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 28s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  1s
No case conflicting files found.

+0 :ok:
prototool
0m  1s
prototool was not available.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 23s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 46s
master passed

+1 :green_heart:
compile
4m 59s
master passed

+1 :green_heart:
checkstyle
1m 31s
master passed

+1 :green_heart:
spotbugs
6m 28s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 40s
the patch passed

+1 :green_heart:
compile
4m 59s
the patch passed

+1 :green_heart:
cc
4m 59s
the patch passed

+1 :green_heart:
javac
4m 59s
the patch passed

+1 :green_heart:
checkstyle
1m 28s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
18m 17s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.

+1 :green_heart:
hbaseprotoc
2m  0s
the patch passed

+1 :green_heart:
spotbugs
7m  1s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 38s
The patch does not generate ASF License warnings.

64m 29s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
dupname asflicense cc hbaseprotoc prototool javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux 5df1c0b109b4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79659d8e66

Default Java
AdoptOpenJDK-1.8.0_282-b08

Max. process+thread count
96 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  5s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 30s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 48s
master passed

+1 :green_heart:
compile
2m 44s
master passed

+1 :green_heart:
shadedjars
9m  1s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 10s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m 44s
the patch passed

+1 :green_heart:
compile
2m 41s
the patch passed

+1 :green_heart:
javac
2m 41s
the patch passed

+1 :green_heart:
shadedjars
9m  3s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 10s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
1m  4s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 40s
hbase-replication in the patch passed.

+1 :green_heart:
unit
183m 59s
hbase-server in the patch passed.

225m 10s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux eed67d3ddb81 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79659d8e66

Default Java
AdoptOpenJDK-11.0.10+9

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/testReport/

Max. process+thread count
2449 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  6s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 15s
master passed

+1 :green_heart:
compile
2m 10s
master passed

+1 :green_heart:
shadedjars
8m 58s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  5s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
4m  4s
the patch passed

+1 :green_heart:
compile
2m  8s
the patch passed

+1 :green_heart:
javac
2m  8s
the patch passed

+1 :green_heart:
shadedjars
9m  1s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  3s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 48s
hbase-protocol-shaded in the patch passed.

+1 :green_heart:
unit
0m 39s
hbase-replication in the patch passed.

+1 :green_heart:
unit
207m 10s
hbase-server in the patch passed.

245m 20s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3430

Optional Tests
unit javac javadoc shadedjars compile

uname
Linux 5ccf9e2af5bf 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79659d8e66

Default Java
AdoptOpenJDK-1.8.0_282-b08

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/testReport/

Max. process+thread count
2327 (vs. ulimit of 30000)

modules
C: hbase-protocol-shaded hbase-replication hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3430/4/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.