text:
HLLSketchMerge aggregator failing for some metrics after upgrade to v0.18

Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").
Affected Version
v0.18 (upgraded from 0.16.0)
Description
The HLLSketchMerge aggregator is failing for some of our metrics after upgrading to druid 0.18.0.  Reverting back to 0.16.0 fixes the issue. I have isolated specific segments where the issue occurs, moved those segments back to our 0.16 historical and have been successfully able to query the same metric.
Re-indexing data does not seem to fix the issue.
Error Message.
{
  "error": "Unknown exception",
  "errorMessage": "java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.datasketches.SketchesArgumentException: Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.",
  "errorClass": "java.lang.RuntimeException",
  "host": null
}

Hi @scrawfor, would you be able to upload one of the specific segments that has this problem?
It sounds like it might be a backwards compatibility issue in DataSketches, but having a copy of the actual sketch image would help debug.
/cc @AlexanderSaydakov for fyi

By the way, Druid 0.16 used DataSketches 0.13.4, and Druid 0.18 uses DataSketches 1.2.0-incubating.

Unfortunately I can't upload it.  I can try to recreate the issue with generic data, but I'm not sure how productive that would be. Are there any other debugging steps I could take?
I tried to load a 0.17.1 historical server to see if it occurred in that release as well, but I ran into issues with the authentication extension and was unable to get the node started.

@scrawfor Perhaps you could extract and upload just the HLL column? It only contains sketches of data, so it's less sensitive than the entire segment.
A good way to do it is to unzip the segment and look at the meta.smoosh file. It has one line per column, where each line has four parts: column name, smoosh file index (usually "0" except for large multipart segments), start byte offset within that smooth file, end byte offset. So for this column:
diffUrl,0,3316799,5665555

You could extract it by running:
dd bs=1 if=00000.smoosh skip=3316799 count=2348756 of=diffUrl

What I'm after is a binary image of a specific sketch that exhibits the problem — I think once we have that it should be easier to find and fix it. If it is possible to reproduce this on some test data that would be great too.

Do we have a stack trace that shows where in the sketch code this occurs?

It would also be helpful to know in what kind of operation: update or merge, getResult(), getEstimate(), etc.

The exception thrown is "java.util.concurrent.ExecutionException".   Our sketches are not thread-safe with the single exception of our Theta Sketch, which has concurrent configuration options.  So if there is more than one thread touching the HLL sketch, all bets are off.
I can't explain why it doesn't fail in Druid 0.16, unless your threading model changed between 0.16 and 0.18.
If concurrency is the problem, the simplest & fastest way to fix this would be to put a synchronized wrapper around the sketch.

@gianm I'll extract it tomorrow. Thanks for the instructions.
@leerho It's a merge aggregator.
{
    "type" : "HLLSketchMerge",
    "name" : "unique_views_hll",
    "fieldName" : "unique_views_hll"
}

Here is a stack strace
2020-04-21T16:42:03,398 ERROR [processing-3] org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2 - Exception with one of the sequences!
org.apache.datasketches.SketchesArgumentException: Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.
	at org.apache.datasketches.hll.Union.<init>(Union.java:100) ~[datasketches-java-1.2.0-incubating.jar:?]
	at org.apache.datasketches.hll.Union.writableWrap(Union.java:140) ~[datasketches-java-1.2.0-incubating.jar:?]
	at org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeBufferAggregator.aggregate(HllSketchMergeBufferAggregator.java:113) ~[?:?]
	at org.apache.druid.query.aggregation.AggregatorAdapters.aggregateBuffered(AggregatorAdapters.java:164) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.aggregate(AbstractBufferHashGrouper.java:161) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.aggregate(SpillingGrouper.java:168) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.aggregate(ConcurrentGrouper.java:267) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.Grouper.aggregate(Grouper.java:85) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.lambda$createGrouperAccumulatorPair$2(RowBasedGrouperHelper.java:330) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:87) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:153) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:246) [druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:233) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-04-21T16:42:03,425 WARN [qtp1446521801-86[groupBy_[analytics-data-primary]_be057876-fdd5-4085-be7f-9c17456e11ca]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [be057876-fdd5-4085-be7f-9c17456e11ca] (java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.datasketches.SketchesArgumentException: Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.)
2020-04-21T16:42:03,400 ERROR [processing-4] org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2 - Exception with one of the sequences!
org.apache.datasketches.SketchesArgumentException: Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.
	at org.apache.datasketches.hll.Union.<init>(Union.java:100) ~[datasketches-java-1.2.0-incubating.jar:?]
	at org.apache.datasketches.hll.Union.writableWrap(Union.java:140) ~[datasketches-java-1.2.0-incubating.jar:?]
	at org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeBufferAggregator.aggregate(HllSketchMergeBufferAggregator.java:113) ~[?:?]
	at org.apache.druid.query.aggregation.AggregatorAdapters.aggregateBuffered(AggregatorAdapters.java:164) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.aggregate(AbstractBufferHashGrouper.java:161) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.aggregate(SpillingGrouper.java:168) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.aggregate(ConcurrentGrouper.java:267) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.Grouper.aggregate(Grouper.java:85) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.lambda$createGrouperAccumulatorPair$2(RowBasedGrouperHelper.java:330) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:87) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:153) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:246) [druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:233) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-04-21T16:42:03,447 ERROR [processing-4] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.RuntimeException: org.apache.datasketches.SketchesArgumentException: Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:253) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:233) ~[druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.apache.datasketches.SketchesArgumentException: Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.
	at org.apache.datasketches.hll.Union.<init>(Union.java:100) ~[datasketches-java-1.2.0-incubating.jar:?]
	at org.apache.datasketches.hll.Union.writableWrap(Union.java:140) ~[datasketches-java-1.2.0-incubating.jar:?]
	at org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeBufferAggregator.aggregate(HllSketchMergeBufferAggregator.java:113) ~[?:?]
	at org.apache.druid.query.aggregation.AggregatorAdapters.aggregateBuffered(AggregatorAdapters.java:164) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.aggregate(AbstractBufferHashGrouper.java:161) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.aggregate(SpillingGrouper.java:168) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.aggregate(ConcurrentGrouper.java:267) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.Grouper.aggregate(Grouper.java:85) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper.lambda$createGrouperAccumulatorPair$2(RowBasedGrouperHelper.java:330) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:87) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:153) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:246) ~[druid-processing-0.18.0.jar:0.18.0]
	... 6 more

Thanks for the stack trace.  It is very helpful. I think I may have a clue what may be happening.
The specific exception thrown by the sketch is:

org.apache.datasketches.SketchesArgumentException: Incoming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.

This occurs in only one place, which is the union.update(HllSketch input) method.  It is detecting that a special flag is set in the input sketch that is only ever set during a union operation.   Thus the incoming sketch had to be an image of a union operator and not that of a streaming updatable sketch nor a streaming compact sketch.
This flag is used to detect that the internal state of the union data structure is not finalized.  It is only finalized when you call

HllSketch out = union.getResult(TgtHllType type);

Using a serialized union operator as input to a union merge operation is not legal and it never has been.   The example code also illustrates the use of union.getResult(type), and that example code has been there for several years.
It is only with version 1.2.0 that this special case is properly detected.
There still might be a problem in the HLL code, but I would appreciate it if you could check your usage code and see if my hunch is correct.
Lee.

@gianm I pulled out all sensitive info and re-indexed our data, so I'm attaching the full segment. The offending metric is unique_views_hll.
hll_segment.zip
Also, I did some tests with other query types and found timeseries and topN queries were successful.
@leerho I'll have to leave that to others more familiar with the druid extension to comment on, but looking at the code it does seem like a lock is acquired.

@scrawfor Thanks for the upload. Do you have an example of a query that exhibits the problem?

@scrawfor Thanks for the upload. Do you have an example of a query that exhibits the problem?

Sure.
{
  "dataSource" : "hlltest",
  "queryType" : "groupBy",
  "intervals" : [ "2020-04-06/2020-04-07" ],
  "granularity" : "ALL",
  "aggregations" : [{
    "type" : "HLLSketchMerge",
    "name" : "unique_views_hll",
    "fieldName" : "unique_views_hll"
  }],
  "limitSpec" : {
    "type" : "default",
    "limit" : 50000,
    "columns" : [ {
      "dimension" : "unique_views_hll",
      "direction" : "descending",
      "dimensionOrder" : "alphanumeric"
    } ]
  },
  "dimensions" : []
}

Thanks for the segment and example query, I can reproduce this issue in the debugger with group by query and can confirm that it doesn't seem to affect timeseries queries, which means it is likely an issue with HllSketchMergeBufferAggregator or how it is being used, (timeseries is using HllSketchMergeAggregator).

The "concurrent" exception that was thrown makes me nervous :)

This is an unfortunate presentation issue, that's not the actual exception on the historical, but the side effect of the broker parallel merging catching an error on an individual query. I'll see if I can try to improve this in the future to make it less confusing.

@clintropolis Thanks for confirming.  I did find that the HllSketchMergeAggregator seemed to be affected as well.
I tried to reissue the query this evening, but it's using HllSketchMergeBufferAggregator instead.
2020-04-22T14:38:27,220 ERROR [qtp1446521801-81[groupBy_[analytics-data-daily-dev]_b5a02c60-3786-499a-a275-8e8bfae81e0e]] org.apache.druid.server.QueryResource - Exception handling request: {class=org.apache.druid.server.QueryResource, exceptionType=class org.apache.datasketches.SketchesArgumentException, exceptionMessage=Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set., query={"queryType":"groupBy","dataSource":{"type":"table","name":"analytics-data-daily-dev"},"intervals":{"type":"LegacySegmentSpec","intervals":["2020-04-01T00:00:00.000Z/2020-04-14T00:00:00.000Z"]},"virtualColumns":[],"filter":null,"granularity":{"type":"all"},"dimensions":[],"aggregations":[{"type":"HLLSketchMerge","name":"unique_views_hll","fieldName":"unique_views_hll","lgK":12,"tgtHllType":"HLL_4","round":false}],"postAggregations":[{"type":"HLLSketchToString","name":"unique_views_hll_sketch","field":{"type":"fieldAccess","name":"unique_views_hll","fieldName":"unique_views_hll"}}],"having":null,"limitSpec":{"type":"default","columns":[{"dimension":"unique_views_hll","direction":"descending","dimensionOrder":{"type":"alphanumeric"}}],"limit":50000},"context":{"groupByStrategy":"v1","queryId":"b5a02c60-3786-499a-a275-8e8bfae81e0e"},"descending":false}, peer=10.89.92.231} (org.apache.datasketches.SketchesArgumentException: Incomming sketch is corrupted, Rebuild_CurMin_Num_KxQ flag is set.)
2020-04-22T14:38:27,221 ERROR [processing-4] org.apache.druid.query.GroupByMergedQueryRunner - Exception with one of the sequences!
java.lang.NullPointerException: null
	at org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeAggregator.aggregate(HllSketchMergeAggregator.java:63) ~[?:?]
	at org.apache.druid.segment.incremental.OnheapIncrementalIndex.doAggregate(OnheapIncrementalIndex.java:252) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.segment.incremental.OnheapIncrementalIndex.addToFacts(OnheapIncrementalIndex.java:162) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:614) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:608) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.GroupByQueryHelper$3.accumulate(GroupByQueryHelper.java:155) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.GroupByQueryHelper$3.accumulate(GroupByQueryHelper.java:139) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:87) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:153) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.GroupByMergedQueryRunner$1$1.call(GroupByMergedQueryRunner.java:121) [druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.GroupByMergedQueryRunner$1$1.call(GroupByMergedQueryRunner.java:111) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2020-04-22T14:38:27,221 ERROR [processing-3] org.apache.druid.query.GroupByMergedQueryRunner - Exception with one of the sequences!
java.lang.NullPointerException: null
	at org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeAggregatorFactory.factorize(HllSketchMergeAggregatorFactory.java:89) ~[?:?]
	at org.apache.druid.segment.incremental.OnheapIncrementalIndex.factorizeAggs(OnheapIncrementalIndex.java:233) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.segment.incremental.OnheapIncrementalIndex.addToFacts(OnheapIncrementalIndex.java:165) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:614) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:608) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.GroupByQueryHelper$3.accumulate(GroupByQueryHelper.java:155) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.groupby.GroupByQueryHelper$3.accumulate(GroupByQueryHelper.java:139) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:41) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:87) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:153) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.18.0.jar:0.18.0]
	at org.apache.druid.query.GroupByMergedQueryRunner$1$1.call(GroupByMergedQueryRunner.java:121) [druid-processing-0.18.0.jar:0.18.0]
	at org.apache.druid.query.GroupByMergedQueryRunner$1$1.call(GroupByMergedQueryRunner.java:111) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247) [druid-processing-0.18.0.jar:0.18.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

How was that segment with sketches created?

@AlexanderSaydakov Using native batch indexer, local Firehose, built the sketch over a string column.
Here's the metric spec.
 {
          "type": "HLLSketchBuild",
          "name": "unique_views_hll",
          "fieldName": "view_id",
          "lgK": 14,
          "tgtHllType": "HLL_4",
          "round": false
 },
``

using what version of Druid?

0.16.0

The exception occurs on this line on i think the 7th call to aggregate using the example query, which is wrapping the memory location it gets from the union it is building for the aggregation, though I guess the previous aggregate operation could have left it in this state? I had some other stuff to do so haven't got to dig in much yet to determine if the issue is with the value stored in the column (pointing to the build aggregator being the issue) or it occurs from the aggregation itself at query time.

We have a fix!
@clintropolis @scrawfor @gianm  @AlexanderSaydakov
I want to thank all of you for your help!  This was truly a team effort!
With clues from @clintropolis and @scrawfor @AlexanderSaydakov was able to reproduce the bug with his knowledge of how the aggregator works.  And from that I was able to locate the bug, which was my fault.  I put in a check for a flag where there did not need to be one.  So it was actually throwing an unnecessary exception.
We will be going over this part of the code carefully, adding unit tests and preparing for a new release.  Due to the dual 72 hour release cycles this will take a week or so.
Thank you for your patience!
Lee.

Is there a schema file somewhere that describes the layout of the hll_segment.zip file that @scrawfor attached to this issue.  I'd like to write a parser so that in the future if we need to pull out just the sketches I can do it more efficiently.  I was able to find some of the sketches by hand, but it is a lot of work :)

Is there a schema file somewhere that describes the layout of the hll_segment.zip file that @scrawfor attached to this issue. I'd like to write a parser so that in the future if we need to pull out just the sketches I can do it more efficiently. I was able to find some of the sketches by hand, but it is a lot of work :)

@gianm described how to get the raw column out of the segment in this comment by finding the position information in the meta.smoosh, but you can also extract base64 serialized versions of the column with the dump-segment tool, if you're wanting to easily look at values for individual rows.
From the druid package directory:
$ java -classpath "lib/*" -Ddruid.extensions.loadList="[\"druid-datasketches\"]" org.apache.druid.cli.Main   tools dump-segment   -d /Users/clint/workspace/data/druid/localStorage/hll_segment/   --dump rows   --column unique_views_hll

which will spit out something like this:
{"unique_views_hll":"AwEHDggYAAGwAAAAAK6iCQFzJwQDzXwGBTMWCgeFZARCyewJDVSOBw8Q6QQRuVwO2jliD4GTxAnxybMEFnw+CRj2xw4Z9/MEGovwBxyq+godenYKHl8CBx9N7AaPkw0KIzRKEQ8VoATGX+QGJnkMBydvIAXWU8YNKqXbBCuKUgYu5Y0LMOZHBzLeBQczolUHJ66KBjVFJA427xcFGB5sB+tuxQs5gRAbYG1uFD13/RA+TOQF+EZ8FIe4PxBCoDwEKhhEF27WWgc9CJYGSFPOD0ld1A3NfH8JS/3JB0ynuAu6ugsETuvmCk/ByAutozQHUVk8FkiEUQdUcOoIVRcHCFZKLgRXXloFNSM8DqxXSwZbS8IEXNjhDF3WcxJeRQYIYGuxB2EK6wdjnhESZNyBBGeCsghoY0UHSCCMBWwN7gRuu5sKb9xvCXES+wpy6RALcx69BXSE1At1ZcgEvF0GBrPFjQx5L7YFeia2Bkwu+A2AfysHgDD6C4GBoQSCmcQLRZ43BO+VRROFOvMHh6nKBrPqiQWLPqQHjcHlBo5JHQSPwzgErnF6DpF0vAWSLWcLaQ4UC5ckBAyYzLcHm+LIBZ3OQAuBHaIFoEwADKGVjASibboM7VpAB6SJAAWlDrwHptSdCaf2UxAdjHAFgSqBG6YGjgiul1EIM19xBLAZqgqyZUEMs2H7EbTFIRC2l1YOuIUZJLlyjw26y6cINoV1CbxJvQi9qn0HvpsMB78T3Qa8Bu4MxgD9Bch6GwjvbC8Hy/MyB5E7aAXN0FEFzmVGCc8vIwfSby4G1ENTD4JTYwTW2XcH18g/BNhTUBr/bVUF2tK0BtxzuAbd9OYR3/THBuCZsQrirXUHrpsqBeUdggnmZX8E5zomDurPIQXr6l0H7Vs3Be5otAY2ua4E8aijCvJQzgf0gtAW+CxWBPrluhH7eqkKrsTVBHKM9wQ="}
{"unique_views_hll":"AgEHDgMIBQDERQoE+BSGB6dz6ATE/HoHZs5tIA=="}
{"unique_views_hll":"AgEHDgMIBQCPrFwWNt2mCkmuNQZFWZgYxJpXDg=="}
....

I ran some tests, and it appears that downgrading datasketches-java to 1.1.0-incubating doesn't have this issue. Since we have an unrelated major regression with stream ingestion and are doing a 0.18.1 release asap, as a precaution I have opened #9751 which makes this change, in case our critical fix release is ready to go before your new version is released. @leerho / @AlexanderSaydakov can you think of any reason not to do this?

I don't see why not.

I tried using the Dump-Segment tool, following the java run command in the documentation, to examine the hll_segment.zip provided by @scrawfor above.
I get a NullPointerException.  I would like to get this working, but I must be doing something wrong. Here is the stack trace:
`C02XG1Q5JGH8:Druid lrhodes$ java -classpath "/Users/lrhodes/dev/git/Apache/druid-0.18.0/apache-druid-0.18.0/lib/*" -Ddruid.extensions.loadList="[]" org.apache.druid.cli.Main \

tools dump-segment
--directory /Users/lrhodes/dev/git/Apache/druid-0.18.0/Dump/hll_segment/
--out /Users/lrhodes/dev/git/Apache/druid-0.18.0/Dump/hll_segment.txt
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.druid.cli.DumpSegment.run(DumpSegment.java:196)
at org.apache.druid.cli.Main.main(Main.java:113)
Caused by: java.lang.NullPointerException
at org.apache.druid.segment.column.SimpleColumnHolder.getColumn(SimpleColumnHolder.java:68)
at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.lambda$getCachedColumn$2(QueryableIndexColumnSelectorFactory.java:175)
at java.util.HashMap.computeIfAbsent(HashMap.java:1127)
at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.getCachedColumn(QueryableIndexColumnSelectorFactory.java:171)
at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.lambda$makeColumnValueSelector$1(QueryableIndexColumnSelectorFactory.java:146)
at org.apache.druid.segment.QueryableIndexColumnSelectorFactory.makeColumnValueSelector(QueryableIndexColumnSelectorFactory.java:160)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at org.apache.druid.cli.DumpSegment$2$1.apply(DumpSegment.java:280)
at org.apache.druid.cli.DumpSegment$2$1.apply(DumpSegment.java:272)
at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41)
at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45)
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
at org.apache.druid.cli.DumpSegment.evaluateSequenceForSideEffects(DumpSegment.java:493)
at org.apache.druid.cli.DumpSegment.access$100(DumpSegment.java:103)
at org.apache.druid.cli.DumpSegment$2.apply(DumpSegment.java:312)
at org.apache.druid.cli.DumpSegment$2.apply(DumpSegment.java:265)
at org.apache.druid.cli.DumpSegment.withOutputStream(DumpSegment.java:426)
at org.apache.druid.cli.DumpSegment.runDump(DumpSegment.java:263)
at org.apache.druid.cli.DumpSegment.run(DumpSegment.java:183)
...`

@clintropolis @gianm
Thanks for pointing me to the dump segment tool.  I got it to work :)

apache/incubator-datasketches-java#308 - Looks like this is the PR with the fix.

Yes.  But I would wait for the full release.
See
https://lists.apache.org/thread.html/r6a69c6689ec303fc2df83ea483b87166b0b1d14422a2803b881b87ef%40<dev.datasketches.apache.org>
On Wed, Apr 29, 2020 at 9:15 AM Suneet Saldanha notifications@github.com
wrote:

apache/incubator-datasketches-java#308
https://github.com/apache/incubator-datasketches-java/pull/308 - Looks
like this is the PR with the fix.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/druid/issues/9736#issuecomment-621313956, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ADCXRQS432M7JNHEVF6CJVTRPBHCRANCNFSM4MNQLJ5Q
.

Folks,
One of the learnings from this debugging exercise is that it would have been really useful to be able to quickly examine the sketches in the hll_segment.zip that @scrawfor posted in this issue.
As a result, I have developed a small tool that takes the output of the dump_segment_tool , and extracts the sketches as binary files.  This allows us to easily examine the details of individual sketches with methods already available in the DataSketches library.
Hopefully, this will make debugging issues involving sketches in Druid much easier and faster.
The question is where should we put this tool so others can use it?  Obviously it makes assumptions about Druid's segment structure and Druid's Dump-Segment tool.  It doesn't make sense to put it in the DataSketches library as it is specific to Druid.   I'd be glad to submit a PR and add it to druid/services/src/main/java/org/apache/druid/cli  directory.  Or perhaps it should be added to the druid/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches folder.
Please advise.
Lee.

@leerho, would the work you did make sense as an additional option on the DumpSegment tool? If so that seems like the most natural place.

@gianm
Could we set up a separate issue for this discussion?  It is a bit off topic for this bug.

Yes, @leerho that makes sense. Please start a new issue and at-mention me on it via @gianm.

We announced a new release on May 7th that fixes this issue on dev@druid.apache.org.

authors: AlexanderSaydakov clintropolis gianm leerho scrawfor suneet-s


text:
Introduce extensibility support for Datasources

Description
Introduce an extension point for the DataSource interface that will allow possibilities to create custom datasource implementations. This extension point will be applicable only for datasource types that handle multiple table data sources. Within core druid, UnionDataSource is one such type.
The extension point would look something similar to this:
public interface MultiDataSource extends DataSource
{
 List<TimelineObjectHolder<String, ServerSelector>> getSegmentsFromIntervals(
     Query baseQuery,
     Map<String, ? extends TimelineLookup<String, T>> timelineMap
 );
}

The implementations of this interface will have the ability to decide which segments would be selected from which base table datasources from the given set of intervals. For every query, the broker needs to decide the list of segments that needs to be queried against the various queryable nodes and If the incoming query is a type of MultiDataSource, the broker can defer the segment selection to its getSegmentsFromIntervals method.
The UnionDataSource can be refactored to be an implementation of this interface and the getSegmentsFromIntervals method would return all the segments from all the datasources defined in the union query. To clarify, all the existing DataSource implementations except UnionDataSource would be unaffected.
For supporting this, the segment selection and related query processing logic would start handling multiple TableDataSources .TimelineServerView.getTimeline  would be refactored to return a map of Datasource -> Timelinelookup , something like:
Optional<? extends Map<String, ? extends TimelineLookup<String, ServerSelector>>>
 getTimeline(DataSourceAnalysis analysis);

Motivation

This feature can help users with validation workflows where changes could be tested out on a non-production datasource before applying the changes onto the production datasource. Detailed example of the usecase is mentioned here: #5350

Union queries are currently executed serially and for a list of datasources, the broker executes the same query for each datasource in a serial fashion. If the Union data source implementation is refactored to implement MultiDataSource, the broker can identify segments to be retrieved from all the requested datasources at the same time therefore the queries to the historical will now include segment specs from all the datasources requested in the union query. This can eliminate the performance bottleneck associated with the serial execution.

@gianm @jihoonson Any comments/concerns regarding this proposal?

Is the idea that MultiDataSource would be something that exists and runs on the Broker, or on Historicals, or both?
This seems sort of similar to the SegmentWrangler interface, which is used by the Broker in LocalQuerySegmentWalker to do direct queries over inline and lookup datasources. I wonder if these can be combined? What do you think?
If not, then I wonder if this thing you're trying to do could be done in a style similar to SegmentWrangler? (Check out its SegmentWranglerModule, etc.)

@gianm  Thanks for taking a look.
Both the Broker and Historical would need to be aware of MultiDataSource, as Brokers would need this info to choose the custom segment selection strategy and Historicals would need to identify interval-specific timeline entries from multiple timelines. I’m not sure if I can use the SegmentWrangler in its current form since MultiDataSource is directed towards table datasources and table datasources would have to use CachingClusteredClient.
A SegmentWrangler implementation could be created with a custom QuerySegmentWalker but the walker might end up being much similar to CachingClusteredClient, especially the getQueryRunnerForIntervals method.

Are there use cases you have in mind other than redesigning UnionDataSource?

Apart from the UnionDataSource redesign, one other useful MultiDataSource implementation would be something like a Priority list datasource that accepts an ordered list of tables.
For example if the tables are table1 and table2, the priority list datasource would attempt to serve every query out of table1 and would use table2 only for intervals with missing segments on table1.
One of our usecases internally for priority list datasource is a validation mechanism for data correction on production tables. It isn’t ideal to test out the data fixes directly by reindexing the prod table,  so it's better to have an experimental table where the validations could be performed before applying it to the prod table.
A Priority list based datasource implementation could be used for this usecase with the table list having the experimental table first followed by the prod table.
Another potential usecase for priority list datasource could be used to represent updates to existing data when combined with realtime ingestion.

The UnionDataSource can be refactored to be an implementation of this interface and the getSegmentsFromIntervals method would return all the segments from all the datasources defined in the union query.

How does this work? I guess the broker needs to keep track of which segments are for which timeline and which sub part of the union query? Does it potentially issue multiple queries to a single server instead of the current single query per historical/peon with specific segment spec that it does now? If not, how does the broker distinguish which results belong to which timeline? I'm having trouble imagining how this works in practice without making caching clustered client know a lot more about the query than it currently does, but maybe I haven't though hard enough about this yet.

Ah, sorry, long day, and I think I confused myself by not thinking too hard about this, or alternatively by thing way too hard about this; I guess this wouldn't really cause any issue for union datasource since it issues a homogenous query across table datasources? I'm going to take a break and then i'll read through this again 😅

@clintropolis Thanks for reviewing this.
In order for the broker to support this, CachingClusteredClient will have to start handling a map of Datasource -> Timeline instead of just a Timeline object. Apart from this, the only other significant change would be something like this in the computeSegmentsToQuery method:
private Set<SegmentServerSelector> computeSegmentsToQuery(Map<String,TimelineLookup<String, ServerSelector>> timeline)
 {
   final List<TimelineObjectHolder<String, ServerSelector>> serversLookup;
   if (query.getDataSource() instanceof MultiDataSource) {
     List<TimelineObjectHolder<String, ServerSelector>> multiServersLookup = ((MultiDataSource) query.getDataSource()).getSegmentsFromIntervals(intervals, timeline);
     serversLookup = toolChest.filterSegments(query, multiServersLookup);
   } else {
// I wouldnt be using toString() as below in the actual PR. This is just to explain my thoughts :)

     serversLookup = toolChest.filterSegments(
         query,
         intervals.stream().flatMap(i -> timeline.get(query.getDataSource().toString()).lookup(i).stream()).collect(Collectors.toList())
     );
   }

The computeUncoveredIntervals method would be modified accordingly as well.
As you said, it is pretty straightforward for Union queries as it only has to deal with table datasources. Its implementation would be something like:
getSegments(
      List<Interval> intervals,
      Map<String, ? extends TimelineLookup<String, T>> timelineMap
  )
  {
    return intervals.stream()
                    .flatMap(i -> getDataSources().stream().filter(d ->timelineMap.containsKey(d.getName()))
                                                  .flatMap(n -> timelineMap.get(n.getName()).lookup(i).stream()))
                    .collect(Collectors.toList());

Based on ^, I guess not, but If not, how does the broker distinguish which results belong to which timeline? Or does it matter? I'm having trouble imagining how this works in practice without making caching clustered client know a lot more about the query than it currently does, but maybe I haven't though hard enough about this yet.

For MultiDatasource implementations, the historicals would still return intervals based sequences as it does now, but the only difference would be the sequences would have results from multiple datasources for the same interval. Do you foresee any issue with this impacting the broker merge process for Union queries in some way?

Apart from the UnionDataSource redesign, one other useful MultiDataSource implementation would be something like a Priority list datasource that accepts an ordered list of tables.
For example if the tables are table1 and table2, the priority list datasource would attempt to serve every query out of table1 and would use table2 only for intervals with missing segments on table1.

Could you explain more details? It would help others understand the design choice of MultiDataSource. It seems like this datasource will involve multiple roundtrips in a single query. How do the broker and historicals interact each other? Also, how do you tell some segments are missing in table1?

@jihoonson Thanks for taking a look.
MultiDataSource is designed as such to provide its implementations a capability to decide the datasources from which segments need to be queried for an interval. This applies only to queries with multiple base tables in the dataSource part of the query. (Only UnionDataSource at this point).  By missing segments, I mean the case when there are no segments in table1 for a specific interval. This is not to be confused with missing segments which are handled by the RetryQueryRunner.
Expanding further on this example, lets say I have a union-based query against table1 and table2, but I need data from table2 only if there are no segments in table1 for the interval. Currently this isn't possible using UnionDataSource . With MultiDataSource, I can create an implementation that satisfies this usecase.
In order to support this, Broker will now identify segments to be queried for all the table datasources from the query in a single pass instead of one table datasource at a time. This makes it easier to push the segment selection logic to the MultiDataSource implementation for such queries. This diff would give a clearer picture: https://github.com/apache/druid/pull/10030/files#diff-14e0f52ca2d35d282c1e92d1c14eb0d1R374-R395
Regarding broker-historical interaction, the queries from the broker to the historical will now include SegmentDescriptors from all the datasources requested in the union-based query.  Thus this would only require one roundtrip between broker and historical (as long as there are no missing segments reported by ReportTimelineMissingSegmentQueryRunner).

@a2l007 thanks for the detailed explanation with an example. It sounds reasonable to me.

source: gharchive/issue

authors: a2l007 clintropolis gianm jihoonson

repo: apache/druid

license_type: permissive

license_source: github-api


text:
"Partition boost" the group by queries in MSQ for better splits

Description
In MSQ, each partition corresponds to a single segment (if it is the end stage) or the unit of work that is assigned to a worker. If the clustering key has low cardinality, then the returned partitions can be too large. For certain stages, we can break up the resulting partitions by introducing a synthetic clustering column "__boost" (which should contain a unique value - incrementally increasing longs for MSQ's implementation) at the end of the clustering keys, so that the keys become unique, and the partition boundaries are respectable. This is done in ScanQueryKit, and can be done in the GroupByPostShuffleFrameProcessor.
This PR introduces this partition boosting to the group by queries as well.
For example, the queries of the form
INSERT INTO table
SELECT dim1, dim2, agg(metric1) as aggregated FROM EXTERN(...)
GROUP BY dim1, dim2
PARTITIONED BY YEAR
CLUSTERED BY aggregated

can run into large segments if the "aggregated" is a very low cardinality column, and the (dim1, dim2) is a high cardinality pair. The solution (before the patch) would be to add more dims into the CLUSTERED BY clause to make it more unique. Post this patch, that won't be required.
ControllerImpl already handles partition-boosted columns specially.
NOTE: Group by queries can fail during the cluster upgrade, if some of the workers are on an older version, while the others are on a newer version.
Release note

Key changed/added classes in this PR

MyFoo
OurBar
TheirBaz

This PR has:

[ ] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] a release note entry in the PR description.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[ ] been tested in a test Druid cluster.

We do not want the boosted column as part of the segment.

This is the job of the controller to remove the boosted column. Also, we have pre-existing MSQInsertTests, which assert on the expected row and the row signature of the segment. The boosted column doesn't show up there.

I have also removed the conditional in the GroupByFrameProcessor and its factory because depending on the signature passed by the query kit, it will choose to add or ignore the partition boosted column. If the signature has __boost column in it, it will pick the column from the selector factory. Else, it will ignore the virtual column altogether. There's no need for additional conditional in the processor or the factory.

Tested that there's a __boost column at the end of the post-processor factory. Also, I went into the debug mode into the GroupByPostShuffleFrameProcessor and the SegmentGeneratorFrameProcessor and the boost column is present in both places with an incrementing value.

Thanks for the review @cryptoe

source: gharchive/pull-request


text:
关于发布到crates的讨论

确认一下，crates.io上的dubbo的包是否跟本项目有关。
如果无关的话，建议dubbo-rust里面的dubbo子package改名为dubbo-core
see more about crates.io/rust
因为crates.io 上的包名遵循先到先得的方式，建议在crates.io上面尽快申请dubbo相关的包名。
https://doc.rust-lang.org/stable/cargo/reference/publishing.html

感谢提醒

crates.io 上的 dubbo 据我所知与本项目无关
包名按照 rust 习惯规则进行重新命名，例如 core -> dubbo-core 将会在下一次主线合并更改。

因为没有跟诸位讨论，
#16   中所以我提的这个mr没有涉及现在的dubbo package命名的改动。。

authors: ChengYueJia Ticsmtc

repo: apache/dubbo-rust


text:
Fix recover method of ZookeeperServiceDiscovery

What is the purpose of the change
改动说明：

AbstractServiceNameMapping 所需要的ApplicationModule通过构造方法注入，可以去除 implements ScopeModelAware
MappingCacheManager#update 方法和父类一致，无需重写，因此可以去掉。
AbstractRegistry#getCacheUrls 方法优先从notified中获取缓存的urls列表，因为直接从properties获取需要解析（split）。
ZookeeperRegistry#destroy 方法移除childListener监听器。
ZookeeperServiceDiscovery#recover 用于在ZK重连后重新注册监听，使用的serviceName应该是先前注册的watcher.getServiceName()，而不是当前消费者应用的serviceName。

Codecov Report

Merging #10614 (3ad7097) into 3.1 (9ab939e) will decrease coverage by 0.08%.
The diff coverage is 30.00%.

@@             Coverage Diff              @@
##                3.1   #10614      +/-   ##
============================================
- Coverage     64.80%   64.71%   -0.09%
+ Complexity      396      390       -6
============================================
  Files          1331     1334       +3
  Lines         56522    56689     +167
  Branches       8370     8385      +15
============================================
+ Hits          36628    36687      +59
- Misses        15951    16043      +92
- Partials       3943     3959      +16

Impacted Files
Coverage Δ

...che/dubbo/metadata/AbstractServiceNameMapping.java
83.47% <ø> (ø)

...org/apache/dubbo/metadata/MappingCacheManager.java
76.47% <ø> (+14.56%)
:arrow_up:

...ubbo/registry/client/AbstractServiceDiscovery.java
80.59% <0.00%> (-4.02%)
:arrow_down:

.../registry/zookeeper/ZookeeperServiceDiscovery.java
68.35% <0.00%> (-2.54%)
:arrow_down:

...keeper/ZookeeperServiceDiscoveryChangeWatcher.java
81.81% <0.00%> (-2.56%)
:arrow_down:

...he/dubbo/registry/zookeeper/ZookeeperRegistry.java
61.05% <17.64%> (-4.27%)
:arrow_down:

...pache/dubbo/registry/support/AbstractRegistry.java
67.10% <100.00%> (-3.23%)
:arrow_down:

...bo/registry/support/CacheableFailbackRegistry.java
78.82% <100.00%> (+1.17%)
:arrow_up:

...nt/metadata/ServiceInstanceHostPortCustomizer.java
65.78% <0.00%> (-21.06%)
:arrow_down:

...dubbo/remoting/exchange/support/DefaultFuture.java
80.17% <0.00%> (-11.21%)
:arrow_down:

... and 47 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: BurningCN codecov-commenter

repo: apache/dubbo


text:
[FLINK-14976][cassandra] Release semaphore on all Throwable's in send()

What is the purpose of the change
https://github.com/apache/flink/pull/8967 changed the Cassandra connector to catch Exception and release the semaphore correctly, however this is not general enough as e.g. Error can still lead to deadlock. Change code to catch all Throwable instead.
Brief change log

Release Semaphore correctly on all Throwable in send()

Verifying this change
This change added tests and can be verified as follows:

Added unit test that validates semaphore released if Error in send(); test fails before change.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

merged into master by 07b66b60dd8c72f5ecd98054fa322d768c774e74

authors: azagrebin mchro

repo: apache/flink