text:
HBASE-27800: Add support for default user quotas

If you're managing HBase for many users, then managing effective user quotas can be a difficult operational burden. At my day job we have thousands of web services, workers, and recurring jobs — each of which we consider a distinct user via request attributes and quota user overrides. A common reliability issue is that some single system runs awry, for whatever reason, and generates a pathologic amount of traffic at the HBase level; this can cause pain for any other systems dependent on the given HBase cluster.
The above situation could potentially be solved by maintaining our own automated user quota management, but even better would be the ability to specify default user quotas at the HBase level. So that's exactly what this PR introduces.
In this PR we add four new configuration options to the QuotaUtil class:
  /*
   * The below defaults, if configured, will be applied to otherwise unthrottled users. For example,
   * set `hbase.quota.default.user.machine.read.size` to `1048576` in your hbase-site.xml to ensure
   * that any given user may not query more than 1mb per second from any given machine, unless
   * explicitly permitted by a persisted quota. All of these defaults use TimeUnit.SECONDS and
   * QuotaScope.MACHINE.
   */
  public static final String QUOTA_DEFAULT_USER_MACHINE_READ_SIZE =
    "hbase.quota.default.user.machine.read.size";
  public static final String QUOTA_DEFAULT_USER_MACHINE_READ_NUM =
    "hbase.quota.default.user.machine.read.num";
  public static final String QUOTA_DEFAULT_USER_MACHINE_REQUEST_NUM =
    "hbase.quota.default.user.machine.req.num";
  public static final String QUOTA_DEFAULT_USER_MACHINE_WRITE_NUM =
    "hbase.quota.default.user.machine.write.num";

These options are exactly what they sound like — one could configure hbase.quota.default.user.machine.read.size and hbase.quota.default.user.machine.req.num to ensure that no users can exceed a given read size and request per-machine throughput (unless explicitly given larger allowance via a defined quota).
cc @bbeaudreault @hgromer @eab148 @bozzkar

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 42s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m  2s
master passed

+1 :green_heart:
compile
2m 30s
master passed

+1 :green_heart:
checkstyle
0m 38s
master passed

+1 :green_heart:
spotless
0m 47s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 33s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 44s
the patch passed

+1 :green_heart:
compile
2m 26s
the patch passed

+1 :green_heart:
javac
2m 26s
the patch passed

+1 :green_heart:
checkstyle
0m 36s
hbase-server: The patch generated 0 new + 2 unchanged - 1 fixed = 2 total (was 3)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
4m 44s
Patch does not cause any errors with Hadoop 3.3.6.

+1 :green_heart:
spotless
0m 42s
patch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 37s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 12s
The patch does not generate ASF License warnings.

28m 18s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5666

JIRA Issue
HBASE-27800

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile

uname
Linux 6fdfddf6256b 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / afbf2adba2

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
80 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m  9s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 11s
master passed

+1 :green_heart:
compile
0m 46s
master passed

+1 :green_heart:
shadedjars
5m 47s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 24s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 58s
the patch passed

+1 :green_heart:
compile
0m 45s
the patch passed

+1 :green_heart:
javac
0m 45s
the patch passed

+1 :green_heart:
shadedjars
5m 41s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 23s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
220m 24s
hbase-server in the patch passed.

246m 29s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5666

JIRA Issue
HBASE-27800

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 074c5b6f6f7f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / afbf2adba2

Default Java
Eclipse Adoptium-11.0.17+8

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/testReport/

Max. process+thread count
4667 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 11s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 41s
master passed

+1 :green_heart:
compile
0m 38s
master passed

+1 :green_heart:
shadedjars
5m 32s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 23s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 26s
the patch passed

+1 :green_heart:
compile
0m 38s
the patch passed

+1 :green_heart:
javac
0m 38s
the patch passed

+1 :green_heart:
shadedjars
5m 31s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 22s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
226m 44s
hbase-server in the patch passed.

251m 39s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5666

JIRA Issue
HBASE-27800

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 99e71ce97c8d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / afbf2adba2

Default Java
Temurin-1.8.0_352-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/testReport/

Max. process+thread count
4803 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 40s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 46s
master passed

+1 :green_heart:
compile
2m 28s
master passed

+1 :green_heart:
checkstyle
0m 36s
master passed

+1 :green_heart:
spotless
0m 41s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 28s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 47s
the patch passed

+1 :green_heart:
compile
2m 26s
the patch passed

+1 :green_heart:
javac
2m 26s
the patch passed

+1 :green_heart:
checkstyle
0m 36s
hbase-server: The patch generated 0 new + 2 unchanged - 1 fixed = 2 total (was 3)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
4m 48s
Patch does not cause any errors with Hadoop 3.3.6.

+1 :green_heart:
spotless
0m 42s
patch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 39s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 10s
The patch does not generate ASF License warnings.

27m 50s

Subsystem
Report/Notes

Docker
ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5666

JIRA Issue
HBASE-27800

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile

uname
Linux d48a820d6911 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / afbf2adba2

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
78 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 27s
Docker mode activated.

-0 :warning:
yetus
0m  2s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 56s
master passed

+1 :green_heart:
compile
0m 46s
master passed

+1 :green_heart:
shadedjars
5m 44s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 23s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 50s
the patch passed

+1 :green_heart:
compile
0m 45s
the patch passed

+1 :green_heart:
javac
0m 45s
the patch passed

+1 :green_heart:
shadedjars
5m 45s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 23s
the patch passed

_ Other Tests _

-1 :x:
unit
222m 59s
hbase-server in the patch failed.

246m 52s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5666

JIRA Issue
HBASE-27800

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux f8a2975dddb0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / afbf2adba2

Default Java
Eclipse Adoptium-11.0.17+8

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/testReport/

Max. process+thread count
5528 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 26s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 45s
master passed

+1 :green_heart:
compile
0m 38s
master passed

+1 :green_heart:
shadedjars
5m 28s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 23s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 22s
the patch passed

+1 :green_heart:
compile
0m 39s
the patch passed

+1 :green_heart:
javac
0m 39s
the patch passed

+1 :green_heart:
shadedjars
5m 27s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 21s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
225m 45s
hbase-server in the patch passed.

248m  9s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5666

JIRA Issue
HBASE-27800

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 20d373a14779 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / afbf2adba2

Default Java
Temurin-1.8.0_352-b08

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/testReport/

Max. process+thread count
5710 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5666/2/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

The TestAcidGuaranteesWithBasicPolicy failure appears unrelated

authors: Apache-HBase rmdmattingly


text:
HIVE-24667: Truncate optimization to avoid unnecessary per partition DB get operations

What changes were proposed in this pull request?

Moved DB fetch out of partition loop
Why are the changes needed?

Performance improvement
Does this PR introduce any user-facing change?

No
How was this patch tested?

Unit tests

LGTM +1 pending tests.

LGTM +1 pending tests.

source: gharchive/pull-request

authors: deniskuzZ pvargacl

repo: apache/hive

license_type: permissive

license_source: github-api


text:
HIVE-26017 : Insert with partition value containing colon and space is creating partition having wrong value.

…

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

+1, Looks good to me


text:
[Bug]: The script transform randomly fails while obtaining scripting language

Apache Hop version?
2.7.0
Java version?
opernjdk 11.0.22
Operating system
Linux
What happened?
I run the same code every few hours and 95% of the time it works, but 5% of the time I get an error on the script transform about loading the engine.  This is with both python and EMCAScript language as the script engine.
ERROR: Error obtaining scripting engine for language ECMAScript
ERROR: java.util.ConcurrentModificationException
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.populateEngineFactoryMap(ScriptUtils.java:117)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.getScriptEngineManager(ScriptUtils.java:94)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.populateEngineFactoryMap(ScriptUtils.java:112)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.createNewScriptEngineByLanguage(ScriptUtils.java:59)
at org.apache.hop.pipeline.transforms.script.Script.init(Script.java:691)
at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
at java.base/java.lang.Thread.run(Thread.java:829
and
ERROR: Error obtaining scripting engine for language python
ERROR: java.util.ConcurrentModificationException
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.populateEngineFactoryMap(ScriptUtils.java:117)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.getScriptEngineManager(ScriptUtils.java:94)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.populateEngineFactoryMap(ScriptUtils.java:112)
at org.apache.hop.pipeline.transforms.script.ScriptUtils.createNewScriptEngineByLanguage(ScriptUtils.java:59)
at org.apache.hop.pipeline.transforms.script.Script.init(Script.java:691)
at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
at java.base/java.lang.Thread.run(Thread.java:829)
Issue Priority
Priority: 3
Issue Component
Component: Transforms

3729-concurrent-exception-in-scripts.hpl.txt

source: gharchive/issue

authors: bamaer fraserward

repo: apache/hop


text:
[SUPPORT] hudi table clustering failed and I want know what happened

Environment Description

Hudi version :
0.11.1
Spark version :
3.2.1
Hadoop version :
2.7.3
Storage :
hdfs

Describe the problem you faced
I have a hudi table which is about 100GB, and I want to cluster it .
I think I have given enough resources , but it clustered failed finally
To Reproduce

submit a clustering job

nohup spark-submit --queue mid --master yarn --name shuiqing_test_clustering --driver-memory 4g --executor-cores 4 --num-executors 8 --executor-memory 36g --conf spark.dynamicAllocation.enabled=false --conf spark.executor.memoryOverhead=12g \
--class org.apache.hudi.utilities.HoodieClusteringJob hudi-utilities-bundle_2.12-0.11.1.jar \
--props file:///home/work/test/clusteringjob.properties \
--mode scheduleAndExecute \
--base-path /user/work/testdb/bill_item \
--table-name bill_item \
--parallelism 8 \
--spark-memory 36g &> clustering.log &

clusteringjob.properties  details
hoodie.clustering.async.enabled=true
hoodie.clustering.async.max.commits=4
hoodie.clustering.plan.strategy.target.file.max.bytes=805306368
hoodie.clustering.plan.strategy.small.file.limit=268435456
hoodie.clustering.execution.strategy.class=org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy
hoodie.clustering.plan.strategy.sort.columns=id

watch spark job detail

table struct

_hoodie_commit_time 	string
_hoodie_commit_seqno	string
_hoodie_record_key  	string
_hoodie_partition_path	string
_hoodie_file_name   	string
id                  	string
rek_id              	string
item_id             	string
.........
item_flag           	string
ts                  	bigint
_key_partition      	string
# Partition Information
# col_name          	data_type           	comment
_key_partition      	string

Stacktrace
org.apache.spark.shuffle.FetchFailedException
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:1165)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:903)
	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:84)
	at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:205)
	at org.apache.spark.util.collection.ExternalSorter.insertAllAndUpdateMetrics(ExternalSorter.scala:686)
	at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:134)
	at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:106)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:106)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:131)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.UnsupportedOperationException
	at org.apache.spark.network.server.StreamManager.openStream(StreamManager.java:59)
	at org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:136)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:106)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:104)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
	at java.lang.Thread.run(Thread.java:748)
	at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:260)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:142)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more

org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
org.apache.hudi.data.HoodieJavaRDD.collectAsList(HoodieJavaRDD.java:157)
org.apache.hudi.table.action.commit.BaseCommitActionExecutor.executeClustering(BaseCommitActionExecutor.java:250)
org.apache.hudi.table.action.cluster.SparkExecuteClusteringCommitActionExecutor.execute(SparkExecuteClusteringCommitActionExecutor.java:53)
org.apache.hudi.table.HoodieSparkCopyOnWriteTable.cluster(HoodieSparkCopyOnWriteTable.java:188)
org.apache.hudi.client.SparkRDDWriteClient.cluster(SparkRDDWriteClient.java:363)
org.apache.hudi.utilities.HoodieClusteringJob.doScheduleAndCluster(HoodieClusteringJob.java:255)
org.apache.hudi.utilities.HoodieClusteringJob.lambda$cluster$0(HoodieClusteringJob.java:168)
org.apache.hudi.utilities.UtilHelpers.retry(UtilHelpers.java:541)
org.apache.hudi.utilities.HoodieClusteringJob.cluster(HoodieClusteringJob.java:155)
org.apache.hudi.utilities.HoodieClusteringJob.main(HoodieClusteringJob.java:132)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

I want know what is the problem, thank you !

Yes, I did . It seems  the version of external shuffle service is too old. Thank you for your help ! @danny0405

repo: apache/hudi


text:
[HUDI-6066] HoodieTableSource supports parquet predicate push down

Change Logs
HoodieTableSource supports the implementation of SupportsFilterPushDown interface that pushes down the CallExpression filter into FileIndex. HoodieTableSource should support parquet predicate push down to improve query performance, which evaluates FilterPredicate from CallExpression to push down FilterPredicate to ParquetColumnarRowSplitReader.
Impact
The impact is mainly query performance via parquet predicate push down.
Risk level (write none, low medium or high below)
low.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[x] Read through contributor's guide
[x] Change Logs and Impact were stated clearly
[x] Adequate tests were added if applicable
[x] CI passed

@danny0405, could you help to review this pull request?

@danny0405, I have addressed above comments. Could you please help to review again?

@danny0405, I have addressed normalized comments mentioned above. Please help to review again.

@danny0405, I have addressed comments. Please help to review again.

@danny0405 @XuQianJin-Stars will merge this PR if you do not have any active comments.

@danny0405, I have added test case whether the parquet predicate push down. PTAL.

@danny0405, I have added test case whether the parquet predicate push down. Meanwhile this pull request could improve the query performance. PTAL.

@danny0405 could you help review this again?

authors: SteNicholas leesf yihua


text:
Possible bug when using Emoji on Strings.

I have an issue which I discovered while trying to store an Emoji character inside a string - string which exists inside a tiny Object value.
My scenario is basically a chain of Get/Update operations where string value is acting strangely and I'm not sure if it is configuration issue or a bug.
Using debug tool, you can see '\0' character being added along with some other chars corrupting the value and reaching size limit eventually.
Not sure but looks like there is an issue either on Serialization/Deserialization of the string.
My setup:

.Net 6 which is referencing Apache.Ignite
Ignite 2.15
JAVA_HOME set to OpenJDK 11.0.20+8 - tried also OpenJDK_17.0.8_7 and same results.

Sample code below on setting up a dummy cache and doing Get/Update:
IIgnite ignite = Ignition.Start();
Apache.Ignite.Core.Cache.ICache<Guid, Person> cache = ignite.GetOrCreateCache<Guid, Person>(nameof(Person));
cache.Put( firstPerson.Id, new Person(){ Id = Guid.NewGuid(), Name="MrQ" } );

const int nbGetAndWrites = 8;
// Get, Update, Put same record multiple times and add each time an Emoji
for (int i = 0; i < nbGetAndWrites; i++)
{
    Person fromCache = cache.Get(firstPerson.Id);
    Console.WriteLine($"Step {i} got Person {fromCache}");
    fromCache.Name += $" [🐝]";
    cache.Put(fromCache.Id, fromCache);
}

Output:
Step 0 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ ]
Step 1 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ [🐝]' ]
Step 2 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ [🐝]' [🐝] ]
Step 3 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ [🐝]' [🐝] [🐝]( ]
Step 4 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ [🐝]' [🐝] [🐝]( [🐝]eee???? ]
Step 5 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ [🐝]' [🐝] [🐝]( [🐝]eee???? [🐝]ee???? ]
Step 6 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ [🐝]' [🐝] [🐝]( [🐝]eee???? [🐝]ee???? [🐝]g/garbage-collector).ee??v?? ]
Step 7 got Person [ Id=764d9799-6cb5-402e-93ba-ee3d1b87c0fb; Name=MrQ [🐝]' [🐝] [🐝]( [🐝]eee???? [🐝]ee???? [🐝]g/garbage-collector).ee??v?? [🐝]?v???v??`?v??`?v??p?v??p?v????v????v?? ]

GitHub Sample Code

Should I switch to Ignite 3 to test or get some feedback? or maybe it's the wrong place to ask/report issues

I could reproduce it. Clearly a bug. Let me dig deeper and get back to you later this week.

Turns out this was fixed long ago: https://issues.apache.org/jira/browse/IGNITE-3139
However, for some reason, the new string serialization mode is not enabled by default.
Add the following line in the beginning of the program:
Environment.SetEnvironmentVariable("IGNITE_BINARY_MARSHALLER_USE_STRING_SERIALIZATION_VER_2", "true");

Thanks for the feedback. All good on my end.

authors: micmnm ptupitsyn

repo: apache/ignite


text:
feature: using GitHub action for CI

Summary
using GitHub action for CI
Full changelog
feature: using GitHub action for CI
update:  .travis/linux_openresty_runner.sh
update: .travis/linux_tengine_runner.sh
update:  t/node/remote-addr-ipv6.t
Issues resolved
Fix #1789

@wingsjj many thx, merged

authors: membphis wingsjj

repo: apache/incubator-apisix


text:
[BEAM-523] Fixed typo in aggregator_test.py

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

@shlee0605 thank you for fixing this. This LGTM. Let's ask for a merge once the integrations tests pass.

The typo fix introduced a pylint error - I'll fix it up while I merge.

Thank you @kennknowles. And thank you again @shlee0605.

This has been merged.

@kennknowles @aaltay thanks!

authors: aaltay kennknowles shlee0605

repo: apache/incubator-beam


text:
树莓派上的编译问题

Is your feature request related to a problem? (你需要的功能是否与某个问题有关?)
我想再树莓派上编译一下brpc
Describe the solution you'd like (描述你期望的解决方法)
我去掉了msse等x86平台的优化参数
但是由于是32位的系统，导致编译不过去。。。。

size_t is 32 bit in 32-bit machine. Try to replace 32UL with (size_t)32 and recompile your code. When all errors are resolved and code can be compiled successfully, pull request is welcomed to solve this issue.

[ 10%] Built target PROTO_LIB
[ 45%] Built target BUTIL_LIB
[ 45%] Building CXX object src/CMakeFiles/OBJ_LIB.dir/bthread/butex.cpp.o
{standard input}: Assembler messages:
{standard input}:7556: 错误：bad instruction pause' {standard input}:7620: 错误：bad instruction pause'
{standard input}:7995: 错误：bad instruction `pause'
src/CMakeFiles/OBJ_LIB.dir/build.make:254: recipe for target 'src/CMakeFiles/OBJ_LIB.dir/bthread/butex.cpp.o' failed
make[2]: *** [src/CMakeFiles/OBJ_LIB.dir/bthread/butex.cpp.o] Error 1
CMakeFiles/Makefile2:208: recipe for target 'src/CMakeFiles/OBJ_LIB.dir/all' failed
make[1]: *** [src/CMakeFiles/OBJ_LIB.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2
i can not figure out how to solve this problem ....

不支持32位系统。

@jamesge 不支持 32 位系统是因为使用了什么 32 位系统中不支持的特性还是只是因为使用场景少而没考虑该问题? 有这方面的什么考虑吗?

authors: cg82616424 jamesge wasphin zyearn

repo: apache/incubator-brpc


text:
[HELM]Exposing pods details to users

What changes were proposed in this pull request?
Move affinity to values.yaml.
Why are the changes needed?
We should expose the details of the pod for users to view and modify in values.yaml
Does this PR introduce any user-facing change?
How was this patch tested?
Local test

Have you opened JIRA for this PR?

Have you opened JIRA for this PR?

I will add a ticket in jira.

Thanks, merging to main/0.2

repo: apache/incubator-celeborn


text:
[Improvement-3668][Server] ProcedureTask,SqlTask code structure standardization and checkstyle

What is the purpose of the pull request
#3668
ProcedureTask,SqlTask code structure standardization and checkstyle
Brief change log
1.ProcedureTask, SqlTask code structure standardization
The constructor is used to load the environment context
The init() method handles task parameters
2.ProcedureTask, SqlTask code checkstyle
Verify this pull request
This pull request is code cleanup without any test coverage.

##拉取请求的目的是什么
ProcedureTask，SqlTask代码结构标准化和检查样式
##简要更改日志
1.ProcedureTask，SqlTask代码结构标准化
构造函数用于加载环境上下文
init（）方法处理任务参数
2.ProcedureTask，SqlTask代码检查样式
##验证此拉取请求
此拉取请求是代码清除，没有任何测试范围。

Codecov Report

Merging #3669 into dev will increase coverage by 0.80%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##                dev    #3669      +/-   ##
============================================
+ Coverage     39.14%   39.95%   +0.80%
- Complexity     2829     2898      +69
============================================
  Files           456      457       +1
  Lines         21617    21669      +52
  Branches       2622     2637      +15
============================================
+ Hits           8462     8657     +195
+ Misses        12365    12195     -170
- Partials        790      817      +27

Impacted Files
Coverage Δ
Complexity Δ

.../server/worker/task/processdure/ProcedureTask.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...lphinscheduler/server/worker/task/sql/SqlTask.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...server/master/processor/TaskResponseProcessor.java
70.83% <0.00%> (-16.67%)
2.00% <0.00%> (-1.00%)

...apache/dolphinscheduler/common/utils/NetUtils.java
44.18% <0.00%> (-8.57%)
26.00% <0.00%> (+6.00%)
:arrow_down:

...ache/dolphinscheduler/remote/utils/NettyUtils.java
38.46% <0.00%> (-4.40%)
3.00% <0.00%> (+2.00%)
:arrow_down:

...pache/dolphinscheduler/common/utils/JSONUtils.java
67.14% <0.00%> (-2.86%)
18.00% <0.00%> (ø%)

...inscheduler/service/zk/CuratorZookeeperClient.java
56.75% <0.00%> (-2.71%)
6.00% <0.00%> (-1.00%)

...er/master/processor/queue/TaskResponseService.java
54.90% <0.00%> (-1.97%)
6.00% <0.00%> (+1.00%)
:arrow_down:

...nscheduler/api/service/ProcessInstanceService.java
73.83% <0.00%> (-0.47%)
36.00% <0.00%> (ø%)

...phinscheduler/remote/config/NettyClientConfig.java
52.00% <0.00%> (-0.39%)
7.00% <0.00%> (+1.00%)
:arrow_down:

... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 69201ac...eaddc82. Read the comment docs.

authors: codecov-commenter zhuangchong

repo: apache/incubator-dolphinscheduler


text:
Remove used doc content about be_rpc_port

Remove used doc content about be_rpc_port

#2695

repo: apache/incubator-doris


text:
push predicate to subquery

Proposed changes
Issue Number: close #8894
Problem Summary:
In some cases, pushing the predicate to subquery can get better performance like tpch q2 and q17
Checklist(Required)

Does it affect the original behavior: (Yes)
Has unit tests been added: (No)
Has document been added or modified: (No)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

rebase error, create a new pr https://github.com/apache/incubator-doris/pull/9686 instead


text:
Check the kafka topic when comparing checkpoints from tasks with the one stored in metastore

The key change is: https://github.com/apache/incubator-druid/compare/master...jihoonson:supervisor-check-topic?expand=1#diff-6eee87b3aa4eb3a516965fe6e93e25a4R1191.
If it doesn't check the topic, the supervisor considers the checkpoints in metastore invalid and tries to find something good offsets (earliest or latest) from Kafka. This causes offset mismatch and makes all tasks failed.
Maybe this is not a bug because we don't support changing topic in the supervisor spec officially, but at least this patch fixes a weird behavior which makes debugging harder.

👍

I'm removing this from 0.12.2 because 0.12.2-rc1 is available now but this isn't backported. I think it's fine because this is not a real bug but making the behavior more understandable when this error happens.

authors: fjy jihoonson

repo: apache/incubator-druid


text:
qos-server can not bind localhost:22222

项目启动的时候报错:qos-server can not bind localhost:22222

just perfect solution

repo: apache/incubator-dubbo


text:
EAGLE-394 : fix NPE and broken test

avoid mismatch metadata caused serialize/deserialize NPE issue
fix broken test in remote branch after merge

Quick(might be hurry) JIRA/Open fix to the exception found.

The is merged in f1a93b1bbc29aadbda5f09ddf2622252e8893b0b.
But looks the pr still show unmerged commit. Nevermind, close this manually.

repo: apache/incubator-eagle

license_source: bigquery


text:
Tooltip中设置了enterable:true   但是不能复制提示框浮层的文本内容

Version
4.2.1
Steps to reproduce
在tooltip中，已经设置了enterable:true   但仅仅只能鼠标进入，文本信息不能被选中，也不能被复制
What is expected?
选中，复制tooltip中的文本信息
What is actually happening?
文本信息不能被选中，也不能被复制

@pissang

已解决，修改了echarts源码，在div中添加了contenteditable属性

源码是在创建的echarts的dom上加了user-select: none;你可以在你的fommater里面的dom修改user-select:text

authors: ahahaah fashiontai

repo: apache/incubator-echarts


text:
Stack堆叠柱图在X轴对应多个柱状条的时候无法获取柱状索引

Version
4.2.1
Steps to reproduce
堆叠柱状图在X轴(维度)对应多个指标时候，在使用tooltip formmat回调的时候无法获取当前柱子的索引(无法匹配指标).相关数据及配置如下:
var measureData = [
		{
			aggregatorName: "去重数",
			field: "xx1",
			fieldName: "平均处置时长",
			unit:"小时"
		},
		{
			aggregatorName: "去重数",
			field: "xx2",
			fieldName: "受理数",
			unit:"件"
		}
];

var option = {
	"tooltip": {
		"trigger": "axis",
		"axisPointer": {
			"type": "shadow"
	},
	formatter: function (params) {
            var colorSpan = color => '';
            let rez = '
'   params[0].axisValue   '
';
            console.log(params); //quite useful for debug
            params.forEach(item => {

            	//TODO 我需要获取当前这个柱子在stack中的的索引

            	//just test but failed
                var seriesId = String(item.seriesId);
                var index = seriesId.trim().split(" ")[1];
                var measure = measureData[index];

                //console.log(item); //quite useful for debug
                var xx = '
'     colorSpan(item.color)   ' '   measure.fieldName   '_'   item.seriesName   ': '   item.data   measure.unit   '
'
                rez  = xx;
            });

            return rez;
	},
	"legend": {
		"type": "scroll",
		"data": ["咨询", "投诉", "求决", "其他", "咨询", "投诉", "求决", "其他"]
	},
	"grid": {
		"left": "3%",
		"right": "4%",
		"bottom": "3%",
		"containLabel": true
	},
	"xAxis": [{
		"type": "category",
		"data": ["街道1", "街道2", "街道3", "街道4", "街道5", "街道6"]
	}],
	"yAxis": [{
		"type": "value"
	}],
	"series": [{
		"name": "咨询",
		"type": "bar",
		"stack": "平均处置时长",
		"barMaxWidth": "30px",
		"data": [2.0266, 0, 0, 0, 0, 0]
	}, {
		"name": "投诉",
		"type": "bar",
		"stack": "平均处置时长",
		"barMaxWidth": "30px",
		"data": [1.6674, 3.7226, 2.7217, 1.6684, 2.2107, 1.3616]
	}, {
		"name": "求决",
		"type": "bar",
		"stack": "平均处置时长",
		"barMaxWidth": "30px",
		"data": [0.8988, 0, 0, 0, 0, 0]
	}, {
		"name": "其他",
		"type": "bar",
		"stack": "平均处置时长",
		"barMaxWidth": "30px",
		"data": [0, 0, 0, 1.7018, 0, 0]
	}, {
		"name": "咨询",
		"type": "bar",
		"stack": "受理数",
		"barMaxWidth": "30px",
		"data": [5, 0, 0, 0, 0, 0]
	}, {
		"name": "投诉",
		"type": "bar",
		"stack": "受理数",
		"barMaxWidth": "30px",
		"data": [33, 31, 29, 22, 23, 8]
	}, {
		"name": "求决",
		"type": "bar",
		"stack": "受理数",
		"barMaxWidth": "30px",
		"data": [1, 0, 0, 0, 0, 0]
	}, {
		"name": "其他",
		"type": "bar",
		"stack": "受理数",
		"barMaxWidth": "30px",
		"data": [0, 0, 0, 2, 0, 0]
	}]
}

What is expected?
在tooltip formmat函数中获取params的所属柱子的索引
What is actually happening?
无法区别

在多指标的情况，多个柱子无法区别当前柱子的索引，我看了官网文档及SO等并没有解决方案，希望给予支持。万分感谢。

Please provide a demo for us to debug, I can't run with your option.

authors: deqingli lapDog


text:
yAxis.axisTick.lineStyle.color - gradient not supported

Version
4.8.0
Reproduction link
https://jsfiddle.net/Lt1xd3su/
Steps to reproduce
See fiddle.
Create an axisTick with a gradient:
"axisTick": {
      "show": true,
      "inside": true,
      "length": 1020,
      "lineStyle": {
        "color": {
          "type": "linear",
          "x": 0,
          "y": 0,
          "x2": 1,
          "y2": 1,
          "colorStops": [{
            "offset": 0,
            "color": "#f00"
          }, {
            "offset": 0.25,
            "color": "#0f0"
          }, {
            "offset": 1,
            "color": "#00f"
          }],
          "global": true
        }
      }
    }

What is expected?
the tick is colored with a linear gradient
What is actually happening?
color at offset 1 is the only visible color

Does not seem to change if tick is inside or not (no gradient in either case)
Was attempting to use a gradient to get around fact that length of axisTick is not scalable (only accepts Number) (by setting to large value, and then hiding end of line with line color matching to background)

ah, ok! Thank you for the explanation & help


text:
tooltip trigger 'axis' | Show tooltip with intersecting data points only

Version
4.1.0-release
Reproduction link
http://jsfiddle.net/vikashvickyssec/vLosyz1p/1/
Steps to reproduce
Multiple series of lines having same value at any one x-axis point shows all the series value at that point.
What is expected?
There should have been an option to display only intersecting data points
What is actually happening?
Currently if tooltip triggered by 'axis' shows all the series value at that point

If multiple series have same value at any x-axis point, tooltip triggered by 'axis' shows all the series value at that point. Is there way, only intersecting points can be displayed in the tooltip.
Even with triggered set to 'item', the series with higher z-index is only shown in the tooltip.
For me I need to display all the overlapping data points in the tooltip.

Sorry, this was by design


text:
官方示例当中的柱状图，添加 Y 轴的 min 值之后，图形向下溢出，如何隐藏掉溢出的部分

One-line summary [问题简述]
官方示例当中的柱状图，添加 Y 轴的 min 值之后，图形向下溢出，如何隐藏掉溢出的部分，如下图所示
经过测试，折线图等无此类问题，可以正常显示 min 和 max 值，柱状图下会有溢出部分

Version & Environment [版本及环境]

ECharts version [ECharts 版本]:  4.1.0,
Browser version [浏览器类型和版本]: Chrome
OS Version [操作系统类型和版本]: MAC

Expected behaviour [期望结果]
添加了 Y 轴的 min 值之后，图形溢出的部分可以隐藏掉就好
ECharts option [ECharts配置项]

使用的官方的基本柱状图模版，仅仅添加了一个 Y 轴的 min 属性
option = {
    color: ['#3398DB'],
    tooltip : {
        trigger: 'axis',
        axisPointer : {            // 坐标轴指示器，坐标轴触发有效
            type : 'shadow'        // 默认为直线，可选为：'line' | 'shadow'
        }
    },
    grid: {
        left: '3%',
        right: '4%',
        bottom: '3%',
        containLabel: true
    },
    xAxis : [
        {
            type : 'category',
            data : ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'],
            axisTick: {
                alignWithLabel: true
            }
        }
    ],
    yAxis : [
        {
            type : 'value',
            min: '100'
        }
    ],
    series : [
        {
            name:'直接访问',
            type:'bar',
            barWidth: '60%',
            data:[10, 52, 200, 334, 390, 330, 220]
        }
    ]
};

解决了吗？

通过修改源码已经解决了，官方示例当中的折线图有溢出隐藏的方法
但是 bar 类图表没有这个方法，copy 一个过去改一改可以解决问题

authors: Zillionnn hanekaoru


text:
showLoading操作不能自定义变量

One-line summary [问题简述]
this.chart.showLoading(
{
text: '请稍后...',
color: 'blue',
textColor: '#fff',
maskColor: 'rgba(0, 0, 0, 0.8)',
zlevel: 0
}
)
无法自定义
Version & Environment [版本及环境]

ECharts version [4.2.0-rc.2 ]:
Browser version [chrome65]:
OS Version [10.13.4 (17E202)]:

Expected behaviour [期望结果]
ECharts option [ECharts配置项]

option = {

}

Other comments [其他信息]

可以参考 https://github.com/apache/incubator-echarts/blob/master/src/loading/default.js 实现自己的 loading 效果，使用 echarts.registerLoading 方法注册，参考 https://github.com/apache/incubator-echarts/blob/master/src/echarts.js#L2330

echarts [4.2.0-rc.2]似乎没有暴露这个 registerLoding   api ...
另外，可否注册到主题中~

可以参考 https://github.com/apache/incubator-echarts/blob/master/src/loading/default.js 实现自己的 loading 效果，使用 echarts.registerLoading 方法注册，参考 https://github.com/apache/incubator-echarts/blob/master/src/echarts.js#L2330

你好，针对这个方法实验后端效果  似乎并不理想,效果没有被覆盖
echarts.registerTheme('customed',...conf);
echarts.registerLoading('customed', {
text: '请稍后...',
color: 'blue',
textColor: '#fff',
maskColor: 'rgba(0, 0, 0, 0.8)',
zlevel: 0
})
结果还是原来多

http://gallery.echartsjs.com/editor.html?c=x-bLMl6BVu&v=1

authors: cuijian-dexter guoqn pissang


text:
[CORE] ParquetFileForamt/OrcFileFormat has code like if (xxx && false)

Description
Not sure why we make this if branch always skipped.
See https://github.com/apache/incubator-gluten/blob/main/shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala#L204

Hi @binmahone, do you remember the reason?

@PHILO-HE does you try if CI pass if remove the false check ?

@PHILO-HE does you try if CI pass if remove the false check ?

@gaoyangxiaozhu, thanks for your comment! I will try to do that.

Hi @binmahone, do you remember the reason?

Sorry I can't really recall. When the intial PR was checked in, velox backend did not actually supported native writer.

Hi @binmahone, do you remember the reason?

Sorry I can't really recall. When the intial PR was checked in, velox backend did not actually supported native writer.

@binmahone, I see. Anyway, thanks for your reply!

As confirmed with @JkSelf, we can remove this code to always use vanilla Spark's API for Velox backend. #6663 is removing the useless code.

authors: PHILO-HE binmahone gaoyangxiaozhu

repo: apache/incubator-gluten


text:
Add two backup urls for zookeeper

I've experiences some flakiness with a few of the zookeeper archive
urls. This adds two backup urls in the event that one of them is down or
a connection cannot be established.
Zookeeper was the only obvious candidate that had mirrors so we can start here and add other mirrors if people experience flakiness for other archives.
Closes https://github.com/apache/incubator-heron/issues/2855

👍

repo: apache/incubator-heron


text:
[WIP] Vectorized read using Arrow

Just a WIP POC right now so is not expected to be merged. This is just for comments.
Things I intend to add:

a separate module iceberg-arrow that will house the code for reading parquet into Arrow
Iceberg's Reader needs to choose Row-wise or Vectorized reading based on config.
issue#311
batch sizing as config issue#312

/cc @danielcweeks @anjalinorwood @samarthjain

tests fail but chefkstyle checks pass.

@prodeezy I'm going to go ahead and commit this now, which will hopefully unblock others to start making pull requests.

authors: danielcweeks prodeezy

repo: apache/incubator-iceberg


text:
[INLONG-806] open GitHub Issue For InLong

Fixes Jira Issue ID:
https://issues.apache.org/jira/projects/INLONG/issues/INLONG-806
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Codecov Report

Merging #597 (58339e3) into master (c13f532) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master    #597      +/-   ##
===========================================
- Coverage      7.73%   7.72%   -0.01%
+ Complexity      565     564       -1
===========================================
  Files           355     355
  Lines         31081   31081
  Branches       4972    4972
===========================================
- Hits           2403    2401       -2
- Misses        28188   28190       +2
  Partials        490     490

Impacted Files
Coverage Δ

.../inlong/tubemq/corebase/policies/FlowCtrlItem.java
38.88% <0.00%> (-1.12%)
:arrow_down:

.../tubemq/corebase/policies/FlowCtrlRuleHandler.java
43.98% <0.00%> (-0.59%)
:arrow_down:

.../producer/qltystats/DefaultBrokerRcvQltyStats.java
45.70% <0.00%> (+0.39%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c13f532...58339e3. Read the comment docs.

authors: codecov-commenter dockerzhang

repo: apache/incubator-inlong


text:
Add the support of the ZINTER, ZINTERCARD commands

Search before asking

[X] I had searched in the issues and found no similar issues.

Motivation
This issue considers to support zinter, zintercard commands.
Solution

[ ] zinter
[ ] zintercard

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

i want to do this

@IoCing Assigned, thanks

Has been done in https://github.com/apache/kvrocks/issues/1983, close.

authors: IoCing git-hulk infdahai jihuayu

repo: apache/incubator-kvrocks


text:
[FEATURE] Support CostMode for PlanOnlyStatement

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the feature
I want to support cost mode for PlanOnlyStatement.
Motivation
Show the statistics of optimizedPlan.
Describe the solution
Add OPTIMIZE_WITH_STATS plan mode
Additional context
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

I want to implement it please assign me.

I want to implement it please assign me.

Thanks, assigned.

authors: Kwafoor wForget

repo: apache/incubator-kyuubi


text:
METRON-102: Unit Tests for parsers and enrichment adapters

Added several new unit tests for existing parsers and enrichment adapters.  One of the tests needed the MockHTable class which currently lives in Metron-Testing.  Since Metron-Testing depends on several other projects (including the project that needs MockHTable), a separate Metron-TestingUtilities project was created with no dependencies on any projects.  Now tests written for any project have access to this class.

+1 this looks really good. Ran through the tests no problem.

@franklu1 put in a lot of work for this PR but his name was in a commit that got squashed.  Whoever merges this please include him so he gets credit.

repo: apache/incubator-metron


text:
test_ndarray.test_ndarray_elementwise has fixed seed that can mask flakiness

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
If numerical instability is inevitable, adjust tolerance level appropriately.

Thanks for filing this issue. We will investigate this Flaky test

authors: apeforest szha

repo: apache/incubator-mxnet


text:
why not the item 'quality' of 'encode_params = [cv2.IMWRITE_PNG_COMPRESSION, quality]' is  from 0 to 9?

When I read the module python/mxnet/recordio.py and notice that the item 'quality' of 'encode_params = [cv2.IMWRITE_PNG_COMPRESSION, quality]' is  from 0 to 9 in function pack_img. I am wondering. Would you like to explain to me?  Thank you .

please refer to opencv's document

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!

authors: JulyUp phunterlau piiswrong


text:
Batch processing for rcnn prediction

I want to send pictures to rcnn network for prediction by batch. Does the current rcnn function like im_detect() and Predictor() support this. If they do not support this function, what change should I do to achieve the goal?

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!
Also, do please check out our forum (and Chinese version) for general "how-to" questions.

authors: YvesHarrison szha


text:
Segmentation Fault

Hello everyone, Im following this post for my studies "https://mxnet.incubator.apache.org/tutorials/embedded/wine_detector.html" but whenever I run the python camera_test.py on the Terminal, this shows up. Can someone tell me what should I do in order to progress. Thanks.
pi@raspberrypi:~ $ python camera_test.py
[13:37:44] src/nnvm/legacy_json_util.cc:190: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
Segmentation fault

@jimm1973 Closing this issue for now, please reopen if issue persists and a core file is available.
@sandeep-krishnamurthy Please add label:  Pending Requester Info, and close this issue
Thanks!

I'm running into the same issue using MXNet v1.2.0 on a Raspberry Pi 3+. I've made a core dump and did a backtrace:
pi@raspberrypi:~/incubator-mxnet/python $ gdb /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so core
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so...done.

warning: core file may not match specified executable file.
[New LWP 5251]
[New LWP 5262]
[New LWP 5255]
[New LWP 5254]
[New LWP 5265]
[New LWP 5263]
[New LWP 5256]
[New LWP 5264]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `python'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x75674978 in nnvm::CreateVariableNode(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
[Current thread is 1 (Thread 0x76f628c0 (LWP 5251))]
(gdb) bt
#0  0x75674978 in nnvm::CreateVariableNode(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#1  0x75674a30 in nnvm::Symbol::CreateVariable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#2  0x750a2d88 in mxnet::UpgradeJSON_Parse(nnvm::Graph) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#3  0x750a8a5c in std::_Function_handler<nnvm::Graph (nnvm::Graph), nnvm::Graph (*)(nnvm::Graph)>::_M_invoke(std::_Any_data const&, nnvm::Graph&&) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#4  0x7509d29c in mxnet::LoadLegacyJSONPass(nnvm::Graph) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#5  0x750a8a5c in std::_Function_handler<nnvm::Graph (nnvm::Graph), nnvm::Graph (*)(nnvm::Graph)>::_M_invoke(std::_Any_data const&, nnvm::Graph&&) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#6  0x75672cf0 in nnvm::ApplyPasses(nnvm::Graph, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#7  0x750d01f0 in nnvm::ApplyPass(nnvm::Graph, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#8  0x755433a8 in MXSymbolCreateFromFile () from /home/pi/incubator-mxnet/python/mxnet/../../lib/libmxnet.so
#9  0x76935dd0 in ffi_call_VFP () from /usr/lib/arm-linux-gnueabihf/libffi.so.6
#10 0x769367ec in ffi_call () from /usr/lib/arm-linux-gnueabihf/libffi.so.6
#11 0x76954b68 in _ctypes_callproc () from /usr/lib/python2.7/lib-dynload/_ctypes.arm-linux-gnueabihf.so
Backtrace stopped: Cannot access memory at address 0x52
(gdb)

See this post on the MXNet forum for more info. Let me know when you need more information.

Can you try with the naive engine? https://mxnet.incubator.apache.org/faq/env_var.html#engine-type @lebeg

I've tried on v1.3.0 (current HEAD)  and v1.2.0 branch (which is actually v1.2.1) and had the same results. It breaks in the same CreateVariableNode function.

It looks like a duplicate to #9094

@tvandergeer how did you build? Our docker builds?

@larroy I followed this howto (select "Devices" and "Raspberry Pi"). I tried with both export USE_OPENCV = 0 and export USE_OPENCV = 1

We also encountered this issue. Thanks for the backtrace!    If you use naive engine I think it won't crash: https://mxnet.incubator.apache.org/faq/env_var.html
Could you confirm?

I've did the following:
make clean
export MXNET_ENGINE_TYPE=NaiveEngine
make

It still segfaults with the same error

I will try to have a look at this next week when I have access to a raspberry Pi. Feel free to ping this issue again if you don't see activity.

Try to set the environment variable export MXNET_ENGINE_TYPE=NaiveEngine during runtime, not during build.

Hi all,
I'm having exaclty the same issue, I tried on both Raspberry Pi 3, and Raspberry Pi 3+.
I compiled the latest mxnet using arm-linux-gnueabihf-gcc and did "export MXNET_ENGINE_TYPE=NaiveEngine" during runtime as suggested above in this post.
I have the same backtrace reported by tvandergeer .
Any suggestions?
Thanks

@lebeg @larroy Bouncing this ...

Let's close this, I suggest reopen with a backtrace with debug symbols if anyone still experience difficulties.

authors: jimm1973 larroy lebeg mdenna-nviso piyushghai roywei tvandergeer


text:
CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR on jetson TX2

Description
CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR on jetson TX2. Cudnn seems not work and inference speed is slow.
Environment info (Required)
Ubuntu 16.04
JetPack 3.1：including CUDA V8.0.72, Cudnn v6.0.21
Mxnet v1.0.0

Package used
Python & cpp
Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
gcc
Build config:
1、make -j8 USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1
2、make -j8 USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda
3、make -j8 with modified config.mk like below

Error Message:
[02:59:29] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
terminate called after throwing an instance of 'dmlc::Error'
what():  [02:59:32] src/engine/./threaded_engine.h:359: [02:59:32] src/operator/nn/./cudnn/cudnn_convolution-inl.h:614: Check failed: e == CUDNN_STATUS_SUCCESS (4 vs. 0) cuDNN: CUDNN_STATUS_INTERNAL_ERROR
Stack trace returned 9 entries:
[bt] (0) ./ssd(_ZN4dmlc10StackTraceB5cxx11Ev+0x48) [0x407dd4]
[bt] (1) ./ssd(_ZN4dmlc15LogMessageFatalD2Ev+0x4c) [0x4080ac]
[bt] (2) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(ZZN5mxnet2op18CuDNNConvolutionOpIfE10SelectAlgoERKNS_7ContextERKSt6vectorIN4nnvm6TShapeESaIS8_EESC_15cudnnDataType_tSD_ENKUlNS_10RunContextENS_6engine18CallbackOnCompleteEE_clESE_SG+0x330) [0x7fa015bd28]
[bt] (3) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(ZNSt17_Function_handlerIFvN5mxnet10RunContextENS0_6engine18CallbackOnCompleteEEZNS0_2op18CuDNNConvolutionOpIfE10SelectAlgoERKNS0_7ContextERKSt6vectorIN4nnvm6TShapeESaISD_EESH_15cudnnDataType_tSI_EUlS1_S3_E_E9_M_invokeERKSt9_Any_dataOS1_OS3+0x2c) [0x7fa015cbe4]
[bt] (4) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x98) [0x7f9f694ff0]
[bt] (5) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5+0x144) [0x7f9f69d2a4]
[bt] (6) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x48) [0x7f9f697c90]
[bt] (7) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f9c870280]
[bt] (8) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f8924dfc4]
A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 7 entries:
[bt] (0) ./ssd(_ZN4dmlc10StackTraceB5cxx11Ev+0x48) [0x407dd4]
[bt] (1) ./ssd(_ZN4dmlc15LogMessageFatalD2Ev+0x4c) [0x4080ac]
[bt] (2) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine15ExecuteOprBlockENS_10RunContextEPNS0_8OprBlockE+0x2d4) [0x7f9f69522c]
[bt] (3) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5+0x144) [0x7f9f69d2a4]
[bt] (4) /home/nvidia/workspace/mxnet-ssd-minning/lib/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x48) [0x7f9f697c90]
[bt] (5) /usr/lib/aarch64-linux-gnu/libstdc++.so.6(+0xb8280) [0x7f9c870280]
[bt] (6) /lib/aarch64-linux-gnu/libpthread.so.0(+0x6fc4) [0x7f8924dfc4]
Steps to reproduce
1、follow the instruction to flash JetPack3.1 to TX2, (http://docs.nvidia.com/jetpack-l4t/index.html#developertools/mobile/jetpack/l4t/3.0/jetpack_l4t_install.htm)
2、follow the guide git clone this ssd repo (https://github.com/zhreshold/mxnet-ssd) and compile mxnet-ssd from source code.
3、try ssd demo with code: python demo.py --gpu 0
What have you tried to solve it?

My test method is to use cv::getTickCount(). My main purpose is to do a body detection demo on TX2. I trained and tested a model on a windows PC where mxnet ssd is way faster than caffe faster-rcnn (mxnet 20ms vs caffe 35ms). But the result reversed on TX2 (mxnet 180ms vs caffe 45ms). And I have to set environment MXNET_CUDNN_AUTOTUNE_DEFAULT=0 to run the gpu demo. When I set MXNET_CUDNN_AUTOTUNE_DEFAULT to 1 I got the error above.
I reinstall mxnet several time trying different make config.mk. Found that the inference speed remains the same no matter if I set USE_CUDNN=1 or not. So I doubt cudnn doesn't work.
I try to update cudnn from 6.0.21 to 7.0.1 under guide of https://devtalk.nvidia.com/default/topic/1024658/where-to-find-cudnn-7-packages-for-tx2/ but still the same error. (I copied the corresponding files to cuda directories.)
I found some config to fix TX2 to a higher clock frequency. And the speed improve from 200ms to 180ms. But I don't think it's due to cudnn.

It's my first issue on such big repo. Please point my question out if I'm not specific. Anything would be appreciated, thanks.

Thank you very much @KellenSunderland !
I am still trying your suggestion (stuck in docker configure while rebuild the kernel). I'll tell you the result when I finished it.

Hi @KellenSunderland ~
I managed to install mxnet on TX2 using docker image build on the host under as you suggested.
There's no error about CUDNN_STATUS_SUCCESS (4 vs. 0), but the speed is still the same. I cannot feel the existence of cudnn. And I think maybe that's because the line 15 in Dockerfile.build.master.jetson, I still built mxnet without cudnn.
When I set line  15 to ENV BUILD_OPTS "USE_OPENCV=0 USE_BLAS=openblas USE_SSE=0 USE_CUDA=1 USE_CUDNN=1 ENABLE_CUDA_RTC=0 USE_NCCL=0 USE_CUDA_PATH=/usr/local/cuda/" and rerun the docker script, I got error like this:
compilation terminated.
In file included from /work/mxnet/mshadow/mshadow/tensor.h:16:0,
from include/mxnet/./base.h:32,
from include/mxnet/operator_util.h:43,
from src/operator/contrib/./bounding_box-inl.h:27,
from src/operator/contrib/bounding_box.cc:27:
/work/mxnet/mshadow/mshadow/./base.h:163:21: fatal error: cudnn.h: No such file or directory
#include <cudnn.h>
^
compilation terminated.
In file included from /work/mxnet/mshadow/mshadow/tensor.h:16:0,
from include/mxnet/./base.h:32,
from include/mxnet/operator.h:38,
from src/operator/contrib/./deformable_psroi_pooling-inl.h:32,
from src/operator/contrib/deformable_psroi_pooling.cc:27:
/work/mxnet/mshadow/mshadow/./base.h:163:21: fatal error: cudnn.h: No such file or directory
#include <cudnn.h>
^
compilation terminated.
Makefile:393: recipe for target 'build/src/operator/contrib/krprod.o' failed
make: *** [build/src/operator/contrib/krprod.o] Error 1
make: *** Waiting for unfinished jobs....
Makefile:393: recipe for target 'build/src/operator/contrib/bounding_box.o' failed
make: *** [build/src/operator/contrib/bounding_box.o] Error 1
Makefile:393: recipe for target 'build/src/operator/nn/cudnn/cudnn_algoreg.o' failed
make: *** [build/src/operator/nn/cudnn/cudnn_algoreg.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/dequantize.o' failed
make: *** [build/src/operator/contrib/dequantize.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/fft.o' failed
make: *** [build/src/operator/contrib/fft.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/multibox_target.o' failed
make: *** [build/src/operator/contrib/multibox_target.o] Error 1
Makefile:393: recipe for target 'build/src/operator/nn/cudnn/cudnn_batch_norm.o' failed
make: *** [build/src/operator/nn/cudnn/cudnn_batch_norm.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/count_sketch.o' failed
make: *** [build/src/operator/contrib/count_sketch.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/multi_proposal.o' failed
make: *** [build/src/operator/contrib/multi_proposal.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/ctc_loss.o' failed
make: *** [build/src/operator/contrib/ctc_loss.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/multibox_prior.o' failed
make: *** [build/src/operator/contrib/multibox_prior.o] Error 1
Makefile:393: recipe for target 'build/src/operator/contrib/deformable_psroi_pooling.o' failed
make: *** [build/src/operator/contrib/deformable_psroi_pooling.o] Error 1
The command '/bin/sh -c make -j$(nproc) $BUILD_OPTS' returned a non-zero code: 2
So~ still trying~

Indeed that seems to be a bug, I should be building with cudnn enabled.  Let me try and update it.  I've run into this compile error a few times before, so hopefully it won't take long to fix.

Thanks again for iterating on this.  The guide is updated with a build that should include CUDNN.  Sorry I missed it in the first file.

Thanks @KellenSunderland.
Some said it may because out of memory (https://github.com/shicai/MobileNet-Caffe/issues/3). I've seen the out-of-gpu-memory error before and checked the RAM usage with tegrastats so I didn't link this error with that reason at first. And I had tried vgg16-ssd & res50-ssd models. But when I tried res18-ssd model, everything works fine. I got the speed of 40-50ms per frame(cuda 9.0 & cudnn 7.0 JetPack 3.2) and 27-30ms per frame(with nvpmodel -m 0 and ./jetson_clocks.sh). Don't know the deep reason but that's the solution for now.
And thank you again @KellenSunderland. Your method of docker installation works great, I learnt a lot from your suggestion.

authors: KellenSunderland yanhn


text:
Error in output.shape[[output.names]]

Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.
For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io
Description
Failed to get some dimension properties from a multidimensional array in R, right in the beginning of the feedforward.create function.
Environment info (Required)
What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.

C:\Users\admin\Desktop>python diagnose.py
----------Python Info----------
Version      : 3.6.3
Compiler     : MSC v.1900 64 bit (AMD64)
Build        : ('v3.6.3:2c5fed8', 'Oct  3 2017 18:11:49')
Arch         : ('64bit', 'WindowsPE')
------------Pip Info-----------
Version      : 9.0.1
Directory    : C:\Users\admin\AppData\Local\Programs\Python\Python36\lib\site-pa
ckages\pip
----------MXNet Info-----------
No MXNet installed.
----------System Info----------
Platform     : Windows-7-6.1.7601-SP1
system       : Windows
node         : ceamphd82
release      : 7
version      : 6.1.7601
----------Hardware Info----------
machine      : AMD64
processor    : Intel64 Family 6 Model 61 Stepping 4, GenuineIntel
Traceback (most recent call last):
  File "diagnose.py", line 177, in <module>
    check_hardware()
  File "diagnose.py", line 143, in check_hardware
    subprocess.call(['wmic', 'cpu', 'get', 'name'])
  File "C:\Users\admin\AppData\Local\Programs\Python\Python36\lib\subprocess.py"
, line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Users\admin\AppData\Local\Programs\Python\Python36\lib\subprocess.py"
, line 709, in __init__
    restore_signals, start_new_session)
  File "C:\Users\admin\AppData\Local\Programs\Python\Python36\lib\subprocess.py"
, line 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

Package used (Python/R/Scala/Julia):
(I'm using R)
For R user, please provide R sessionInfo():
R version 3.4.1 (2017-06-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
Matrix products: default
locale:
[1] LC_COLLATE=English_United Kingdom.1252
[2] LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats     graphics  grDevices utils     datasets
[6] methods   base
other attached packages:
[1] mxnet_0.10.1   RMySQL_0.10.13 DBI_0.7
loaded via a namespace (and not attached):
[1] Rcpp_0.12.12       pillar_1.1.0
[3] compiler_3.4.1     RColorBrewer_1.1-2
[5] influenceR_0.1.0   plyr_1.8.4
[7] bindr_0.1          viridis_0.4.0
[9] tools_3.4.1        digest_0.6.12
[11] jsonlite_1.5       viridisLite_0.2.0
[13] tibble_1.4.1       gtable_0.2.0
[15] rgexf_0.15.3       pkgconfig_2.0.1
[17] rlang_0.1.2        igraph_1.1.2
[19] rstudioapi_0.7     yaml_2.1.14
[21] bindrcpp_0.2       gridExtra_2.3
[23] downloader_0.4     DiagrammeR_0.9.2
[25] dplyr_0.7.3        stringr_1.2.0
[27] htmlwidgets_0.9    hms_0.3
[29] grid_3.4.1         glue_1.1.1
[31] R6_2.2.2           Rook_1.1-1
[33] XML_3.98-1.9       readr_1.1.1
[35] purrr_0.2.3        tidyr_0.7.2
[37] ggplot2_2.2.1      magrittr_1.5
[39] codetools_0.2-15   scales_0.5.0
[41] htmltools_0.3.6    assertthat_0.2.0
[43] colorspace_1.3-2   brew_1.0-6
[45] stringi_1.1.5      visNetwork_2.0.1
[47] lazyeval_0.2.0     munsell_0.4.3
Error Message:
Error in output.shape[[output.names]] <- dim((X$value())$label) :
attempt to select less than one element in OneIndex
Minimum reproducible example
(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)
Steps to reproduce
x is Large array (492 * 426 * 3 * 126) - width, height,channel, samples/
y is factor vector of size 126;

mo <- mx.symbol.Variable('data');
m1 <- mx.symbol.FullyConnected(data = mo, num.hidden = 20, no.bias = FALSE, name ='damn')
Model <- mx.model.FeedForward.create(m1,X=x,y=YDataset$Classification, ctx = cpu, num.round = 30, array.batch.size = 100, learning.rate = 0.05, momentum = 0.9, wd=0.0001, eval.metric = mx.metric.accuracy) <- Error only happens here.

What have you tried to solve it?

I have no idea what to do when your x data has no labels. Is this necessary?
I cannot attach the data since its bigger than 10MB xD. Here it is: https://drive.google.com/file/d/1uM8AotCKGUGMsrAZNYwJYWuVUv7K0JWT/view?usp=sharing

mx.model.FeedForward.create is a function intended for training of the model, it is therefore expected to have a label passed as the y argument.
Your model should have a loss function for in order to train; for example, mx.symbol.LinearRegressionOutput. Please take a look at this MNIST tutorial for example:  http://mxnet.incubator.apache.org/tutorials/r/mnistCompetition.html.
You need to be aware of the the data dimensions through the model: if 126 samples in X, Y should also be of length 126. Check the model arhcitecture used: a FC on images (which should normally be of equal WH dimensions) is likely not what is needed (check for convolution layers). And if performing a 2 class classification, then needs a final FC with num_hidden = 2 followed by a softmaxoutput or rather a FC of size 1 followed by a LogisticLinearOutput.

@mxnet-label-bot add [R]

authors: AndreGuerra123 jeremiedb vandanavk


text:
Revert "Static memory allocation for cached_op (#10817)"

This reverts commit 2dbd143e4892bb9ad4aa1835c79f0046603e3531.
See https://github.com/apache/incubator-mxnet/issues/11171 for reference. #10817 has not been approved and is causing regressions.

can we move forward instead of backward?

close due to bugfix introduced in #11117

https://github.com/apache/incubator-mxnet/pull/11310

Who talked about not self-merging....

with great power comes great responsibility. I would never abused that power to any changes just to make a point and bring things backward.

authors: marcoabreu tqchen zheng-da


text:
Disable broken test test_io.test_CSVIter

https://github.com/apache/incubator-mxnet/issues/12139

LGTM

Thank you, I have changed the description and also edited the title of your issue.

LGTM, please merge this we are blocked: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-12148/2/pipeline

authors: larroy marcoabreu


text:
GET_ENVIRON_PTR()

  Description: get_environ_ptr() (sched/sched_getenvironptr.c) is not implemented.
               The representation of the environment strings selected for
               NuttX is not compatible with the operation.  Some significant
               re-design would be required to implement this function and that
               effort is thought to be not worth the result.
  Status:      Open.  No change is planned.
  Priority:    Low -- There is no plan to implement this.

https://github.com/apache/incubator-nuttx/issues/3391
Description: The definition of environ in stdlib.h is bogus and will not
               work as it should.  This is because the underlying
               representation of the environment is not an array of pointers.
  Status:      Open
  Priority:    Medium

Fix by https://github.com/apache/incubator-nuttx/pull/6083

repo: apache/incubator-nuttx


text:
sched_note:Allow for external endpoints

Summary
Fixes breakage introduced by https://github.com/apache/incubator-nuttx/pull/1520
Breaks build that used the board supported interfaces.
This PR restore the ability to not use ONLY the transport and keeps the contract that is stated in Kconfig,
Impact
None on current development,
Fixes Legacy usage.
Testing
Build with and without option.

CONFIG_SCHED_INSTRUMENTATION_EXTENAL is misspelled (missing R in EXTERNAL)

Oh, you are right @hartmannathan , @davids5 could you provide new PR to fix this issue.

@hartmannathan @xiaoxiang781216  done here https://github.com/apache/incubator-nuttx/pull/1749


text:
add header x-namespace-id

Closes #139

Codecov Report

Merging #140 into master will decrease coverage by 0.25%.
The diff coverage is 50%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
- Coverage   99.23%   98.98%   -0.26%
==========================================
  Files          14       14
  Lines         391      393       +2
==========================================
+ Hits          388      389       +1
- Misses          3        4       +1

Impacted Files
Coverage Δ

lib/client.js
97.5% <50%> (-1.22%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b9e6ce...b0f33b1. Read the comment docs.

Code LGTM to me
Needs docs and tests?

don't think we need docs, no changes to interface.
yep, I will add a test

@jthomas this should be ready for merge now.

authors: codecov-io csantanapr jthomas

repo: apache/incubator-openwhisk-client-js


text:
Add third_party/base64 & third_party/brotli

Signed-off-by: Otto van der Schaaf oschaaf@we-amp.com

This may need a little bit of work later on when we want to use these dependencies to get the include paths right when the dependencies are put to use. But I feel this is a first good step

repo: apache/incubator-pagespeed-mod


text:
[Bug] file.compression do no work for parquet format

Search before asking

[X] I searched in the issues and found nothing similar.

Paimon version
master
Compute Engine
parquet format
Minimal reproduce step

In parquet format, it will use compress key to extract from the option, and it has the higher preference than from the file.compression. So the actual work key is compress
What doesn't meet your expectations?
the actual compression algorithm
Anything else?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

I test it in my computer,it is ok, could you provide more information?
Flink SQL> create table t6(id int , name string) with ('file.format'='parquet','file.compression'='ZSTD');
[INFO] Execute statement succeed.

Flink SQL> insert into t6 select 6,'bbb';
[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: 4a4712de0ecf72630ba801aad35702cf

Flink SQL> select * from t6;
+----+------+
| id | name |
+----+------+
|  6 |  bbb |
+----+------+
1 row in set

Flink SQL>

Hi @zhangjun0x01 , have you verified that the 'file.compression'='ZSTD' has actually worked ?

@zhangjun0x01  I just push a fix for this, could you help take a look ?

@Aitozi This issue should be "parquet.compression not work"?

parquet.compression can work, file.compression not work.
The option in configuration is file.compression. I think parquet.compression is not the recommended way now?

authors: Aitozi JingsongLi zhangjun0x01

repo: apache/incubator-paimon


text:
fix: coredump when table name contains '_' and prometheus is enabled

What problem does this PR solve?
After we enable prometheus(set "perf_counter_sink" to "prometheus"), pegasus ended up core dumped after restart.
We debuged the core, found out it was cored while analysing table name that contains "_".
This PR fixes the bug by changing the order of formating table names.
What is changed and how does it work?
Prometheus does not support some special charactor in metric name.
This original code first formated the table name by replacing special charactor to "" or "@".
Then, it try to parse gpid as prometheus labels by spliting the foramted name by ":" or "".
Unfortunately, tables that already contains "_" may results in unexpected group result, which eventually ended up core.
Checklist
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to be included in the release note

As this picture shows below, when core dumps it was trying to analyze metric naming replica_econ_replica_table_level_RPC_RRDB_RRDB_REMOVE_latency_ns_:impala_historical_profile_p999, where impala_historical_profile is a table we created earlier that contains multiple underscore.

As the code shown below on line 256, it tries to split metrics_name by : first, then split the last part by _ and put the splited tokens in an array that length is 3. Because this table already contains underscore, there are more than 3 tokens, so it got core dumped after at line 259 trying to set the fourth element of the array.

This patch fix it by switching these format order:
Firstly, it try to parse gpid as prometheus labels by spliting the foramted name by : or @.
Then, it  formated the table name by replacing special charactor to _ or "@".
In this order tables that contains _ can be parsed expectedly in the first step.

repo: apache/incubator-pegasus


text:
put license and notice in all jars

I updated the NOTICE to match what we have in main pekko repo
From my understanding, we do not need to include HPack license - or even include details in our NOTICE file

it is Apache licensed
https://github.com/twitter/hpack has no NOTICE so we don't have copy that non-existent notice into our notice

From my understanding, we do not need to include HPack license - or even include details in our NOTICE file
* it is Apache licensed

* [twitter/hpack](https://github.com/twitter/hpack) has no NOTICE so we don't have copy that non-existent notice into our notice

Not sure I fully agree.

We should note somewhere that we included code and the notice file seems a good place
Since we are adapting akka-http we still have to keep akka-http's NOTICE file (based on the Apache License terms), so we cannot just delete it because we think the information is not useful to keep...

authors: jrudolph pjfanning

repo: apache/incubator-pekko-http


text:
use generic snapshot link for snapshot scaladoc/javadoc

relates to #213

Would it be possible to do this properly and use projectInfoVersion so that all versions referenced in docs are properly aligned, i.e. what was done in this PR https://github.com/apache/incubator-pekko/pull/392/files#diff-82e8a219706e01e038685e70db53a3e5c7de3d6648730ca5b632f91004bb0154R136 ?
In other words, set projectInfoVersion to if (!isSnapshot.value) "current" else "snapshot" and then use projectInfoVersion elsewhere for doc links since projectInfoVersion is also used within paradox/project-info.conf

I've made a change like the one you suggested


text:
Delete extra refreshed segments after segment push

Adds a config to make possible deleting extra segments that are pushed within the same configured time segment or refreshed
Future work: deal with invalid prefixes (when people accidentally change their prefix, when they keep a huge amount of time in the same segment for append segments, etc)
#4353

Codecov Report

Merging #4464 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #4464      +/-   ##
============================================
- Coverage     65.65%   65.61%   -0.04%
  Complexity       20       20
============================================
  Files          1065     1065
  Lines         55250    55250
  Branches       8025     8025
============================================
- Hits          36276    36254      -22
- Misses        16359    16387      +28
+ Partials       2615     2609       -6

Impacted Files
Coverage Δ
Complexity Δ

...ller/validation/OfflineSegmentIntervalChecker.java
71.79% <0%> (-12.83%)
0% <0%> (ø)

...he/pinot/controller/util/SegmentIntervalUtils.java
0% <0%> (-7.15%)
0% <0%> (ø)

...impl/dictionary/DoubleOnHeapMutableDictionary.java
69.56% <0%> (-6.53%)
0% <0%> (ø)

...e/impl/dictionary/LongOnHeapMutableDictionary.java
89.13% <0%> (-6.53%)
0% <0%> (ø)

...e/operator/dociditerators/SVScanDocIdIterator.java
67.85% <0%> (-3.58%)
0% <0%> (ø)

...e/operator/dociditerators/MVScanDocIdIterator.java
60.6% <0%> (-3.04%)
0% <0%> (ø)

.../org/apache/pinot/transport/netty/NettyServer.java
80.8% <0%> (-3.04%)
0% <0%> (ø)

...not/broker/broker/helix/ClusterChangeMediator.java
80.21% <0%> (-2.2%)
0% <0%> (ø)

...ata/manager/realtime/RealtimeTableDataManager.java
41.17% <0%> (-1.48%)
0% <0%> (ø)

...ot/core/query/pruner/ColumnValueSegmentPruner.java
86.3% <0%> (-1.37%)
0% <0%> (ø)

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 551b5c3...8ec059a. Read the comment docs.

Your comment seems to have disappeared. What would you like me to remove?
As the preconditions was added in your commit for multiple push and is tied with your feature, I think you probably have a better idea regarding all the edge cases you require this preconditions check, etc. It may be better for you to call out cases where that preconditions will be required as you merge subsequent prs for your feature.

authors: codecov-io jenniferdai

repo: apache/incubator-pinot