text:
Dubbo服务合并第一次请求数据不准确

使用dubbo-test-examples/src/main/java/com/alibaba/dubbo/examples/merge测试发现。Consumer第一次请求时数据未合并，第二次请求开始数据被合并。
(0) group-2.1, group-2.2 group-3.1, group-3.2, group-1.1, group-1.2, group-2.1, group-2.2 group-3.1, group-3.2, group-1.1, group-1.2, group-2.1, group-2.2 group-3.1, group-3.2, group-1.1, group-1.2, group-2.1, group-2.2 group-3.1, group-3.2, group-1.1, group-1.2, group-2.1, group-2.2 [group-3.1, group-3.2, group-1.1, group-1.2, group-2.1, group-2.2]

The dubbo-test-examples/src/main/java/com/alibaba/dubbo/examples/merge is not working as expected.
I have added merger=true in merge-consumer.xml and merge-consumer2.xml. Now it is working as expected. The data will be merged from the first invocation.
(0) [group-1.1, group-1.2, group-3.1, group-3.2, group-2.1, group-2.2]
(1) [group-1.1, group-1.2, group-3.1, group-3.2, group-2.1, group-2.2]
(2) [group-1.1, group-1.2, group-3.1, group-3.2, group-2.1, group-2.2]
(3) [group-1.1, group-1.2, group-3.1, group-3.2, group-2.1, group-2.2]
(4) [group-1.1, group-1.2, group-3.1, group-3.2, group-2.1, group-2.2]
(5) [group-1.1, group-1.2, group-3.1, group-3.2, group-2.1, group-2.2]

Your issue should be fixed, please check out the latest code.


text:
[Bug] Invite for Slack channel seems to be no longer valid

Search before asking

[X] I had searched in the issues and found no similar issues.

Environment
Mac
EventMesh version
master
What happened
Invite for Slack channel appears to be invalid in [README.md] file (https://github.com/apache/incubator-eventmesh#readme).

How to reproduce
NA
Debug logs
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

I will update the invite link.

@iamkarthikselvaraj  here is the link https://join.slack.com/t/apacheeventmesh/shared_invite/zt-1hgto0wcm-KXKz8pJGUwrWdk00juaQSw

source: gharchive/issue

authors: iamkarthikselvaraj qqeasonchen

repo: apache/incubator-eventmesh

license_type: permissive

license_source: github-api


text:
[Enhancement] Possible null pointer dereference due to return value of called method  [ClientSessionGroupMapping]

Search before asking

[X] I had searched in the issues and found no similar issues.

Enhancement Request

located at:
org/apache/eventmesh/runtime/core/protocol/tcp/client/group/ClientSessionGroupMapping.java line 250 269 275 286-288 300 309 301 329 330 335-337 340-342 347 348 351 352 358 359
analysis and explanation:
The return value from a method is dereferenced without a null check, and the return value of that method is one that should generally be checked for null. This may lead to a NullPointerException when the code is executed.
Describe the solution you'd like
add NPE exception check.
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

@xwm1992 I think I'm capable of fixing this issue, would u pls assign this to me？

I would like to contribute. Can you please assign me?

authors: Alonexc NewtonVan Yamilini2000


text:
Besides Centos and Ubuntu systems, does the gluten project support the Maipo system？

Description

As shown in the figure, my Linux machine is a Maipo system. Does the Gluten project support running on it? I am using version 1.1.1 gluten and encountered the 'invalid pointer' error

See https://github.com/apache/incubator-gluten/issues/4935#issuecomment-1992859638.

authors: PHILO-HE bigmancomeon

repo: apache/incubator-gluten


text:
Example to illustrate how to build custom source and extractor in Gobblin.

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[ ] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-XXX

Description

[ ] Here are some details about my PR, including screenshots (if applicable):

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Created a JIRA for this PR.
https://issues.apache.org/jira/browse/GOBBLIN-339

source: gharchive/pull-request

repo: apache/incubator-gobblin


text:
Shutdown hook was not called using spark on Kubernetes

HoodieDeltaStreamer would not exit when running spark on k8s or spark on yarn
Should manual call jssc.stop() at the end of sync method to fix it?

could you obtain a thread dump to see where its hanging?  vungle deploys hudi on k8s. have not heard this issue before.. thoughts?

@eisig : Any updates on this issue ?

I just call  jssc.stop() right now.
It always happen event with the office spark example.
Build the image following the office docs(https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images)
Run the simple JavaSparkPi( https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/JavaSparkPi.java).
If I comment out the 'spark.stop();`  The driver and executor pods  keep Running.
spark version is 2.4.3.
Here is the thread dump:
Full thread dump OpenJDK 64-Bit Server VM (25.212-b04 mixed mode):

"Attach Listener" #71 daemon prio=9 os_prio=0 tid=0x000055bb1ad87800 nid=0x7a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"DestroyJavaVM" #61 prio=5 os_prio=0 tid=0x000055bb17ca0800 nid=0x19 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"task-result-getter-1" #60 daemon prio=5 os_prio=0 tid=0x000055bb17fd0800 nid=0x55 waiting on condition [0x00007f5fd52d8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5a71380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"task-result-getter-0" #59 daemon prio=5 os_prio=0 tid=0x000055bb1a5ce000 nid=0x54 waiting on condition [0x00007f5fd53d9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5a71380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"shuffle-server-5-2" #58 daemon prio=5 os_prio=0 tid=0x000055bb18e03800 nid=0x53 runnable [0x00007f5fd66db000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000d602c1b0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000d60311a8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000d602c168> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

"spark-listener-group-shared" #57 daemon prio=5 os_prio=0 tid=0x000055bb1907c000 nid=0x52 waiting on condition [0x00007f5fd6be9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d87c42e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

"threadDeathWatcher-6-1" #56 daemon prio=1 os_prio=0 tid=0x000055bb1aeb9000 nid=0x51 waiting on condition [0x00007f5fd8eec000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

"rpc-server-3-2" #55 daemon prio=5 os_prio=0 tid=0x000055bb17602800 nid=0x50 runnable [0x00007f5fddaac000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000d590d4c0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000d5916758> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000d590d478> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

"element-tracking-store-worker" #54 daemon prio=5 os_prio=0 tid=0x000055bb19ae8800 nid=0x4f waiting on condition [0x00007f5fd8fed000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d586a5e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"spark-listener-group-eventLog" #51 daemon prio=5 os_prio=0 tid=0x000055bb17e4b000 nid=0x4e waiting on condition [0x00007f5fd90ee000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d63eeec0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

"spark-listener-group-executorManagement" #34 daemon prio=5 os_prio=0 tid=0x000055bb17e4a000 nid=0x4d waiting on condition [0x00007f5fd91ef000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5a62f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

"spark-listener-group-appStatus" #13 daemon prio=5 os_prio=0 tid=0x000055bb17e48000 nid=0x4c waiting on condition [0x00007f5fd92f0000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5870cd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply$mcJ$sp(AsyncEventQueue.scala:97)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:87)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
	at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:87)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:83)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:82)

"context-cleaner-periodic-gc" #53 daemon prio=5 os_prio=0 tid=0x000055bb18ad1800 nid=0x4b waiting on condition [0x00007f5fd93f1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d63ef248> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"Spark Context Cleaner" #52 daemon prio=5 os_prio=0 tid=0x000055bb18ad7800 nid=0x4a in Object.wait() [0x00007f5fd9682000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x00000000d63ef4b0> (a java.lang.ref.ReferenceQueue$Lock)
	at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:181)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1302)
	at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
	at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)

"java-sdk-http-connection-reaper" #50 daemon prio=5 os_prio=0 tid=0x000055bb189cf000 nid=0x49 waiting on condition [0x00007f5fd9e4d000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:112)

"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #49 daemon prio=5 os_prio=0 tid=0x000055bb171b0800 nid=0x48 in Object.wait() [0x00007f5fd94f2000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x00000000d626bb10> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)
	at java.lang.Thread.run(Thread.java:748)

"shuffle-server-5-1" #48 daemon prio=5 os_prio=0 tid=0x000055bb16d3e800 nid=0x47 runnable [0x00007f5fda14e000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000d600cde0> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000d600cdf8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000d600cd98> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

"kubernetes-executor-pod-polling-sync" #47 daemon prio=5 os_prio=0 tid=0x000055bb16d38800 nid=0x46 waiting on condition [0x00007f5fda24f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5f590a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"OkHttp WebSocket https://kubernetes.default.svc/..." #46 prio=5 os_prio=0 tid=0x000055bb18d21800 nid=0x45 waiting on condition [0x00007f5fda350000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d600d218> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"OkHttp https://kubernetes.default.svc/..." #45 daemon prio=5 os_prio=0 tid=0x000055bb19bc5000 nid=0x44 runnable [0x00007f5fda450000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
	- locked <0x00000000d5fed308> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	- locked <0x00000000d5fed3c8> (a sun.security.ssl.AppInputStream)
	at okio.Okio$2.read(Okio.java:139)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	at okio.RealBufferedSource.request(RealBufferedSource.java:67)
	at okio.RealBufferedSource.require(RealBufferedSource.java:60)
	at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
	at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:262)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:201)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:141)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"kubernetes-executor-snapshots-subscribers-1" #44 daemon prio=5 os_prio=0 tid=0x000055bb170f4800 nid=0x43 waiting on condition [0x00007f5fda552000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5c40650> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"kubernetes-executor-snapshots-subscribers-0" #43 daemon prio=5 os_prio=0 tid=0x000055bb17939000 nid=0x42 waiting on condition [0x00007f5fda653000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5c40650> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"driver-revive-thread" #42 daemon prio=5 os_prio=0 tid=0x000055bb17936000 nid=0x41 waiting on condition [0x00007f5fda754000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5f21c98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"dag-scheduler-event-loop" #41 daemon prio=5 os_prio=0 tid=0x000055bb19274000 nid=0x40 waiting on condition [0x00007f5fda855000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5fefc80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:47)

"Okio Watchdog" #40 daemon prio=5 os_prio=0 tid=0x000055bb183c8800 nid=0x3f in Object.wait() [0x00007f5fdab56000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:460)
	at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:361)
	at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:312)
	- locked <0x00000000d5c9f358> (a java.lang.Class for okio.AsyncTimeout)

"OkHttp ConnectionPool" #39 daemon prio=5 os_prio=0 tid=0x000055bb17b6b800 nid=0x3e in Object.wait() [0x00007f5fdac57000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:460)
	at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
	- locked <0x00000000d5a96810> (a okhttp3.ConnectionPool)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"netty-rpc-env-timeout" #38 daemon prio=5 os_prio=0 tid=0x000055bb1895c000 nid=0x3d waiting on condition [0x00007f5fdb611000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d59029f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"Timer-1" #37 daemon prio=5 os_prio=0 tid=0x000055bb18948000 nid=0x3c in Object.wait() [0x00007f5fdb712000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000d5a67b98> (a java.util.TaskQueue)
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	- locked <0x00000000d5a67b98> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"Timer-0" #36 daemon prio=5 os_prio=0 tid=0x000055bb18c12000 nid=0x3b in Object.wait() [0x00007f5fdb813000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	- locked <0x00000000d5a56f00> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"heartbeat-receiver-event-loop-thread" #35 daemon prio=5 os_prio=0 tid=0x000055bb18d9f800 nid=0x3a waiting on condition [0x00007f5fdb914000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5a57090> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-33" #33 daemon prio=5 os_prio=0 tid=0x000055bb18f30800 nid=0x39 waiting on condition [0x00007f5fdbdee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d59abd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-32" #32 daemon prio=5 os_prio=0 tid=0x000055bb1926e800 nid=0x38 waiting on condition [0x00007f5fdbeef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d59abd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-31" #31 daemon prio=5 os_prio=0 tid=0x000055bb1926c000 nid=0x37 waiting on condition [0x00007f5fdbff0000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d59abd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-30" #30 daemon prio=5 os_prio=0 tid=0x000055bb1926a000 nid=0x36 waiting on condition [0x00007f5fdc0f1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d59abd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-29" #29 daemon prio=5 os_prio=0 tid=0x000055bb19268000 nid=0x35 waiting on condition [0x00007f5fdc1f2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d59abd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-28" #28 daemon prio=5 os_prio=0 tid=0x000055bb18de4000 nid=0x34 waiting on condition [0x00007f5fdc2f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d59abd80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at org.spark_project.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-27-acceptor-0@64020831-ServerConnector@6ac5fac1{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}" #27 daemon prio=3 os_prio=0 tid=0x000055bb18de2800 nid=0x33 runnable [0x00007f5fdc3f4000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
	- locked <0x00000000d59ac890> (a java.lang.Object)
	at org.spark_project.jetty.server.ServerConnector.accept(ServerConnector.java:397)
	at org.spark_project.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

"SparkUI-26" #26 daemon prio=5 os_prio=0 tid=0x000055bb18dcc000 nid=0x32 runnable [0x00007f5fdc4f4000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000d59acb78> (a sun.nio.ch.Util$3)
	- locked <0x00000000d59acb88> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000d59acb30> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
	at org.spark_project.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
	at org.spark_project.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

"RemoteBlock-temp-file-clean-thread" #25 daemon prio=5 os_prio=0 tid=0x000055bb17082000 nid=0x31 in Object.wait() [0x00007f5fdc7f6000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x00000000d594eaf0> (a java.lang.ref.ReferenceQueue$Lock)
	at org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager.org$apache$spark$storage$BlockManager$RemoteBlockDownloadFileManager$$keepCleaning(BlockManager.scala:1724)
	at org.apache.spark.storage.BlockManager$RemoteBlockDownloadFileManager$$anon$2.run(BlockManager.scala:1692)

"map-output-dispatcher-7" #24 daemon prio=5 os_prio=0 tid=0x000055bb1702d000 nid=0x30 waiting on condition [0x00007f5fdcaf7000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"map-output-dispatcher-6" #23 daemon prio=5 os_prio=0 tid=0x000055bb188f6000 nid=0x2f waiting on condition [0x00007f5fdcbf8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"map-output-dispatcher-5" #22 daemon prio=5 os_prio=0 tid=0x000055bb188f4000 nid=0x2e waiting on condition [0x00007f5fdccf9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"map-output-dispatcher-4" #21 daemon prio=5 os_prio=0 tid=0x000055bb188d5800 nid=0x2d waiting on condition [0x00007f5fdcdfa000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"map-output-dispatcher-3" #20 daemon prio=5 os_prio=0 tid=0x000055bb188d3800 nid=0x2c waiting on condition [0x00007f5fdcefb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"map-output-dispatcher-2" #19 daemon prio=5 os_prio=0 tid=0x000055bb186fe800 nid=0x2b waiting on condition [0x00007f5fdcffc000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"map-output-dispatcher-1" #18 daemon prio=5 os_prio=0 tid=0x000055bb16f8a000 nid=0x2a waiting on condition [0x00007f5fdd0fd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"map-output-dispatcher-0" #17 daemon prio=5 os_prio=0 tid=0x000055bb16f89000 nid=0x29 waiting on condition [0x00007f5fdd1fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d592fd88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:384)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"rpc-server-3-1" #16 daemon prio=5 os_prio=0 tid=0x000055bb17185000 nid=0x28 runnable [0x00007f5fdd2ff000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000000d58e4530> (a io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000d58e4548> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000d58e44e8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:748)

"dispatcher-event-loop-1" #15 daemon prio=5 os_prio=0 tid=0x000055bb187a6000 nid=0x27 waiting on condition [0x00007f5fdd600000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5887810> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"dispatcher-event-loop-0" #14 daemon prio=5 os_prio=0 tid=0x000055bb187a5000 nid=0x26 waiting on condition [0x00007f5fdd701000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d5887810> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:215)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x000055bb16d58800 nid=0x20 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x000055bb16cf3000 nid=0x1f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x000055bb16ceb800 nid=0x1e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x000055bb16ce9000 nid=0x1d runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x000055bb16caf000 nid=0x1c in Object.wait() [0x00007f5fe2a39000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x00000000d5569b20> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x000055bb16cac000 nid=0x1b in Object.wait() [0x00007f5fe2b3a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x00000000d5569cd8> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"VM Thread" os_prio=0 tid=0x000055bb16ca2000 nid=0x1a runnable

"VM Periodic Task Thread" os_prio=0 tid=0x000055bb16d5b000 nid=0x21 waiting on condition

JNI global references: 296

Thanks for testing it out.. In that case, may be we can stop the spark context in this line in the main method?
https://github.com/apache/incubator-hudi/blob/a7e6cf51974f96c6b4bc945db25658cbb4bc48c7/hoodie-utilities/src/main/java/com/uber/hoodie/utilities/deltastreamer/HoodieDeltaStreamer.java#L469
are you interested in opening a small PR?

I have another problem. If I add --filter-dupes, even if I stop the spark context, the driver will not exit after the executors exit. So I need to test more.

@vinothchandar , If I add --filter-dupes, the spark job could not exit.
It could be reproduced by  following  https://hudi.apache.org/docker_demo.html with the  master branch。
add --filter-dupes to step2

That is weird. let me repro and get back.

@eisig : We found the issue. It is caused by not properly shutting down a client which is opened in the filter-dupes path.
Have a PR https://github.com/apache/incubator-hudi/pull/725
We will merge it hopefully soon but meanwhile you can use this PR and try.

This is merged.

authors: bvaradar eisig vinothchandar

repo: apache/incubator-hudi


text:
[INLONG-4487][Manager] Fix json parse exception for streamSource/streamSink

Prepare a Pull Request
(Change the title refer to the following example)

Title Example: [INLONG-4487][Manager] Fix json parse exception for streamSource/streamSink

(The following XYZ should be replaced by the actual GitHub Issue number)

Fixes #4487

Motivation
Modifications
Verifying this change
(Please pick either of the following options)

[ ] This change is a trivial rework/code cleanup without any test coverage.

[ ] This change is already covered by existing tests, such as:
(please describe tests)

[ ] This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

debugging branch, close

repo: apache/incubator-inlong


text:
[KYUUBI #1829] Support executing statement async for Flink SQL engine

Why are the changes needed?
Support executing statement async for Flink SQL engine.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #1845 (d9c2741) into master (53d59a0) will increase coverage by 0.24%.
The diff coverage is 100.00%.

:exclamation: Current head d9c2741 differs from pull request most recent head 616ddd2. Consider uploading reports for the commit 616ddd2 to get more accurate results

@@             Coverage Diff              @@
##             master    #1845      +/-   ##
============================================
+ Coverage     59.46%   59.71%   +0.24%
+ Complexity      272      271       -1
============================================
  Files           277      282       +5
  Lines         13853    13996     +143
  Branches       1776     1785       +9
============================================
+ Hits           8238     8358     +120
- Misses         4911     4927      +16
- Partials        704      711       +7

Impacted Files
Coverage Δ

...uubi/engine/flink/operation/ExecuteStatement.scala
79.04% <100.00%> (+1.49%)
:arrow_up:

...e/authentication/KyuubiAuthenticationFactory.scala
75.86% <0.00%> (-2.32%)
:arrow_down:

...rg/apache/kyuubi/engine/trino/TrinoStatement.scala
64.70% <0.00%> (-1.18%)
:arrow_down:

...apache/kyuubi/service/TBinaryFrontendService.scala
85.10% <0.00%> (-0.32%)
:arrow_down:

.../authentication/EngineSecuritySecretProvider.scala
100.00% <0.00%> (ø)

...uthentication/KyuubiInternalAccessIdentifier.scala
100.00% <0.00%> (ø)

...on/ZooKeeperEngineSecuritySecretProviderImpl.scala
0.00% <0.00%> (ø)

...ervice/authentication/EngineSecurityAccessor.scala
82.00% <0.00%> (ø)

...ation/EngineSecureAuthenticationProviderImpl.scala
0.00% <0.00%> (ø)

...a/org/apache/kyuubi/service/TFrontendService.scala
94.75% <0.00%> (+0.01%)
:arrow_up:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53d59a0...616ddd2. Read the comment docs.

@pan3793 @yanghua @yaooqinn , could you please help to review this pull request?

authors: SteNicholas codecov-commenter

repo: apache/incubator-kyuubi


text:
[KYUUBI #3052][FOLLOWUP] Do not use the ip in proxy http header for authentication to prevent CVE

Why are the changes needed?

Because the client can specify any ip in http header, to prevent CVE issue, we do not use it for authentication.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #3078 (9006a85) into master (ce72a50) will increase coverage by 0.03%.
The diff coverage is 90.32%.

@@             Coverage Diff              @@
##             master    #3078      +/-   ##
============================================
+ Coverage     51.31%   51.34%   +0.03%
  Complexity        6        6
============================================
  Files           458      458
  Lines         25375    25392      +17
  Branches       3540     3541       +1
============================================
+ Hits          13020    13038      +18
+ Misses        11114    11111       -3
- Partials       1241     1243       +2

Impacted Files
Coverage Δ

...apache/kyuubi/engine/JpsApplicationOperation.scala
79.41% <50.00%> (-4.46%)
:arrow_down:

...ver/http/authentication/AuthenticationFilter.scala
90.76% <85.71%> (-0.90%)
:arrow_down:

...in/scala/org/apache/kyuubi/config/KyuubiConf.scala
97.25% <100.00%> (-0.10%)
:arrow_down:

...ache/kyuubi/server/KyuubiRestFrontendService.scala
80.95% <100.00%> (+0.70%)
:arrow_up:

...che/kyuubi/server/KyuubiTHttpFrontendService.scala
64.23% <100.00%> (+0.26%)
:arrow_up:

...rg/apache/kyuubi/server/api/v1/AdminResource.scala
90.90% <100.00%> (ø)

.../apache/kyuubi/server/api/v1/BatchesResource.scala
69.74% <100.00%> (ø)

...apache/kyuubi/server/api/v1/SessionsResource.scala
71.42% <100.00%> (ø)

.../apache/kyuubi/server/http/ThriftHttpServlet.scala
70.22% <100.00%> (+0.46%)
:arrow_up:

...pache/kyuubi/server/http/util/SessionManager.scala
76.00% <100.00%> (+4.57%)
:arrow_up:

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0857786...9006a85. Read the comment docs.

thanks, merged to master

authors: codecov-commenter turboFei


text:
[KYUUBI #821] Improper exception for closing a non-existent session

Why are the changes needed?

ROOT CAUSE
When server receive request of close session, will call FrontendService#CloseSession. When an error is encountered when closing the session, CURRENT_SERVER_CONTEXT will never be called.
2021-08-02 09:41:50.945 INFO service.FrontendService: Received request of closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.946 INFO session.KyuubiSessionManager: SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72] is closed, current opening sessions 0
2021-08-02 09:41:50.949 WARN service.FrontendService: Error closing session:
org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources
	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
	at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:89)
	at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:74)
	at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
	at org.apache.kyuubi.service.FrontendService.CloseSession(FrontendService.scala:218)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1397)
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1382)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
	at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
	at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
	at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
	at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:191)
	at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:178)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$1(KyuubiSyncThriftClient.scala:67)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:47)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:67)
	at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:86)
	... 12 more
2021-08-02 09:41:50.953 INFO service.FrontendService: Finished closing SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
2021-08-02 09:41:50.957 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.963 INFO service.FrontendService: Session [SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]] disconnected without closing properly, close it now
2021-08-02 09:41:50.964 WARN service.FrontendService: Failed closing session
org.apache.kyuubi.KyuubiSQLException: Invalid SessionHandle [f3545c54-2e94-4090-aaa6-4397e653bb72]
	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
	at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:71)
	at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
	at org.apache.kyuubi.service.FrontendService$FeTServerEventHandler.deleteContext(FrontendService.scala:529)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-08-02 09:42:30.908 INFO server.ZooKeeperServer: Expiring session 0x1000da7829f0002, timeout of 60000ms exceeded
2021-08-02 09:42:30.909 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x1000da7829f0002

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Codecov Report

Merging #879 (f485f58) into master (6c3b722) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #879      +/-   ##
============================================
- Coverage     78.79%   78.77%   -0.02%
  Complexity       10       10
============================================
  Files           137      137
  Lines          5187     5187
  Branches        644      644
============================================
- Hits           4087     4086       -1
  Misses          747      747
- Partials        353      354       +1

Impacted Files
Coverage Δ

...la/org/apache/kyuubi/service/FrontendService.scala
97.00% <100.00%> (ø)

...pache/kyuubi/sql/KyuubiQueryStagePreparation.scala
79.41% <0.00%> (-0.99%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c3b722...f485f58. Read the comment docs.

LGTM

Thanks, merging to master for v1.3.0


text:
How does linkis support Kerberos

Search before asking

[X] I searched the issues and found no similar issues.

Linkis Component
linkis-commons
What happened + What you expected to happen
How does linkis support Kerberos
Relevent platform
。
Reproduction script
。
Anything else
This question comes from the QA documentation of the Linkis community
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Solution:
The file system of Hadoop obtained in linkis is implemented through hdfsutils class, so we put Kerberos in this class. Users can see the logic of this class. The login modes supported now are as follows:
if(KERBEROS_ENABLE.getValue){
val path =newFile(KEYTAB_FILE.getValue ， userName +".keytab").getPath
val user =getKerberosUser(userName)
UserGroupInformation.setConfiguration(getConfiguration(userName))
UserGroupInformation.loginUserFromKeytabAndReturnUGI(user， path)
}else{
UserGroupInformation.createRemoteUser(userName)
}
wds.linkis.keytab.enable=true
wds.linkis.keytab.file=/appcom/keytab/ #keytabPlace a directory that stores the of multiple usersusername.keytabDocuments
wds.linkis.keytab.host.enabled=false #Whether the client is authenticated with the principal
wds.linkis.keytab.host=127.0.0.1#Principle authentication needs to bring the client IP

repo: apache/incubator-linkis


text:
Check failed: !ndinputs.back()->is_none()

I came across with an error of Check failed: !ndinputs.back()->is_none() node_58 0. The full code is too complicated to post here. I am trying to get a minimum sample to reproduce this error. Briefly, the model is the input variable going through a gluon Block, and some ndarray ops are used to calculate the loss.
Something I have tried:

I tried to call hybridize() for the Block, and the error disappears.
It seems to be related to the function below when calculating the loss:

    def binarize(self, x, i):
        rand = mx.nd.random_uniform(low=0, high=1,
                                        shape=self.rand_shape[i],
                                        ctx=self.model.ctx)
        sample = mx.nd.broadcast_lesser_equal(rand, mx.nd.sigmoid(x))
        sample_shape = [-1] + list(self.model.data_shape[i])[1:]
        return sample.reshape(sample_shape)

If I replace the function with
    def binarize(self, x, i):
        return x

The error disappears.
Besides, I found there is tojson() method for symbol, but not for ndarray. Is there a way to visualize the computation graph so that I can know what operation of node_58 is?
Any help will be appreciated.

Here is a minimum sample to reproduce the error

import mxnet as nd
from mxnet import nd, sym, autograd
from mxnet import gluon as g

inp = nd.ones((20, 10))
class Test(g.nn.HybridBlock):
    def __init__(self):
        super(Test, self).__init__()
        self.fc1 = g.nn.Dense(100)
        self.fc2 = g.nn.Dense(20)

    def hybrid_forward(self, F, x):
        A = self.fc1(x)
        X = self.fc2(x)
        A = A.reshape((-1, 10, 10))
        X = X.reshape((-1, 10, 2))
        return A, X

test = Test()
#test.hybridize()
test.initialize()
with autograd.record():
    A, X = test(inp)
    A = 0.5 <= A
    X = 0.5 <= X
    s = nd.batch_dot(A, X)
    l = s.sum()

l.backward()

As stated earlier, calling hybridize() will make the error disappear.

repo: apache/incubator-mxnet


text:
enabling mkldnn leads to segfault in bytePS

I am using bytepsimage/mxnet docker image to test bytePS. However, I find recent MXNet versions lead to segfault.
Specifically, if i build mxnet-cu100 variant with commit b6b1de092b2bbc6ab7207a98dcb1c08fe67ca14b, the following command works:
docker pull bytepsimage/mxnet

nvidia-docker run -it --net=host --shm-size=32768m bytepsimage/mxnet bash

# now you are in docker environment
export NVIDIA_VISIBLE_DEVICES=0,1,2,3  # gpus list
export DMLC_WORKER_ID=0 # your worker id
export DMLC_NUM_WORKER=1 # one worker
export DMLC_ROLE=worker

# the following value does not matter for non-distributed jobs
export DMLC_NUM_SERVER=1
export DMLC_PS_ROOT_URI=10.0.0.1
export DMLC_PS_ROOT_PORT=1234

bpslaunch python3 /usr/local/byteps/example/mxnet/train_gluon_mnist_byteps.py

However, if i use commit 2f6cdd383abbf46a37b84a5fad013726b5c62169, it gives me segfault.
I used source tools/staticbuild/build.sh mxnet-cu100 pip to build the pip package.
And if I use the latest nightly build, it also gives me segfault
@TaoLv any idea why?
related issue https://github.com/bytedance/byteps/issues/222

Is there any test case for bytePS to cover this kind of issue?

bytePS CI only test stable mxnet, not mxnet nightly.
I assume bytepsimage/mxnet uses official mxnet - @ymjiang please correct me if i'm wrong

Yes, bytepsimage/mxnet uses official mxnet, and not nightly version: https://github.com/bytedance/byteps/blob/master/docker/Dockerfile#L46

@eric-haibin-lin @ymjiang Sorry, I have no idea because the back trace doesn't provide useful information. Can you try to disable MXNET_MKLDNN_ENABLED before running the model?

I also tried to build both mxnet and bytePS from source, with gcc 7.4, and the following is the stacktrace:
BytePS launching worker
warning: Error disabling address space randomization: Operation not permitted
--------------------------------------------------------------------------------
        Layer (type)                                Output Shape         Param #
================================================================================
               Input                              (1, 1, 28, 28)               0
        Activation-1   <Symbol hybridsequential0_conv0_relu_fwd>               0
        Activation-2                             (1, 20, 24, 24)               0
            Conv2D-3                             (1, 20, 24, 24)             520
         MaxPool2D-4                             (1, 20, 12, 12)               0
        Activation-5   <Symbol hybridsequential0_conv1_relu_fwd>               0
        Activation-6                               (1, 50, 8, 8)               0
            Conv2D-7                               (1, 50, 8, 8)           25050
         MaxPool2D-8                               (1, 50, 4, 4)               0
           Flatten-9                                    (1, 800)               0
       Activation-10  <Symbol hybridsequential0_dense0_relu_fwd>               0
       Activation-11                                    (1, 512)               0
            Dense-12                                    (1, 512)          410112
            Dense-13                                     (1, 10)            5130
================================================================================
Parameters in forward computation graph, duplicate included
   Total params: 440812
   Trainable params: 440812
   Non-trainable params: 0
Shared params in forward computation graph: 0
Unique parameters in model: 440812
--------------------------------------------------------------------------------
Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
__GI___pthread_mutex_lock (mutex=0x20) at ../nptl/pthread_mutex_lock.c:65
65      ../nptl/pthread_mutex_lock.c: No such file or directory.
#0  __GI___pthread_mutex_lock (mutex=0x20) at ../nptl/pthread_mutex_lock.c:65
#1  0x00007f283d52f170 in mxnet::engine::ThreadedVar::AppendWriteDependency(mxnet::engine::OprBlock*) () from /mxnet/python/mxnet/../../build/libmxnet.so
#2  0x00007f283d52ad3f in mxnet::engine::ThreadedEngine::Push(mxnet::engine::Opr*, mxnet::Context, int, bool) () from /mxnet/python/mxnet/../../build/libmxnet.so
#3  0x00007f283d527b85 in mxnet::engine::ThreadedEngine::PushAsync(std::function<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete)>, mxnet::Context, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, mxnet::FnProperty, int, char const*, bool) () from /mxnet/python/mxnet/../../build/libmxnet.so
#4  0x00007f283d3dbcb1 in MXEnginePushAsync () from /mxnet/python/mxnet/../../build/libmxnet.so
#5  0x00007f271d56618d in byteps::mxnet::byteps_mxnet_push_pull_async (tensor=0x45797570, name=<optimized out>, version=0, priority=0, is_average=<optimized out>) at byteps/mxnet/ops.cc:116
#6  0x00007f28f290edae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#7  0x00007f28f290e71f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#8  0x00007f28f2b225c4 in _ctypes_callproc () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
#9  0x00007f28f2b22c33 in ?? () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so

setting MXNET_MKLDNN_ENABLED=0 did not help.

@ymjiang have you seen this err before? Error disabling address space randomization: Operation not permitted

With @leezu 's help I built mxnet in debug mode. Here's the stacktrace with line number:
Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
__GI___pthread_mutex_lock (mutex=0x20) at ../nptl/pthread_mutex_lock.c:65
65      ../nptl/pthread_mutex_lock.c: No such file or directory.
#0  __GI___pthread_mutex_lock (mutex=0x20) at ../nptl/pthread_mutex_lock.c:65
#1  0x00007fff5314d857 in __gthread_mutex_lock (__mutex=0x20) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748
#2  0x00007fff5316b1c6 in std::mutex::lock (this=0x20) at /usr/include/c++/7/bits/std_mutex.h:103
#3  0x00007fff53182d14 in std::lock_guard<std::mutex>::lock_guard (this=0x7fffffffcb30, __m=...) at /usr/include/c++/7/bits/std_mutex.h:162
#4  0x00007fff53359852 in mxnet::engine::ThreadedVar::AppendWriteDependency (this=0x0, opr_block=0x294c2a8) at ../src/engine/threaded_engine.cc:74
#5  0x00007fff53355fd2 in mxnet::engine::ThreadedEngine::Push (this=0x2946590, op=0x294a708, exec_ctx=..., priority=0, profiling=false) at ../src/engine/threaded_engine.cc:311
#6  0x00007fff53356400 in mxnet::engine::ThreadedEngine::PushAsync(std::function<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete)>, mxnet::Context, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, mxnet::FnProperty, int, char const*, bool) (this=0x2946590, fn=..., exec_ctx=..., const_vars=std::vector of length 0, capacity 0, mutable_vars=std::vector of length 1, capacity 1 = {...}, prop=mxnet::FnProperty::kCPUPrioritized, priority=0, opr_name=0x7ffe356374ac "BytePSPushPull", wait=false) at ../src/engine/threaded_engine.cc:343
#7  0x00007fff53164592 in MXEnginePushAsync (async_func=0x7ffe355663e0 <byteps::mxnet::DoPushPull(void*, void*, void*)>, func_param=0x40f29520, deleter=0x7ffe35565a30 <byteps::mxnet::(anonymous namespace)::DeletePushPullParam(void*)>, ctx_handle=0x7ffe3bff8a40 <byteps::mxnet::(anonymous namespace)::MX_EXEC_CTX>, const_vars_handle=0x0, num_const_vars=0, mutable_vars_handle=0x7fffffffd168, num_mutable_vars=1, prop_handle=0x7ffe35637540 <byteps::mxnet::(anonymous namespace)::MX_FUNC_PROP>, priority=0, opr_name=0x7ffe356374ac "BytePSPushPull", wait=false) at ../src/c_api/c_api.cc:2482
#8  0x00007ffe3556618d in byteps::mxnet::byteps_mxnet_push_pull_async (tensor=0x409bf190, name=<optimized out>, version=0, priority=0, is_average=<optimized out>) at byteps/mxnet/ops.cc:116
#9  0x00007ffff65a6dae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#10 0x00007ffff65a671f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#11 0x00007ffff67ba5c4 in _ctypes_callproc () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
#12 0x00007ffff67bac33 in ?? () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so

Based on the backtrace, it appears that some mutex may not be initialized correctly leading to segfault?

another data point: mxnet-cu100 (1.6.0b20200212) works, but mxnet-cu100mkl (1.6.0b20200212) doesn't

https://repo.mxnet.io/dist/python/cu100/mxnet_cu100-1.6.0-py2.py3-none-manylinux1_x86_64.whl works,
https://repo.mxnet.io/dist/python/cu100mkl/mxnet_cu100mkl-1.6.0-py2.py3-none-manylinux1_x86_64.whl doesn't.
So, the issue has been there for a while ..

@ymjiang have you seen this err before? Error disabling address space randomization: Operation not permitted. It shows up when I used gdb though. might not be related to the original problem
Update: Error disabling address space randomization: Operation not permitted is resolved by --security-opt seccomp=unconfined. It does not help the segmentation fault problem

@eric-haibin-lin I am not aware of this before. Is there any version of MXNet-MKL that works? I suspect it might be due to some incompatibility between BytePS and MKL, if none of existing MKL works.

@eric-haibin-lin @ymjiang I still can't see anything related to MKL-DNN operators from the back trace. Agree that there might be some incompatibility issues between MKL-DNN integration and later BytePS integration.
BTW, have you ever tried NaiveEngine? Can w have some clues from there?

Build latest MXNet w/o MKLDNN also encounter this issue:
cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_MKL_IF_AVAILABLE=OFF -DUSE_CUDA=ON -DUSE_MKLDNN=OFF -G Ninja ..

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
__GI___pthread_mutex_lock (mutex=0x3a6772617f) at ../nptl/pthread_mutex_lock.c:65
65	../nptl/pthread_mutex_lock.c: No such file or directory.
#0  __GI___pthread_mutex_lock (mutex=0x3a6772617f) at ../nptl/pthread_mutex_lock.c:65
#1  0x00007fa43648b65b in __gthread_mutex_lock (__mutex=0x3a6772617f) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748
#2  0x00007fa4364adf3a in std::mutex::lock (this=0x3a6772617f) at /usr/include/c++/7/bits/std_mutex.h:103
#3  0x00007fa4364c5bf4 in std::lock_guard<std::mutex>::lock_guard (this=0x7ffd3340e270, __m=...) at /usr/include/c++/7/bits/std_mutex.h:162
#4  0x00007fa4366b2d6e in mxnet::engine::ThreadedVar::AppendWriteDependency (this=0x3a6772615f, opr_block=0x2f08190) at ../src/engine/threaded_engine.cc:74
#5  0x00007fa4366af4f7 in mxnet::engine::ThreadedEngine::Push (this=0x2f053a0, op=0x2f06630, exec_ctx=..., priority=0, profiling=false) at ../src/engine/threaded_engine.cc:311
#6  0x00007fa4366af924 in mxnet::engine::ThreadedEngine::PushAsync(std::function<void (mxnet::RunContext, mxnet::engine::CallbackOnComplete)>, mxnet::Context, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, mxnet::FnProperty, int, char const*, bool) (this=0x2f053a0, fn=..., exec_ctx=..., const_vars=std::vector of length 0, capacity 0, mutable_vars=std::vector of length 1, capacity 1 = {...}, prop=mxnet::FnProperty::kCPUPrioritized, priority=0, opr_name=0x7fa2b16372ec "BytePSPushPull", wait=false) at ../src/engine/threaded_engine.cc:343
#7  0x00007fa4364a72f6 in MXEnginePushAsync (async_func=0x7fa2b15659f0 <byteps::mxnet::DoPushPull(void*, void*, void*)>, func_param=0x6ee84170, deleter=0x7fa2b1565040 <byteps::mxnet::(anonymous namespace)::DeletePushPullParam(void*)>, ctx_handle=0x7fa2b7ff8a40 <byteps::mxnet::(anonymous namespace)::MX_EXEC_CTX>, const_vars_handle=0x0, num_const_vars=0, mutable_vars_handle=0x7ffd3340e8a8, num_mutable_vars=1, prop_handle=0x7fa2b1637380 <byteps::mxnet::(anonymous namespace)::MX_FUNC_PROP>, priority=0, opr_name=0x7fa2b16372ec "BytePSPushPull", wait=false) at ../src/c_api/c_api.cc:2665
#8  0x00007fa2b156579d in byteps::mxnet::byteps_mxnet_push_pull_async (tensor=0x6d41f620, name=<optimized out>, version=0, priority=0, is_average=<optimized out>) at byteps/mxnet/ops.cc:116
#9  0x00007fa50630fdae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#10 0x00007fa50630f71f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#11 0x00007fa5065235c4 in _ctypes_callproc () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
#12 0x00007fa506523c33 in ?? () from /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so

@xinyu-intel the pip build is built using make. Actually, I'm not sure if there's which version/commit built with cmake that works with BytePS..

@TaoLv The communication API uses the PushAsync api from engine, which unfortunately is not supported by the NaiveEngine..

@eric-haibin-lin I am not aware of this before. Is there any version of MXNet-MKL that works? I suspect it might be due to some incompatibility between BytePS and MKL, if none of existing MKL works.

@ymjiang that's possible.. I have not tested BytePS with mxnet-mkl before

@TaoLv @ciyongch @xinyu-intel @PatricZhao can someone look into why mxnet-cuxx-mkl build leads to the segfault? This may be a potential blocker for mxnet 1.7. If possible we should also patch it with the mxnet 1.6.1 release

Another thing is that even if I build mxnet 1.6.x from source using make as the following, running a simple push pull test also leads to segmentation fault:
cp make/config.mk .
echo "USE_BLAS=openblas" >>config.mk
echo "ADD_CFLAGS += -I/usr/include/openblas" >>config.mk
echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
echo "USE_CUDNN=0" >>config.mk
echo "USE_OPENCV=0" >>config.mk
make -j

I also tried to upgrade to gcc 9 (suggested by @leezu) but got error including cuda headers:
[April 19, 2020, 3:47 PM] Lin, Haibin: /usr/local/cuda/include/crt/host_config.h:129:2: error: #error -- unsupported GNU version! gcc versions later than 7 are not supported!
  129 | #error -- unsupported GNU version! gcc versions later than 7 are not supported!
      |  ^~~~~
In file included from /usr/local/cuda/include/cuda_runtime.h:83,
                 from <command-line>:
/usr/local/cuda/include/crt/host_config.h:129:2: error: #error -- unsupported GNU version! gcc versions later than 7 are not supported!
  129 | #error -- unsupported GNU version! gcc versions later than 7 are not supported!
      |  ^~~~~

For cuda 10.1, gcc8 is supported

@eric-haibin-lin based on the comments from @TaoLv @xinyu-intel, we don't see anything related to MKLDNN. I think bytePS developer should spend some efforts to locate the reason and get back with clear analysis.

We will investigate this from BytePS-side.

@eric-haibin-lin Could you try this PR? https://github.com/bytedance/byteps/pull/244

@xinyu-intel could the issue you report in https://github.com/apache/incubator-mxnet/issues/18014#issuecomment-612375492 be the same as https://github.com/apache/incubator-mxnet/issues/18090
BytePS side fixed some bugs that caused a different crash when using MXNet MKLDNN build, so the remaining issue now is only ThreadedEngine.

Another thing is that even if I build mxnet 1.6.x from source using make as the following (without MKLDNN), running a simple push pull test also leads to segmentation fault:
cp make/config.mk .
echo "USE_BLAS=openblas" >>config.mk
echo "ADD_CFLAGS += -I/usr/include/openblas" >>config.mk
echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
echo "USE_CUDNN=0" >>config.mk
echo "USE_OPENCV=0" >>config.mk
make -j

This PR https://github.com/bytedance/byteps/pull/244 also resolves the segfault issue when MXNet is built from source with MKLDNN=0.

l

authors: TaoLv eric-haibin-lin leezu pengzhao-intel xinyu-intel ymjiang


text:
The DataIter problem with symbol.list_arguments()

In a LSTM bucketing network, we have tow forms of data Iterator to provide the data:
One is  provide the data wrapped together:
self.provide_data = [('data', (batch_size, self.default_bucket_key)),
                             ('mask', (batch_size, self.default_bucket_key))] + init_states
self.provide_label = [('softmax_label', (self.batch_size, self.default_bucket_key))]

The other is to provide the data at each step:
self.provide_data = [('%s/%d' % (self.data_name, t), (self.batch_size,)) for t in range(self.default_bucket_key)] +
        [('%s/%d' % (self.mask_name, k), (self.batch_size,)) for k in range(self.default_bucket_key)] + init_states
self.provide_label = [('%s/%d' % (self.label_name, t), (self.batch_size,)) for t in range(self.default_bucket_key)]

And in the fit process, especially at the time we create a DataParallelExecutorGroup:
self.aux_names = sym.list_auxiliary_states()
self.param_idx = [i for i in range(len(arg_names)) if arg_names[i] in param_names]
self.param_names = [arg_names[i] for i in self.param_idx]

and
self.param_arrays = [[e.arg_arrays[i] for e in self.train_execs]
                             for i in self.param_idx]
self.grad_arrays = [[e.grad_arrays[i] for e in self.train_execs]
                            for i in self.param_idx]

Note: The above code is in the executor_manager.py
We assign some values to the params in the network. However the param_idx is get from the arg_names.
I have following questions:

Does symbol.list_arguments has a order? For example, all network parameters followed by the input and output parameters.
Does the arg_names change because of the change of bucket in the lstm network?
In my code, it has following error:

    epoch_end_callback = [ mx.callback.do_checkpoint( '%s/%s' % (params_dir, expt_name), 100) ]
  File "/home/zhaozijian/mxnet/python/mxnet/model.py", line 785, in fit
    sym_gen=self.sym_gen)
  File "/home/zhaozijian/mxnet/python/mxnet/model.py", line 217, in _train_multi_device
    executor_manager.load_data_batch(data_batch)
  File "/home/zhaozijian/mxnet/python/mxnet/executor_manager.py", line 386, in load_data_batch
    shared_group=self.execgrp)
  File "/home/zhaozijian/mxnet/python/mxnet/executor_manager.py", line 232, in __init__
    for i in self.param_idx]
IndexError: list index out of range

And I print the parameters, it seems to have no order:
arg_names: ['data/12', 'embed_weight', 'dec_l0_i2h_weight', 'dec_l0_i2h_bias', 'data/11', 'enc_l0_i2h_weight', 'enc_l0_i2h_bias', 'data/10', 'data/9', 'data/8', 'data/7', 'data/6', 'data/5', 'data/4', 'data/3', 'data/2', 'data/1', 'data/0', 'l0_init_h', 'enc_l0_h2h_weight', 'enc_l0_h2h_bias', 'l0_init_c', 'mask/0', 'mask/1', 'mask/2', 'mask/3', 'mask/4', 'mask/5', 'mask/6', 'mask/7', 'mask/8', 'mask/9', 'mask/10', 'mask/11', 'dec_l0_h2h_weight', 'dec_l0_h2h_bias', 'mask/12', 'cls_weight', 'cls_bias', 'label/0', 'data/13', 'mask/13', 'label/1', 'data/14', 'mask/14', 'label/2', 'data/15', 'mask/15', 'label/3', 'data/16', 'mask/16', 'label/4', 'data/17', 'mask/17', 'label/5', 'data/18', 'mask/18', 'label/6', 'data/19', 'mask/19', 'label/7', 'data/20', 'mask/20', 'label/8', 'data/21', 'mask/21', 'label/9', 'data/22', 'mask/22', 'label/10', 'data/23', 'mask/23', 'label/11', 'data/24', 'mask/24', 'label/12']
param_names: ['embed_weight', 'dec_l0_i2h_weight', 'dec_l0_i2h_bias', 'enc_l0_i2h_weight', 'enc_l0_i2h_bias', 'enc_l0_h2h_weight', 'enc_l0_h2h_bias', 'dec_l0_h2h_weight', 'dec_l0_h2h_bias', 'cls_weight', 'cls_bias']

I believe the error arises from the change of bucket but I'm not quite sure.
Look forward to your reply.

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!

authors: ZiJianZhao yajiedesign


text:
Fix compile error in Windows-MSVC 2015

Description
(I try to compile in Windows-MSVC 2015, which seems working well after fixing some minor issues)
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] Changes are complete (i.e. I finished coding on this PR)
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

is fixed in https://github.com/apache/incubator-mxnet/pull/10629

Seems that this is no longer relevant. @dongeliu if this is still an issue, feel free to ping me to reopen, or open another pull request.

authors: dongeliu szha yajiedesign


text:
[WIP] Fixes test_nccl_pushpull flaky test

Description
(Brief description on what this PR is about)
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

CI doesn't have build with NCCL yet. cc @marcoabreu

@mxnet-label-bot [pr-work-in-progress]

@lebeg do you know if multiple gpus are required to reproduce the flakyness here? I haven't been able to reproduce on my single 1080 Ti (too poor to run dual T_T)

@perdasilva  Yes, I was. In fact if you look closer to the test than you realise that it would pass only in the case you have 1 GPU :) WIP fix is here https://github.com/apache/incubator-mxnet/pull/12477

@lebeg I'm sorry I came in too late for the game. I'll close this PR and look for another flaky test =D

authors: eric-haibin-lin lebeg perdasilva vandanavk