text:
Have problom in BERT pre-training: how to training on multiple GPUs

Description

I want to train BERT model on GPU, but have some problems. My configuration:

Software environment: Python: 3.7.7,  Cuda: 10.2
Install MXNet: pip install mxnet-cu102  , verion is 1.7.0
Download Model script:  https://github.com/dmlc/gluon-nlp, which branch is 2.0.

Run script gluon-nlp/scripts/bert/run_pretraining.py:

Reference the instruction: https://nlp.gluon.ai/model_zoo/bert/index.html#bert-model-zoo
And download DataSet alse in above web.

  $  mpirun -np 8 -H localhost:8 -mca pml ob1 -mca btl ^openib \
       -mca btl_tcp_if_exclude docker0,lo --map-by ppr:4:socket \
       --mca plm_rsh_agent 'ssh -q -o StrictHostKeyChecking=no' \
       -x NCCL_MIN_NRINGS=8 -x NCCL_DEBUG=INFO -x HOROVOD_HIERARCHICAL_ALLREDUCE=1 \
       -x MXNET_SAFE_ACCUMULATION=1 --tag-output \
  python run_pretraining.py --verbose --model="bert_12_768_12" --warmup_ratio=1 --comm_backend="horovod" \
  	--accumulate=1 --max_seq_length=128 --raw --max_predictions_per_seq=20 --log_interval=1 --ckpt_interval=1000 \
  	--no_compute_acc --data=/home/yangshuo/mxnet/Dataset/pre-train-datasets/enwiki-feb-doc-split/*.train \
  	--num_steps=1000 --total_batch_size=128 --dtype="float16"

Result error:

Seek help:
I have read the guidance, but still don't known how to  running.
Please help me, or can I have correct instruction or suggestion ? thanks.

Please provide the complete error message

Please provide the complete error message

Please provide the complete error message

the whole message:
[1,5]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,4]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,7]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,6]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,2]<stderr>:[21:43:11] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,1]<stderr>:[21:43:11] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,0]<stderr>:[21:43:11] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,3]<stderr>:[21:43:11] src/storage/storage.cc:[1,3]<stderr>:110: Using GPUPooledRoundedStorageManager.
[1,7]<stderr>:INFO:root:Model created
[1,7]<stderr>:DEBUG:root:Random seed set to 91
[1,7]<stderr>:INFO:root:Begin process dataset......
[1,7]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 7
[1,7]<stderr>:INFO:root:400 files are found.
[1,4]<stderr>:INFO:root:Model created
[1,4]<stderr>:DEBUG:root:Random seed set to 580
[1,4]<stderr>:INFO:root:Begin process dataset......
[1,4]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 4
[1,4]<stderr>:INFO:root:400 files are found.
[1,6]<stderr>:INFO:root:Model created
[1,6]<stderr>:DEBUG:root:Random seed set to 555
[1,6]<stderr>:INFO:root:Begin process dataset......
[1,6]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 6
[1,6]<stderr>:INFO:root:400 files are found.
[1,5]<stderr>:INFO:root:Model created
[1,5]<stderr>:DEBUG:root:Random seed set to 185
[1,5]<stderr>:INFO:root:Begin process dataset......
[1,5]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 5
[1,5]<stderr>:INFO:root:400 files are found.
[1,7]<stderr>:[node106:26504:0:26504] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,7]<stderr>:==== backtrace ====
[1,7]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f5c21681cec]
[1,7]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f5c21681f64]
[1,7]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f5e1fe55d44]
[1,7]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f5dc2100564]
[1,7]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f5dc2103790]
[1,7]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f5dc20fbed1]
[1,7]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f5dc20d69d4]
[1,7]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f5c3750818f]
[1,7]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f5c374ffd84]
[1,7]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f5e1ee829dd]
[1,7]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f5e1ee82067]
[1,7]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f5e200b327e]
[1,7]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f5e200b3cb4]
[1,7]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x55d260c8500b]
[1,7]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x55d260ce99a1]
[1,7]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x55d260c2d2b9]
[1,7]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x55d260c7d497]
[1,7]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x55d260ce5cba]
[1,7]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x55d260c2d2b9]
[1,7]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x55d260c7d497]
[1,7]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x55d260ce5cba]
[1,7]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x55d260c7d20b]
[1,7]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x55d260ce4be6]
[1,7]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x55d260c2d2b9]
[1,7]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x55d260c2e1d4]
[1,7]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x55d260c2e1fc]
[1,7]<stderr>:   26  python(+0x22bf44) [0x55d260d43f44]
[1,7]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x55d260d4e2b1]
[1,7]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x55d260d4e4a3]
[1,7]<stderr>:   29  python(+0x2375d5) [0x55d260d4f5d5]
[1,7]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x55d260d4f6fc]
[1,7]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f5e1faa2840]
[1,7]<stderr>:   32  python(+0x1dc3c0) [0x55d260cf43c0]
[1,7]<stderr>:===================
[1,4]<stderr>:[node106:26501:0:26501] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,4]<stderr>:==== backtrace ====
[1,4]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f5fb1eb6cec]
[1,4]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f5fb1eb6f64]
[1,4]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f61b05e1d44]
[1,4]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f615288c564]
[1,4]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f615288f790]
[1,4]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f6152887ed1]
[1,4]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f61528629d4]
[1,4]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f5fc7ca718f]
[1,4]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f5fc7c9ed84]
[1,4]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f61af60e9dd]
[1,4]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f61af60e067]
[1,4]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f61b083f27e]
[1,4]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f61b083fcb4]
[1,4]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x55e8922d700b]
[1,4]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x55e89233b9a1]
[1,4]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x55e89227f2b9]
[1,4]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x55e8922cf497]
[1,4]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x55e892337cba]
[1,4]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x55e89227f2b9]
[1,4]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x55e8922cf497]
[1,4]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x55e892337cba]
[1,4]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x55e8922cf20b]
[1,4]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x55e892336be6]
[1,4]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x55e89227f2b9]
[1,4]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x55e8922801d4]
[1,4]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x55e8922801fc]
[1,4]<stderr>:   26  python(+0x22bf44) [0x55e892395f44]
[1,4]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x55e8923a02b1]
[1,4]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x55e8923a04a3]
[1,4]<stderr>:   29  python(+0x2375d5) [0x55e8923a15d5]
[1,4]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x55e8923a16fc]
[1,4]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f61b022e840]
[1,4]<stderr>:   32  python(+0x1dc3c0) [0x55e8923463c0]
[1,4]<stderr>:===================
[1,5]<stderr>:[node106:26502:0:26502] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,5]<stderr>:==== backtrace ====
[1,6]<stderr>:[node106:26503:0:26503] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,6]<stderr>:==== backtrace ====
[1,5]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f40f065bcec]
[1,5]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f40f065bf64]
[1,5]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f42ead77d44]
[1,5]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f428d022564]
[1,5]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f428d025790]
[1,5]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f428d01ded1]
[1,5]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f428cff89d4]
[1,5]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f410243a18f]
[1,5]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f4102431d84]
[1,5]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f42e9da49dd]
[1,5]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f42e9da4067]
[1,5]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f42eafd527e]
[1,5]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f42eafd5cb4]
[1,5]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x564d0453c00b]
[1,5]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x564d045a09a1]
[1,5]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x564d044e42b9]
[1,5]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x564d04534497]
[1,5]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x564d0459ccba]
[1,5]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x564d044e42b9]
[1,5]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x564d04534497]
[1,5]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x564d0459ccba]
[1,5]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x564d0453420b]
[1,5]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x564d0459bbe6]
[1,5]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x564d044e42b9]
[1,5]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x564d044e51d4]
[1,5]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x564d044e51fc]
[1,5]<stderr>:   26  python(+0x22bf44) [0x564d045faf44]
[1,5]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x564d046052b1]
[1,5]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x564d046054a3]
[1,5]<stderr>:   29  python(+0x2375d5) [0x564d046065d5]
[1,5]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x564d046066fc]
[1,5]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f42ea9c4840]
[1,5]<stderr>:   32  python(+0x1dc3c0) [0x564d045ab3c0]
[1,5]<stderr>:===================
[1,6]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f1a6c25bcec]
[1,6]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f1a6c25bf64]
[1,6]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f1c66a2ad44]
[1,6]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f1c08cd5564]
[1,6]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f1c08cd8790]
[1,6]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f1c08cd0ed1]
[1,6]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f1c08cab9d4]
[1,6]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f1a7e0e118f]
[1,6]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f1a7e0d8d84]
[1,6]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f1c65a579dd]
[1,6]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f1c65a57067]
[1,6]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f1c66c8827e]
[1,6]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f1c66c88cb4]
[1,6]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x562df52e800b]
[1,6]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x562df534c9a1]
[1,6]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x562df52902b9]
[1,6]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x562df52e0497]
[1,6]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x562df5348cba]
[1,6]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x562df52902b9]
[1,6]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x562df52e0497]
[1,6]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x562df5348cba]
[1,6]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x562df52e020b]
[1,6]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x562df5347be6]
[1,6]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x562df52902b9]
[1,6]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x562df52911d4]
[1,6]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x562df52911fc]
[1,6]<stderr>:   26  python(+0x22bf44) [0x562df53a6f44]
[1,6]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x562df53b12b1]
[1,6]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x562df53b14a3]
[1,6]<stderr>:   29  python(+0x2375d5) [0x562df53b25d5]
[1,6]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x562df53b26fc]
[1,6]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1c66677840]
[1,6]<stderr>:   32  python(+0x1dc3c0) [0x562df53573c0]
[1,6]<stderr>:===================
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 0 on node node106 exited on signal 11 (Segmentation fault).

Please provide the complete error message

the whole message:
[1,5]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,4]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,7]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,6]<stderr>:[21:43:10] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,2]<stderr>:[21:43:11] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,1]<stderr>:[21:43:11] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,0]<stderr>:[21:43:11] src/storage/storage.cc:110: Using GPUPooledRoundedStorageManager.
[1,3]<stderr>:[21:43:11] src/storage/storage.cc:[1,3]<stderr>:110: Using GPUPooledRoundedStorageManager.
[1,7]<stderr>:INFO:root:Model created
[1,7]<stderr>:DEBUG:root:Random seed set to 91
[1,7]<stderr>:INFO:root:Begin process dataset......
[1,7]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 7
[1,7]<stderr>:INFO:root:400 files are found.
[1,4]<stderr>:INFO:root:Model created
[1,4]<stderr>:DEBUG:root:Random seed set to 580
[1,4]<stderr>:INFO:root:Begin process dataset......
[1,4]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 4
[1,4]<stderr>:INFO:root:400 files are found.
[1,6]<stderr>:INFO:root:Model created
[1,6]<stderr>:DEBUG:root:Random seed set to 555
[1,6]<stderr>:INFO:root:Begin process dataset......
[1,6]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 6
[1,6]<stderr>:INFO:root:400 files are found.
[1,5]<stderr>:INFO:root:Model created
[1,5]<stderr>:DEBUG:root:Random seed set to 185
[1,5]<stderr>:INFO:root:Begin process dataset......
[1,5]<stderr>:INFO:root:args.num_buckets: 1, num_workers: 8, rank: 5
[1,5]<stderr>:INFO:root:400 files are found.
[1,7]<stderr>:[node106:26504:0:26504] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,7]<stderr>:==== backtrace ====
[1,7]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f5c21681cec]
[1,7]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f5c21681f64]
[1,7]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f5e1fe55d44]
[1,7]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f5dc2100564]
[1,7]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f5dc2103790]
[1,7]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f5dc20fbed1]
[1,7]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f5dc20d69d4]
[1,7]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f5c3750818f]
[1,7]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f5c374ffd84]
[1,7]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f5e1ee829dd]
[1,7]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f5e1ee82067]
[1,7]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f5e200b327e]
[1,7]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f5e200b3cb4]
[1,7]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x55d260c8500b]
[1,7]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x55d260ce99a1]
[1,7]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x55d260c2d2b9]
[1,7]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x55d260c7d497]
[1,7]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x55d260ce5cba]
[1,7]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x55d260c2d2b9]
[1,7]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x55d260c7d497]
[1,7]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x55d260ce5cba]
[1,7]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x55d260c7d20b]
[1,7]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x55d260ce4be6]
[1,7]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x55d260c2d2b9]
[1,7]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x55d260c2e1d4]
[1,7]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x55d260c2e1fc]
[1,7]<stderr>:   26  python(+0x22bf44) [0x55d260d43f44]
[1,7]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x55d260d4e2b1]
[1,7]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x55d260d4e4a3]
[1,7]<stderr>:   29  python(+0x2375d5) [0x55d260d4f5d5]
[1,7]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x55d260d4f6fc]
[1,7]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f5e1faa2840]
[1,7]<stderr>:   32  python(+0x1dc3c0) [0x55d260cf43c0]
[1,7]<stderr>:===================
[1,4]<stderr>:[node106:26501:0:26501] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,4]<stderr>:==== backtrace ====
[1,4]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f5fb1eb6cec]
[1,4]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f5fb1eb6f64]
[1,4]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f61b05e1d44]
[1,4]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f615288c564]
[1,4]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f615288f790]
[1,4]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f6152887ed1]
[1,4]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f61528629d4]
[1,4]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f5fc7ca718f]
[1,4]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f5fc7c9ed84]
[1,4]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f61af60e9dd]
[1,4]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f61af60e067]
[1,4]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f61b083f27e]
[1,4]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f61b083fcb4]
[1,4]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x55e8922d700b]
[1,4]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x55e89233b9a1]
[1,4]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x55e89227f2b9]
[1,4]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x55e8922cf497]
[1,4]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x55e892337cba]
[1,4]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x55e89227f2b9]
[1,4]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x55e8922cf497]
[1,4]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x55e892337cba]
[1,4]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x55e8922cf20b]
[1,4]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x55e892336be6]
[1,4]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x55e89227f2b9]
[1,4]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x55e8922801d4]
[1,4]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x55e8922801fc]
[1,4]<stderr>:   26  python(+0x22bf44) [0x55e892395f44]
[1,4]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x55e8923a02b1]
[1,4]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x55e8923a04a3]
[1,4]<stderr>:   29  python(+0x2375d5) [0x55e8923a15d5]
[1,4]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x55e8923a16fc]
[1,4]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f61b022e840]
[1,4]<stderr>:   32  python(+0x1dc3c0) [0x55e8923463c0]
[1,4]<stderr>:===================
[1,5]<stderr>:[node106:26502:0:26502] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,5]<stderr>:==== backtrace ====
[1,6]<stderr>:[node106:26503:0:26503] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x30)
[1,6]<stderr>:==== backtrace ====
[1,5]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f40f065bcec]
[1,5]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f40f065bf64]
[1,5]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f42ead77d44]
[1,5]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f428d022564]
[1,5]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f428d025790]
[1,5]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f428d01ded1]
[1,5]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f428cff89d4]
[1,5]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f410243a18f]
[1,5]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f4102431d84]
[1,5]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f42e9da49dd]
[1,5]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f42e9da4067]
[1,5]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f42eafd527e]
[1,5]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f42eafd5cb4]
[1,5]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x564d0453c00b]
[1,5]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x564d045a09a1]
[1,5]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x564d044e42b9]
[1,5]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x564d04534497]
[1,5]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x564d0459ccba]
[1,5]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x564d044e42b9]
[1,5]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x564d04534497]
[1,5]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x564d0459ccba]
[1,5]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x564d0453420b]
[1,5]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x564d0459bbe6]
[1,5]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x564d044e42b9]
[1,5]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x564d044e51d4]
[1,5]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x564d044e51fc]
[1,5]<stderr>:   26  python(+0x22bf44) [0x564d045faf44]
[1,5]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x564d046052b1]
[1,5]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x564d046054a3]
[1,5]<stderr>:   29  python(+0x2375d5) [0x564d046065d5]
[1,5]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x564d046066fc]
[1,5]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f42ea9c4840]
[1,5]<stderr>:   32  python(+0x1dc3c0) [0x564d045ab3c0]
[1,5]<stderr>:===================
[1,6]<stderr>:    0  /usr/lib/libucs.so.0(+0x1fcec) [0x7f1a6c25bcec]
[1,6]<stderr>:    1  /usr/lib/libucs.so.0(+0x1ff64) [0x7f1a6c25bf64]
[1,6]<stderr>:    2  /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x4) [0x7f1c66a2ad44]
[1,6]<stderr>:    3  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine11ThreadedVar21AppendWriteDependencyEPNS0_8OprBlockE+0x44) [0x7f1c08cd5564]
[1,6]<stderr>:    4  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine4PushEPNS0_3OprENS_7ContextEib+0x280) [0x7f1c08cd8790]
[1,6]<stderr>:    5  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet6engine14ThreadedEngine9PushAsyncESt8functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEENS_7ContextERKSt6vectorIPNS0_3VarESaISA_EESE_NS_10FnPropertyEiPKcb+0x131) [0x7f1c08cd0ed1]
[1,6]<stderr>:    6  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/mxnet/libmxnet.so(_ZN5mxnet10CopyFromToERKNS_7NDArrayES2_ib+0xaf4) [0x7f1c08cab9d4]
[1,6]<stderr>:    7  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(_ZN7horovod5mxnet29PushHorovodOperationCudaOnCPUENS_6common7Request11RequestTypeEPN5mxnet7NDArrayES6_PKcii+0xe6f) [0x7f1a7e0e118f]
[1,6]<stderr>:    8  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so(horovod_mxnet_broadcast_async+0x54) [0x7f1a7e0d8d84]
[1,6]<stderr>:    9  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f1c65a579dd]
[1,6]<stderr>:   10  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/../../libffi.so.7(+0x6067) [0x7f1c65a57067]
[1,6]<stderr>:   11  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f1c66c8827e]
[1,6]<stderr>:   12  /home/yangshuo/miniconda3/envs/yangshuo/lib/python3.7/lib-dynload/_ctypes.cpython-37m-x86_64-linux-gnu.so(+0x12cb4) [0x7f1c66c88cb4]
[1,6]<stderr>:   13  python(_PyObject_FastCallKeywords+0x48b) [0x562df52e800b]
[1,6]<stderr>:   14  python(_PyEval_EvalFrameDefault+0x51d1) [0x562df534c9a1]
[1,6]<stderr>:   15  python(_PyEval_EvalCodeWithName+0x2f9) [0x562df52902b9]
[1,6]<stderr>:   16  python(_PyFunction_FastCallKeywords+0x387) [0x562df52e0497]
[1,6]<stderr>:   17  python(_PyEval_EvalFrameDefault+0x14ea) [0x562df5348cba]
[1,6]<stderr>:   18  python(_PyEval_EvalCodeWithName+0x2f9) [0x562df52902b9]
[1,6]<stderr>:   19  python(_PyFunction_FastCallKeywords+0x387) [0x562df52e0497]
[1,6]<stderr>:   20  python(_PyEval_EvalFrameDefault+0x14ea) [0x562df5348cba]
[1,6]<stderr>:   21  python(_PyFunction_FastCallKeywords+0xfb) [0x562df52e020b]
[1,6]<stderr>:   22  python(_PyEval_EvalFrameDefault+0x416) [0x562df5347be6]
[1,6]<stderr>:   23  python(_PyEval_EvalCodeWithName+0x2f9) [0x562df52902b9]
[1,6]<stderr>:   24  python(PyEval_EvalCodeEx+0x44) [0x562df52911d4]
[1,6]<stderr>:   25  python(PyEval_EvalCode+0x1c) [0x562df52911fc]
[1,6]<stderr>:   26  python(+0x22bf44) [0x562df53a6f44]
[1,6]<stderr>:   27  python(PyRun_FileExFlags+0xa1) [0x562df53b12b1]
[1,6]<stderr>:   28  python(PyRun_SimpleFileExFlags+0x1c3) [0x562df53b14a3]
[1,6]<stderr>:   29  python(+0x2375d5) [0x562df53b25d5]
[1,6]<stderr>:   30  python(_Py_UnixMain+0x3c) [0x562df53b26fc]
[1,6]<stderr>:   31  /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1c66677840]
[1,6]<stderr>:   32  python(+0x1dc3c0) [0x562df53573c0]
[1,6]<stderr>:===================
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 0 on node node106 exited on signal 11 (Segmentation fault).

Firstly, I want to make sure: is my method correct for pre-training BERT model on multiply   @@GPUs?
@leezu

Firstly, I want to make sure: is my method correct for pre-training BERT model on multiply   @@GPUs?
@leezu

Did you compile MXNet from source? There might be an issue due to your usage of conda (ie. you compiled MXNet without conda but you're now using it with conda; this can be a problem, because conda provides it's own gcc which is incompatible with system toolchain)

Did you compile MXNet from source? There might be an issue due to your usage of conda (ie. you compiled MXNet without conda but you're now using it with conda; this can be a problem, because conda provides it's own gcc which is incompatible with system toolchain)

Software environment: Python: 3.7.7, Cuda: 10.2
Install MXNet: pip install mxnet-cu102 , verion is 1.7.0
Download Model script: https://github.com/dmlc/gluon-nlp, which branch is 2.0.

Do you mean that you use gluon-nlp master branch with MXNet 1.7? It's not supported. You need to use MXNet 2 Alpha release https://github.com/apache/incubator-mxnet/releases/edit/v2.0.0-alpha for using GluonNLP master branch

Software environment: Python: 3.7.7, Cuda: 10.2
Install MXNet: pip install mxnet-cu102 , verion is 1.7.0
Download Model script: https://github.com/dmlc/gluon-nlp, which branch is 2.0.

Do you mean that you use gluon-nlp master branch with MXNet 1.7? It's not supported. You need to use MXNet 2 Alpha release https://github.com/apache/incubator-mxnet/releases/edit/v2.0.0-alpha for using GluonNLP master branch

Software environment: Python: 3.7.7, Cuda: 10.2
Install MXNet: pip install mxnet-cu102 , verion is 1.7.0
Download Model script: https://github.com/dmlc/gluon-nlp, which branch is 2.0.

Do you mean that you use gluon-nlp master branch with MXNet 1.7? It's not supported. You need to use MXNet 2 Alpha release https://github.com/apache/incubator-mxnet/releases/v2.0.0-alpha for using GluonNLP master branch. If you don't like to compile MXNet from source, you can also just follow https://github.com/dmlc/gluon-nlp#installation

I use gluon-nlp branch 2.0 with MXNet 1.7. Is it also not supported?
I will try as you suggest. think you.

Software environment: Python: 3.7.7, Cuda: 10.2
Install MXNet: pip install mxnet-cu102 , verion is 1.7.0
Download Model script: https://github.com/dmlc/gluon-nlp, which branch is 2.0.

Do you mean that you use gluon-nlp master branch with MXNet 1.7? It's not supported. You need to use MXNet 2 Alpha release https://github.com/apache/incubator-mxnet/releases/v2.0.0-alpha for using GluonNLP master branch. If you don't like to compile MXNet from source, you can also just follow https://github.com/dmlc/gluon-nlp#installation

I use gluon-nlp branch 2.0 with MXNet 1.7. Is it also not supported?
I will try as you suggest. think you.

I think my environment of 'mpirun'  mybe wrong, such as  optional parameters:
mpirun -np 8 -H localhost:8 -mca pml ob1 -mca btl ^openib \
       -mca btl_tcp_if_exclude docker0,lo --map-by ppr:4:socket \
       --mca plm_rsh_agent 'ssh -q -o StrictHostKeyChecking=no' \
       -x NCCL_MIN_NRINGS=8 -x NCCL_DEBUG=INFO -x HOROVOD_HIERARCHICAL_ALLREDUCE=1 \
       -x MXNET_SAFE_ACCUMULATION=1 --tag-output \

it may causes problems with inter-process communication. So, what parameters need to  set for Multi-GPU training ?
@leezu

I think my environment of 'mpirun'  mybe wrong, such as  optional parameters:
mpirun -np 8 -H localhost:8 -mca pml ob1 -mca btl ^openib \
       -mca btl_tcp_if_exclude docker0,lo --map-by ppr:4:socket \
       --mca plm_rsh_agent 'ssh -q -o StrictHostKeyChecking=no' \
       -x NCCL_MIN_NRINGS=8 -x NCCL_DEBUG=INFO -x HOROVOD_HIERARCHICAL_ALLREDUCE=1 \
       -x MXNET_SAFE_ACCUMULATION=1 --tag-output \

it may causes problems with inter-process communication. So, what parameters need to  set for Multi-GPU training ?
@leezu

I use gluon-nlp branch 2.0 with MXNet 1.7. Is it also not supported?

I don't know how this branch was created, but there is actually no gluon-nlp 2.0. cc @szha @sxjscience let's delete the branch?
The branch contains commits of GluonNLP 0.x, so yes, it should work with MXNet 1.7

I use gluon-nlp branch 2.0 with MXNet 1.7. Is it also not supported?

I don't know how this branch was created, but there is actually no gluon-nlp 2.0. cc @szha @sxjscience let's delete the branch?
The branch contains commits of GluonNLP 0.x, so yes, it should work with MXNet 1.7

I have no idea about the 2.0 branch. We may just delete it.
@yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert

I have no idea about the 2.0 branch. We may just delete it.
@yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert

I have no idea about the 2.0 branch. We may just delete it.
@yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert

I have tried gluon-nlp branch 0.10.0, and also happened this error. So, gluon-nlp(0.10.0) and MXNet(1.6.0 or 1.7.0) are compatibled, right ? I will check other software environment...

I have no idea about the 2.0 branch. We may just delete it.
@yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert

I have tried gluon-nlp branch 0.10.0, and also happened this error. So, gluon-nlp(0.10.0) and MXNet(1.6.0 or 1.7.0) are compatibled, right ? I will check other software environment...

That should work. In fact, is it feasible to try out our new version with the custom version of MXNet 2.0 and the GluonNLP master branch?
Get Outlook for iOShttps://aka.ms/o0ukef

From: yangshuo0323 notifications@github.com
Sent: Friday, January 29, 2021 7:54:06 PM
To: dmlc/gluon-nlp gluon-nlp@noreply.github.com
Cc: Xingjian SHI xshiab@connect.ust.hk; Mention mention@noreply.github.com
Subject: Re: [dmlc/gluon-nlp] Have problom in BERT pre-training: how to training on multiple GPUs (#1508)
I have no idea about the 2.0 branch. We may just delete it.
@yangshuo0323https://github.com/yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert
I have tried gluon-nlp branch 0.10.0, and also happened this error. So, gluon-nlp(0.10.0) and MXNet(1.6.0 or 1.7.0) are compatibled, right ? I will check other software environment...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/dmlc/gluon-nlp/issues/1508#issuecomment-770151635, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3X3PUCHIMHSXGPBYLLS4N7F5ANCNFSM4WWUK4MA.

That should work. In fact, is it feasible to try out our new version with the custom version of MXNet 2.0 and the GluonNLP master branch?
Get Outlook for iOShttps://aka.ms/o0ukef

From: yangshuo0323 notifications@github.com
Sent: Friday, January 29, 2021 7:54:06 PM
To: dmlc/gluon-nlp gluon-nlp@noreply.github.com
Cc: Xingjian SHI xshiab@connect.ust.hk; Mention mention@noreply.github.com
Subject: Re: [dmlc/gluon-nlp] Have problom in BERT pre-training: how to training on multiple GPUs (#1508)
I have no idea about the 2.0 branch. We may just delete it.
@yangshuo0323https://github.com/yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert
I have tried gluon-nlp branch 0.10.0, and also happened this error. So, gluon-nlp(0.10.0) and MXNet(1.6.0 or 1.7.0) are compatibled, right ? I will check other software environment...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/dmlc/gluon-nlp/issues/1508#issuecomment-770151635, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3X3PUCHIMHSXGPBYLLS4N7F5ANCNFSM4WWUK4MA.

Ok, I will try out the new version of MXNet and GluonNLP. Thank you so much!

That should work. In fact, is it feasible to try out our new version with the custom version of MXNet 2.0 and the GluonNLP master branch? Get Outlook for iOShttps://aka.ms/o0ukef
…
________________________________ From: yangshuo0323 notifications@github.com Sent: Friday, January 29, 2021 7:54:06 PM To: dmlc/gluon-nlp gluon-nlp@noreply.github.com Cc: Xingjian SHI xshiab@connect.ust.hk; Mention mention@noreply.github.com Subject: Re: [dmlc/gluon-nlp] Have problom in BERT pre-training: how to training on multiple GPUs (#1508) I have no idea about the 2.0 branch. We may just delete it. @yangshuo0323https://github.com/yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert I have tried gluon-nlp branch 0.10.0, and also happened this error. So, gluon-nlp(0.10.0) and MXNet(1.6.0 or 1.7.0) are compatibled, right ? I will check other software environment... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1508 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3X3PUCHIMHSXGPBYLLS4N7F5ANCNFSM4WWUK4MA.

Ok, I will try out the new version of MXNet and GluonNLP. Thank you so much!

That should work. In fact, is it feasible to try out our new version with the custom version of MXNet 2.0 and the GluonNLP master branch? Get Outlook for iOShttps://aka.ms/o0ukef
…
________________________________ From: yangshuo0323 notifications@github.com Sent: Friday, January 29, 2021 7:54:06 PM To: dmlc/gluon-nlp gluon-nlp@noreply.github.com Cc: Xingjian SHI xshiab@connect.ust.hk; Mention mention@noreply.github.com Subject: Re: [dmlc/gluon-nlp] Have problom in BERT pre-training: how to training on multiple GPUs (#1508) I have no idea about the 2.0 branch. We may just delete it. @yangshuo0323https://github.com/yangshuo0323 Feel free try out the BERT pretraining code in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert I have tried gluon-nlp branch 0.10.0, and also happened this error. So, gluon-nlp(0.10.0) and MXNet(1.6.0 or 1.7.0) are compatibled, right ? I will check other software environment... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1508 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABHQH3X3PUCHIMHSXGPBYLLS4N7F5ANCNFSM4WWUK4MA.

@yangshuo0323 Thanks! I will encourage to try our new version and we can help you if you meet any problems in training the model. To try the new MXNet, you can install with the following command:
# Install the version with CUDA 10.1
python3 -m pip install -U --pre "mxnet-cu101>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the version with CUDA 10.2
python3 -m pip install -U --pre "mxnet-cu102>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the version with CUDA 11
python3 -m pip install -U --pre "mxnet-cu110>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the cpu-only version
python3 -m pip install -U --pre "mxnet>=2.0.0b20210121" -f https://dist.mxnet.io/python

Also, you can just clone gluonnlp/master and install via the following command:
python3 -m pip install -U -e ."[extras]"

This will give the nlp_data and nlp_process CLI. You can use nlp_data to download corpus like wikipedia and bookcorpus and
Also, you are recommended to install horovod via
HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITHOUT_GLOO=1 HOROVOD_WITH_MPI=1 HOROVOD_WITH_MXNET=1 HOROVOD_WITHOUT_TENSORFLOW=1 python3 -m pip install --no-cache-dir horovod

After that, feel free to try out the example in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert. We will try to help with any issues that you met.

@yangshuo0323 Thanks! I will encourage to try our new version and we can help you if you meet any problems in training the model. To try the new MXNet, you can install with the following command:
# Install the version with CUDA 10.1
python3 -m pip install -U --pre "mxnet-cu101>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the version with CUDA 10.2
python3 -m pip install -U --pre "mxnet-cu102>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the version with CUDA 11
python3 -m pip install -U --pre "mxnet-cu110>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the cpu-only version
python3 -m pip install -U --pre "mxnet>=2.0.0b20210121" -f https://dist.mxnet.io/python

Also, you can just clone gluonnlp/master and install via the following command:
python3 -m pip install -U -e ."[extras]"

This will give the nlp_data and nlp_process CLI. You can use nlp_data to download corpus like wikipedia and bookcorpus and
Also, you are recommended to install horovod via
HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITHOUT_GLOO=1 HOROVOD_WITH_MPI=1 HOROVOD_WITH_MXNET=1 HOROVOD_WITHOUT_TENSORFLOW=1 python3 -m pip install --no-cache-dir horovod

After that, feel free to try out the example in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert. We will try to help with any issues that you met.

The previous error was due to the wrong installation of horovod, which maybe not use the env HOROVOD_WITH_MXNET.
Thanks to everyone who give me advice above.
I will enjoy to try the new version as you advice.

@yangshuo0323 Thanks! I will encourage to try our new version and we can help you if you meet any problems in training the model. To try the new MXNet, you can install with the following command:
# Install the version with CUDA 10.1
python3 -m pip install -U --pre "mxnet-cu101>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the version with CUDA 10.2
python3 -m pip install -U --pre "mxnet-cu102>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the version with CUDA 11
python3 -m pip install -U --pre "mxnet-cu110>=2.0.0b20210121" -f https://dist.mxnet.io/python

# Install the cpu-only version
python3 -m pip install -U --pre "mxnet>=2.0.0b20210121" -f https://dist.mxnet.io/python

Also, you can just clone gluonnlp/master and install via the following command:
python3 -m pip install -U -e ."[extras]"

This will give the nlp_data and nlp_process CLI. You can use nlp_data to download corpus like wikipedia and bookcorpus and
Also, you are recommended to install horovod via
HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITHOUT_GLOO=1 HOROVOD_WITH_MPI=1 HOROVOD_WITH_MXNET=1 HOROVOD_WITHOUT_TENSORFLOW=1 python3 -m pip install --no-cache-dir horovod

After that, feel free to try out the example in https://github.com/dmlc/gluon-nlp/tree/master/scripts/pretraining/bert. We will try to help with any issues that you met.

authors: leezu sxjscience yangshuo0323

repo: dmlc/gluon-nlp


text:
How to use mx.lr_scheduler?

Recently, I'm try to use mx.lr_scheduler.FactorScheduler to reduce my lr periodly. I have read the doc and now how does it work. But I encounter a problem when I apply it in my network. I think I don't understand its' inner mechanism actually.  Does anyone can help me? Any reply will be grateful. ****

@sxjscience Could you explain it for me?

source: gharchive/issue

repo: dmlc/mxnet

license_type: permissive

license_source: github-api


text:
run two machine with two gpu is slower than one machine with one gpu

I used two machine(every machine has one gpu) to use Alexnet run multile machine.the result saids that it is slower than one machine use one gpu. the net is km.
is multi machine is limited by the net?

could you please give more information? e.g. cpu/gpu info, network bandwidth, batch size you use, dist_sync or dist_async?

gpu k80 only use gpu=0, network bandwidth= 1Gbps,   dist_sync;    use one machine batch size=256,use two machine batch size=512.  train_imagenet  Alexnet

are you train on imagenet? i suggest to change the batch size into 1024

alexnet has a high network bandwidth need.
I suggest you try inception-bn again

@freepjf  try incepiton-bn, this model has about 1/10 network bandwidth need of alexnet

@qiaohaijun there is a trick to reduce 4x network traffic which has been tested on cxxnet before. that is, convert a float into an int8 with randomized rounding.

@mli  we use infiniband connect to eash other with two machine（every one with one gpu），set batch size=512 every pc，the speed is only 200 picture/s，but when we use one machine with one gpu ，the speed is 260。what is the problem

what's the network usage? you can check it by nmon if you are using linux

I test with the alexnet, win7, infiniband
local:                               195samples/s (1pc with 1gpu)                 613(1pc with 4gpu)
local_allreduce_cpu:       319samples/s(up on)                              1240samples/s(up on)
dist_sync                         300(2pc with 2gpu)                                 1200(2pc with 8gpu)
one machine two cpu with 8 gpu on two sides is similar with 4gpu on the same side
@mli  is it normal?
and I want to know How many cluster systems are supported in theory with mxnet?

@freepjf
i met the same issue. what's the bandwidth of your IB net? 56G or 100G？
Adding more parameter server may help, but i think bandwidth is the main bottleneck here

authors: GoodJoey freepjf mli qiaohaijun


text:
boost on continuous predictions

The vignette on boosting from predictions in R shows an example for probabilistic outcomes. I am trying to replicate the example with a continuous outcome. I would image that the parameter outputmargin in predict() should be set to FALSE. How would I then use the predictions for xgboost to pick up from?
In the example, the following is used:
setinfo(dtrain, "base_margin", ptrain)
This is specific to when the outcome is probabilistic?

The outputmargin makes no difference for regression.
The setinfo is important since it tells the continuing training process where to start.

repo: dmlc/xgboost


text:
Different AUC score from sklearn.metrics function (binary:logistic)

I'm using xgboost's sklearn wrapper for a binary classifcation task and then use sklearn.metrics' auc for scoring
for reproduce the issue, I will use breast_cancer dataset from sklearn for illustration.
df = load_breast_cancer().data
y = load_breast_cancer().target

X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.33, random_state=7)

params={
    'n_estimators':500,
    'objective': 'binary:logistic',
    'learning_rate': 0.05,
    'gamma':0.1,
    'subsample':0.8,
    'colsample_bytree':0.3,
    'min_child_weight':3,
    'max_depth':6,
    'seed':1024,
    }

clf = XGBClassifier(**params)
clf.fit(X_train, y_train, \
        eval_set=[(X_train, y_train), (X_test, y_test)], \
                  eval_metric='auc', early_stopping_rounds = 50)

the output gives the result as :
Stopping. Best iteration: [168]   validation_0-auc:0.998853       validation_1-auc:0.997766
but when I use auc scoring function by following code:
y_pred = clf.predict(X_test)
print metrics.roc_auc_score(y_test,y_pred)

the result is :
0.957627118644
if I use xgb directly. the auc result matches the sklearn.metrics output
anything wrong with my code?
Environment info
Operating System: Windows 10
Compiler: MinGW-W64-builds-4.2.0
Package used (python/R/jvm/C++): Python
xgboost version used: 0.6
If you are using python package, please provide

The python version and distribution Anaconda python version 2.7

Hi,
you have to predict probabilities (clf.predict_proba) instead of classes to calculate the ROC AUC score:
y_pred = clf.predict_proba(X_test)[:,1]
print 'Last fitted model score:', roc_auc_score(y_test,y_pred)
>> Last fitted model score: 0.997503613191

And by he way, if you use early_stopping you have to refit the classifier with the number of trees from the best round.  Fit will return the model from the last iteration, not the best one. See docs for details.
print 'Best nr of trees:', clf.best_ntree_limit
clf.set_params(**{'n_estimators': clf.best_ntree_limit})
clf.fit(X_train, y_train)

y_pred = clf.predict_proba(X_test)[:,1]
print 'Best eval model score:', roc_auc_score(y_test,y_pred)
>> Best nr of trees: 169
>> Best eval model score: 0.99776639075

authors: Gunthard segalou


text:
[jvm-packages] xgboost4j-spark 0.72: customized objective function "obj_type" error

Hi, found the same question as
https://github.com/dmlc/xgboost/issues/3620
since xgboost4j-spark 0.8 support customized objective function
it looks like the xgboost4j-spark 0.72 still not support customized objective function

meet the same error as below

java.lang.IllegalArgumentException: requirement failed: parameter "obj_type" is not defined, you have to specify the objective type as classification or regression with a customized objective function
  at scala.Predef$.require(Predef.scala:224)
  at ml.dmlc.xgboost4j.scala.spark.XGBoost$.trainDistributed(XGBoost.scala:311)
  at ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator.train(XGBoostEstimator.scala:139)
  at ml.dmlc.xgboost4j.scala.spark.XGBoostEstimator.train(XGBoostEstimator.scala:36)
  at org.apache.spark.ml.Predictor.fit(Predictor.scala:118)
  at ml.dmlc.xgboost4j.scala.spark.XGBoost$.trainWithDataFrame(XGBoost.scala:193)
  ... 55 elided

@hcho3 Our version of spark is 2.1.0, will cost much to update to 2.3.0, so any possibility to fix the issue in xgboost4j-spark 0.72?


text:
Fix Scorecard Github Action not working

Regarding #8255 and https://github.com/dmlc/xgboost/pull/8263#issuecomment-1294613038
I've created a issue to the scorecard team to take a closer look in the error https://github.com/ossf/scorecard-action/issues/998 but it seemed to be fixed in the 2.0.6, as far as I've been able to test.
This PR is upgrading the Scorecard Github Action from 2.0.3 to 2.0.6

Thank you!

source: gharchive/pull-request


text:
🛑 Hacker News is down

In ecc7347, Hacker News (https://news.ycombinator.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Hacker News is back up in 864919e after 49 minutes.

repo: dmuinoo/uptime

license: MIT


text:
logging messages not appearing, TestInit.setLogging issue

Despite the level of sophistication, the !TestInit.setLogging() method returns immediately. It's impossible to get messages INFO, WARNING appearing when running tests manually.
Matt's solution
{{{
logging.basicConfig(level = logLevel)
}}}
makes log messages appear as desired.

closing this issue. Not sure still valid.

authors: ticoann zdenekmaxa

repo: dmwm/WMCore


text:
Filter by "input dataset" in wmstats does not work

Or maybe it doesn't work for TaskChain workflows only...
Example in production is this input dataset
/ttH_M125_mtop1735_TuneCP5_13TeV-powheg-pythia8/RunIIFall17DRPremix-PU2017_94X_mc2017_realistic_v11-v1/AODSIM
which belongs to this workflow (which is active)
https://cmsweb.cern.ch/reqmgr2/fetch?rid=pdmvserv_task_HIG-RunIIFall17MiniAODv2-01000__v1_T_180414_215925_7580

no query, just wmstats web ui

Actually that never worked for TaskChain or StepChain. Filter only looks for the top level properties not in Task/Step level properties.

authors: amaltaro ticoann


text:
Allow runtime environment to resize the cores in the job.

This allows WMRuntime to resize the number of threads in the pset
if the job is in the condor runtime environment (_CONDOR_MACHINE_AD
env var is set and the slot's ad is in the referenced file), the
Cpus attribute is set to an integer, and WMCore_ResizeJob is set to
true.
So, by default this does nothing.  However, Ops can then explicitly
enable job resizing (and setting RequestCpus and RequestMemory
appropriately to reflect this) for a given workflow or subtask in
condor.  Since I suspect we'll need to experiment a bit with the
correct values for RequestMemory / RequestCpus, I don't currently
change those in the CondorPlugin.
This just gives us the flexibility to explore the phase space.
Cases of the resizeCores function tested by hand:

When _CONDOR_MACHINE_AD env var is missing.
When machine ad is unparseable.
When resize job attribute is missing, true, or false.
Non-parseable values for Cpus or WMCore_ResizeJob
Cpus is greater than, equal to, or less than numCores

@hufnagel - this is the idea we discussed over email.

Brian, I'm probably not seeing the whole picture here, thus I have a few questions:

where does the "WMCore_ResizeJob" var (job classAd?) come from? It's not set anywhere in the agent, is it some site condor configuration?
same question for 'cpus' attribute, where does it come from?
At which stage - exactly - Ops would override these values (RequestCpus and RequestMemory)? I believe while the job is still pending on the local schedd(?)

To be honest, I'd rather see the agent making these changes during assignment. This flexibility should be on the WMAgent hands, not on Ops, IMO.
In addition to that, it's strongly coupled to the job splitting, which is one more reason to make this flexibility the right way to avoid future issues and making our life 'miserably' for debugging issues.

Hi @amaltaro -

WMCore_ResizeJob would come from the job ad... actually, that's a bug.  I only look at the machine ad.  I'll fix that with the other items @ericvaandering mentions.
Cpus comes from machine ad - this is set my HTCondor.
Ops would decide whether they want to resize jobs while the job is still pending in the queue.  In fact, talking to Jean-Roch, he prefers the other direction - starting everything with single core and then selectively marking as multicore.  This way, there's less chance we overrun the job runtime estimates.

Basically, we, I would prefer that the WMAgent would mark the workflow as "single core but use multicore as available".  However, I think this will take some tuning to get right.  Hence, I would like to:

Add this to the runtime so Ops has the ability to set "single core but use multicore as available" per-job.
Figure out the correct tunings - particularly, we'll need to get RequestMemory and MaxWallTimeMins correct.
Once we decide we're happy, put it back into the agent as an option at the workflow level.

I don't see a strong advantage to change both the runtime code and the BossAir code in a single step.

Please keep in mind that due to the way the code is written, this doesn't allow you to convert singlecore workflows to multicore. The code path you modified is completely bypassed for singlecore due to backwards compatibility. We do not not override numberOfCores for 1 for singlecore jobs, we don't do anything and rely on the singlecore CMSSW default. We either have to give up on the default and always override or we have to restructure the code a bit.

@ericvaandering - can this roll in yet?

No, we can put this in next week once Seangchan is back and presumably all the tests and comments have been resolved.

(not unit tests, the current failures are noise)

This need to be updated current production agent, right? If Alan's test is good we will make the the new patch for 1.0.14_wmagent and apply to agent.
So for this test and for temporary, Ops will do condor_q edit fo change WMCore_ResizeJob to true for individual Job, correct?

@ticoann - yes, for now Ops will take responsibility for enabling WMCore_ResizeJob.

We can put this in a patch release when it's validated by Alan's tests. Since we're not gaining much with this there is no "drop everything" rush.

I'd merge shouldResizeResources function into resizeResources, since half of that code is again in resizeResources.

Finally, if it's not asking too much. A unit test to validate that function would be awesome :-)

Ok, I think I got all your comments incorporated.  Let's see what pylint says.
I'll be happy to do a unit test ... in a separate PR (file a ticket as a reminder?).  It should actually be straightforward to do, but might first take me awhile to figure out how to do this in WMCore in the first place -- I'd rather not hold up this work for that.

test this please

As I mentioned to Alan, I verified the runtime jobs now look correct.

Brian, I just took a quick look at those two *.ad files and the key names seems to be written according to what we set in PyCondor plugin. Why do you want to always lower() those keys (besides a possible change in gWMS that would break the agent...)?
For clarity reasons, I'd rather have the exact key matching over there.

ClassAd keys are not case sensitive (and we don't control some of the keys, such as memory or cpus).  We have indeed seen cases where the underlying implementation changes slightly and case-sensitive comparisons break.
Doing a case-sensitive comparison for a case-insensitive language makes no sense, even if it increases readability.

Ok, we have what we have then :)

Another concern that I have is with the max/min limits defined by the agent:
https://github.com/dmwm/WMCore/blob/5f0e0751bbeb6ec6b2f7d2fdf492e5a8d55fe33c/etc/WMAgentConfig.py#L201
I've seen jobs requesting (actually matching to pilots) 60h of walltime (and could be worse).
Can we assume that
MaxWallTimeMins = WMCore_ResizeJob ? ( 10800 / RequestCpus + 15 ) : 2700, if such job run in single core, the pilot lifetime would guarantee at least 10815mins (otherwise it would not match against a single core, I suppose...)?
Then another obstacle is with the Timeouts defined for the JobStatusLite:
https://github.com/dmwm/WMCore/blob/5f0e0751bbeb6ec6b2f7d2fdf492e5a8d55fe33c/etc/WMAgentConfig.py#L226
which by default considers pilot lifetime of 48h, at the best case. Of course we can tweak this number and increase it, however these settings are in an agent basis and we should target the most common lifetime out there.

Correct - if the lifetime on a single core works out to be 60 hours, you need a pilot with at least 60 hours remaining (actually, the way things are setup, probably closer to 72 hours).
While all this is good to keep in mind, it's also a driving reason why we plan to inject as single-core and increase the core count (rather than decreasing).  Hence, the jobs should always get shorter - it avoids some of the job lifetime issues.

As an additional comment, there will be cases not behaving well and taking too much resources. E.g.:
Resizing job.  Initial resources: {'cores': 8}
Resizing job.  Resulting resources: {'cores': 1}
...
2016-04-19 16:45:19,654:INFO:Watchdog:  maxRSS: 4558000
2016-04-19 16:45:19,654:INFO:Watchdog:  softTimeout: 1040400
2016-04-19 16:45:19,654:INFO:Watchdog:  hardTimeout: 1048400

It was originally requesting 8GB of RAM (for 8 cores), then running in single core it requests 4.5GB.
This job actually surprises me that it found glideins providing ~1M secs of lifetime (?) Is it something that we should worry about?

Yup - there's upper-level logic needed to avoid somewhat-hopeless cases like the one above.
Indeed, there are glideins (particularly, at CERN) that last for 30 days.

@bbockelm squash please

@ticoann do it! :-P ready to be merged

@ticoann sorry for messing with yes and no... but we better keep this and the condor resize PR on hold.
I added you to a private email reporting issues in frontend.

superseeded by #7363

authors: amaltaro bbockelm ericvaandering hufnagel ticoann


text:
Fix a python2.6 regression code

Jenkins are complaining about these py2.6 code that was recently inserted.

test this please

test failure doesn't seems to be related to this.

test this please


text:
🛑 AR Panel is down

In 4c89e17, AR Panel (https://panel.siembro.com) was down:

HTTP code: 404
Response time: 159 ms

Resolved: AR Panel is back up in 01a193d after 20 minutes.


text:
Multi-stream TCP and UDP connections, encryption and performance

What's the PR:

Adjustable multiple TCP and UDP connections per each client in client binary.
Adjustable multiple UDP connections per each client's TCP stream in server binary.
Encryption API added and XOR encryption implemented.
Client is now lock-free. (On UDP connect, recv_from will not try to receive packets related to connected sockets)
Used more performant data structures in fake-tcp crate.
fake-tcp randomly choosing tun replaced with linear atomic rotating integer and also, ACK and RST choosing tun algorithm will follow this algorithm, not choosing tun[0] repeatedly.
fake-tcp will choose a port without panic.
Prevent panics in server and client event loop.
Zero-sized datagrams are acceptable over UDP and now allowed in both server and client.
Cargo can now generate faster binaries.

Hello,
Thanks for your contribution. However, I am a bit concerned about the introduction of unsafe code into Phantun. One of the huge advantage of Phantun is that it is written in 100% safe Rust and I am not willing to give it up easily without some good reasoning.
Have you ran any performance benchmark of this change? What is the improvement?

Hello,
Thanks for your contribution. However, I am a bit concerned about the introduction of unsafe code into Phantun. One of the huge advantage of Phantun is that it is written in 100% safe Rust and I am not willing to give it up easily without some good reasoning.
Have you ran any performance benchmark of this change? What is the improvement?

What is the "unsafe" code meaning?The merge code not written by RUST?Or add some "unsafe" crates?

I'd used unsafe to bypass vectors' boundary check where both vectors' index and size were deterministic, but I just removed them to make it 100% safe code.
And for the benchmark, this repository doesn't contain any test suite to benchmark the code and evaluate improvements and regressions. The tests are on yours to get these results, but it is expected that these changes will make it faster. The server code wasn't written to be bidirectional and it could just send or receive one at the time. Also, using faster data structures can make huge difference on load.

authors: Handsome1080P dndx sabify

repo: dndx/phantun


text:
Updated esprima checksum

I just built a clean develop branch and got the following yarn error:
➤ YN0018: │ esprima@https://github.com/substack/esprima.git#commit=0a7f8489a11b44b019ce168506f535f22d0be290: The remote archive doesn't match the expected checksum

I had to run the following to make it work
YARN_CHECKSUM_BEHAVIOR=update yarn

which essentially updated the checksum in the yarn.lock file.

Looks like it did not like it...

Yeah :-) trying to figure out what went wrong

I don't know what's wrong. When I build it locally, I need the updated checksum or otherwise it fails for me. But then the CI pipeline doesn't like the new checksum. It seems to be the same issue reported here: https://github.com/yarnpkg/berry/issues/1142
Anyway, the build works for me locally with the updated checksum, so I'll go ahead and close this PR. If the issue surfaces again we can take a deeper look.

authors: daguiler valadas

license: mit

license_source: bigquery


text:
Blank screen in chroot after a few hours

Entering /mnt/stateful_partition/crouton/chroots/crouton...
crouton: version 1-20180718130616~master:5d4c3014
release: xenial
architecture: armhf
xmethod: xorg
targets: xorg,lxde,xiwi
host: version 11151.59.0 (Official Build) beta-channel veyron_speedy
kernel: Linux localhost 3.14.0 #1 SMP PREEMPT Mon Dec 10 23:36:54 PST 2018 armv7l armv7l armv7l GNU/Linux
freon: yes
Not unmounting /mnt/stateful_partition/crouton/chroots/crouton as another instance is using it.

Recently, I've been getting a problem where when switching to my chroot (Arch Linux ARM, inside an Ubuntu chroot), I get a blank screen with no backlight - I can still use the chroot, switch back to ChromeOS, etc.
Relevant parts of the X.Org log file:
X.Org X Server 1.18.4
Release Date: 2016-07-19
X Protocol Version 11, Revision 0
Build Operating System: Linux 4.4.0-131-generic armv7l Ubuntu
Current Operating System: Linux localhost 3.14.0 #1 SMP PREEMPT Mon Dec 10 23:36:54 PST 2018 armv7l
Kernel command line: cros_secure console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 3334144 verity payload=PARTUUID=123b886b-155b-b74f-9423-a69ab4b9fd17/PARTNROFF=1 hashtree=PARTUUID=123b886b-155b-b74f-9423-a69ab4b9fd17/PARTNROFF=1 hashstart=3334144 alg=sha1 root_hexdigest=7824f715ef5de374c95dc02f2280264aaa0aa95d salt=139a69be229bd25e77a6350d1746f4456d158c00ca1046656076ba382165ce2d" noinitrd vt.global_cursor_default=0 kern_guid=123b886b-155b-b74f-9423-a69ab4b9fd17
Build Date: 10 August 2018  12:03:42PM
xorg-server 2:1.18.4-0ubuntu0.8 (For technical support please see http://www.ubuntu.com/support)
Current version of pixman: 0.33.6
        Before reporting problems, check http://wiki.x.org
        to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(++) Log file: "/tmp/Xorg.crouton.1.log", Time: Wed Feb  6 21:27:45 2019
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
Error org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.LibCrosService was not provided by any .service files
gbm: failed to open any driver (search paths /usr/lib/arm-linux-gnueabihf/dri:${ORIGIN}/dri:/usr/lib/dri)
gbm: Last dlopen error: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory
failed to load driver: rockchip
EGL_MESA_drm_image required.
crouton: version 1-20180718130616~master:5d4c3014
release: xenial
architecture: armhf
xmethod: xorg
targets: xorg,lxde,xiwi
host: version 11151.59.0 (Official Build) beta-channel veyron_speedy
kernel: Linux localhost 3.14.0 #1 SMP PREEMPT Mon Dec 10 23:36:54 PST 2018 armv7l armv7l armv7l GNU/Linux
freon: yes
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-ixn'
 (0xbec22aa0) Warning: LockScreenManager couldn't start

...

 (0xbec22aa0) Warning: "Session 'session', new input device '', keyboard setting will be (optionaly) reloaded..."
gbm: failed to open any driver (search paths /usr/lib/arm-linux-gnueabihf/dri:${ORIGIN}/dri:/usr/lib/dri)
 (0xbec22aa0) Warning: "Session 'session', new input device '', keyboard setting will be (optionaly) reloaded..."
 (0xbec22aa0) Warning: "Session 'session', new input device '', keyboard setting will be (optionaly) reloaded..."
gbm: Last dlopen error: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory
failed to load driver: rockchip
 (0xbec22aa0) Warning: "Session 'session', new input device '', keyboard setting will be (optionaly) reloaded..."
 (0xbec22aa0) Warning: "Session 'session', new input device '', keyboard setting will be (optionaly) reloaded..."
EGL_MESA_drm_image required.
 (0xbec22aa0) Warning: "Session 'session', new input device '/dev/input/event4', keyboard setting will be (optionaly) reloaded..."
 (0xbec22aa0) Warning: "Session 'session', new input device '/dev/input/event2', keyboard setting will be (optionaly) reloaded..."
 (0xbec22aa0) Warning: "Session 'session', new input device '/dev/input/event0', keyboard setting will be (optionaly) reloaded..."
xf86: found device 1
 (0xbec22aa0) Warning: "Session 'session', new input device '/dev/input/event3', keyboard setting will be (optionaly) reloaded..."
 (0xbec22aa0) Warning: "Session 'session', new input device '/dev/input/event1', keyboard setting will be (optionaly) reloaded..."

Errors in about:gpu (this appears every time I switch to the chroot and back):
[1462:1493:0207/125510.876822:VERBOSE1:drm_util.cc(467)] : Failed to get EDID blob for connector 32
[1462:1493:0207/125510.876987:WARNING:screen_manager.cc(112)] : Display controller (crtc=21) already present.
[1462:1493:0207/125510.877684:VERBOSE1:drm_display.cc(104)] : DRM configuring: device=/sys/devices/display-subsystem.5/drm/card1 crtc=21 connector=32 origin=0,0 size=0x0
[1462:1493:0207/125512.962957:VERBOSE1:drm_util.cc(467)] : Failed to get EDID blob for connector 32
[1462:1493:0207/125512.963136:WARNING:screen_manager.cc(112)] : Display controller (crtc=21) already present.
[1462:1493:0207/125512.983162:VERBOSE1:drm_display.cc(104)] : DRM configuring: device=/sys/devices/display-subsystem.5/drm/card1 crtc=21 connector=32 origin=0,0 size=1366x768

I think this problem is related to udev, as I've been stopping it due to spi0 using a lot of CPU, and only starting it when I need to plug in a USB drive - the blank screen occurs after starting udev, which switches back to ChromeOS.
Is there anything I can do other than use Xiwi? Would it be possible to change the udev config to fix this?
I think starting and stopping udev from a crosh tab might help rather than breaking out of my Arch chroot and starting it form there - I'll try this next time I restart.

Yup, looks like starting and stopping udev from a crosh tab does work.
I still want to know if there are any other workarounds, as pressing Ctrl-Shift-Alt-F2, Ctrl-Alt-t, typing shell, sudo start udev and my password takes way too long :)
A fix for spi0 using a lot of CPU, so I don't have a reason to stop udev in the first place would also be nice...

It looks like this problem might not be connected with udev, after all...

After Chrome OS updating a week or so ago, I haven't had as many problems, but I still have to kill udev to save on CPU usage.
I haven't had the screen go blank yet, so I think that at least has been fixed...

After Chrome OS updating a week or so ago, I haven't had as many problems, but I still have to kill udev to save on CPU usage.
I haven't had the screen go blank yet, so I think that at least has been fixed...

repo: dnschneid/crouton


text:
🛑 Site SCAE RGA is down

In 88e1930, Site SCAE RGA (https://www.sca-rga.interieur.gouv.fr/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Site SCAE RGA is back up in 17c448b.

repo: dnum-mi/stats-sites-api


text:
Using "crud().updateStatic()" always include a "USING TTL=0" even when TTL is not set

Hi,
I am using the following code to update static column of a table :
Person_Manager person_manager = managerFactory.forPerson();
Person p = new Person();
p.setStaticDate(new Date(System.currentTimeMillis()));
p.setUserUid("myuseruid")
mgr.crud().updateStatic(p).generateAndGetBoundStatement();

As you can see I don't set the TTL in the query. But When I look at the statement that is generated I have the following trace :
UPDATE ks.person USING TTL :ttl SET static_date=:static_date WHERE user_uid=:user_uid; { [ttl:0][static_date:Fri Apr 28 15:56:23 CEST 2017][user_uid:myuseruid] }
There is an issue because I expect the TTL to be set only when I use the method ".usingTimeToLive()" and I didn't use it here.
Regards

Not a problem. Setting TTL = 0 means no TTL for Cassandra

Ok, thanks.
That's good to know.
However I still found it strange to have a CQL BoundStatement generated with something I didn't asked.

It's just an impl detail, it avoid to prepare 2 queries, one without TTL and one with TTL

authors: doanduyhai mykapp

repo: doanduyhai/Achilles


text:
Scroll into view of persistent toc is not working

I upgraded my documentation from xslTNG 1.5.4 to 2.1.9, and now the scroll into view of the persistent toc is no longer working.
In my browser console, the following is reported:

ToC scroll, no match: file/documentation/chunk/en/refdocs/fblangref50/fblangref50-ddl-table.html#fblangref50-ddl-tbl-alter

The match doesn't occur, because in the persistent-toc.html the href is fblangref50-ddl-table.html#fblangref50-ddl-tbl-alter, instead of the full path it's trying now.
Manually executing the following in my browser console will scroll it into view:
var target = document.querySelector("nav.toc div a[rel-path='fblangref50-ddl-table.html#fblangref50-ddl-tbl-alter']")
target.scrollIntoView()

Example of an affected document: https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/firebird-50-language-reference.html

I don't really program JavaScript, but I guess that changing the code in persistent-toc.js to the following will do the trick:
      let url = window.location.pathname;
      let hash = window.location.hash;
      if (window.location.search === "?toc") {
        // Remove ?toc from the URI so that if it's bookmarked,
        // the ToC reference isn't part of the bookmark.
        window.history.replaceState({}, document.title,
                                    window.location.origin
                                    + url
                                    + hash);
      }

      let path = url.substring(url.lastIndexOf('/') + 1) + hash;
      let target = document.querySelector("nav.toc div a[rel-path='"+path+"']");
      if (target) {
        target.scrollIntoView();
      } else if (hash === '') {
        target = document.querySelector("nav.toc div a[rel-path^='"+path+"']");
        if (target) {
          target.scrollIntoView();
        } else {
          console.log(`ToC scroll, no match: ${path}`);
        }
      } else {
        console.log(`ToC scroll, no match: ${path}`);
      }

NOTE: This also addresses another issue if the current location doesn't have a hash, but the entry in the toc has a hash, by retrying with ^= (starts with).
However, I don't know if this might be too naive for all cases.

Interesting. Thanks for the report. I need to sort out some unit testing for this, I guess. I'll try to investigate this evening.

I locally tried my suggested workaround and it works for my documentation. I will leave the document linked above as-is for the time being for reference.

It's curious. I have reproduced the problem, but it was in some sense the opposite of what you reported. My test case was looking for ptoc-001.html#id when the persistent ToC only contained #id. I don't actually see how the persistent ToC can contain href values that don't have hashes. But I've adapted your approach to handle all three cases: path#id, #id, path. It probably needs a more thoughtful solution, but this seems better than leaving it broken.

Unfortunately the change in 2.2.0 did not fix this issue for me, so I'm now using the fix I proposed initially in a custom JS file set with persistent-toc-js.

authors: mrotteveel ndw

repo: docbook/xslTNG


text:
Relicense

Would it be possible to relicense this project under LGPL or similar?

Yes plz :)
I worry the answer is no because of the preceding code. But then, I also don't know how licensing works if all traces of the original licensed work have disappeared?

authors: alexwhittemore zaturox

repo: doceme/py-spidev


text:
Unable to connect externally with 9.5.10-1.pgdg80+1

After pulling latest image from this repo where PG_VERSION=9.5.10-1.pgdg80+1 we see the following exception when trying to connect externally through SQLWorkbench:
2017-11-10 16:12:35 ERROR SQL Exception when connecting The connection attempt failed.
java.sql.SQLException: The connection attempt failed.
    at workbench.db.DbDriver.connect(DbDriver.java:548)
    at workbench.db.ConnectionMgr.connect(ConnectionMgr.java:255)
    at workbench.db.ConnectionMgr.getConnection(ConnectionMgr.java:182)
    at workbench.gui.components.ConnectionSelector.doConnect(ConnectionSelector.java:227)
    at workbench.gui.components.ConnectionSelector$1.run(ConnectionSelector.java:131)
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:275)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
    at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:194)
    at org.postgresql.Driver.makeConnection(Driver.java:450)
    at org.postgresql.Driver.connect(Driver.java:252)
    at workbench.db.DbDriver.connect(DbDriver.java:511)
    ... 4 more
Caused by: java.io.EOFException
    at org.postgresql.core.PGStream.receiveChar(PGStream.java:290)
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:418)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:222)
    ... 9 more

This worked just fine when PG_VERSION=9.5.9-1.pgdg80+1
Additionally we are unable to connect using the psql client:
➜  rocket git:(dockerize-postgres) PGPASSWORD=postgres psql -h localhost -p 5432 -d dwenigma -U postgres --no-password
psql: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
➜  rocket git:(dockerize-postgres)

This was also working fine with the older version...  Any ideas why this is happening?

How are you running postgres (ie what does your docker run or docker-compose.yml look like)?
We run a connection test on every version bump using scripts in the docker-library/official-images. See the most recent TravisCI test here.  I also ran a container locally and was able to connect with psql in another container.
See, postgres-basics and postgres-initdb for the specifics of the tests.

I am getting server closed the connection unexpectedly error while trying to connect to postgres container using pgadmin. I tried both, new pgadmin4 container version and desktop version as well, both return me same error.
Postgres container were running using docker run and docker-compose, I had no changes to docker-compose file, and it was working couple weeks ago.
I do port exposing to host machine.

authors: ahanoff rbanikaz yosifkit

repo: docker-library/postgres


text:
fix misspell "scrutiny" in content/content.go

Signed-off-by: fate-grand-order chenjg@harmonycloud.cn

LGTM

authors: fate-grand-order hqhq

repo: docker/containerd


text:
Separate authentication specification from JWT specific implementation

Currently the authentication specification for the V2 API contains two main sections. The first describing the role of an authentication server and the second describing a JWT token specification which is used by the hub and integrated into the distribution project. These sections should be split out and the JWT token specification should be listed as one example of an authentication server with distribution server side support. The authentication server should be described more generally.  In the future a wider range of tokens should be accepted including macaroon based authentication.
cc @mattmoor raising this issue in the working group

In particular, I would word this as "tokens should be treated as opaque data by the Docker client".
It should be assumed that only the registry that provided a token can interpret it.
thanks for opening this.

authors: dmcgowan mattmoor

repo: docker/distribution


text:
[17.07] backport Fix api server null pointer def on inspect/ls null ipam-driver networks

backport:

moby/moby/pull/34395 Fix api server null pointer def on inspect/ls null ipam-driver networks.

with cherry pick moby/moby@beebfc0
$ git cherry-pick -s -x -Xsubtree=components/engine beebfc0

Test failures are consistent to what we know has been failing:
FOR: https://jenkins.dockerproject.org/job/docker-ce-pr/95/consoleText
Passed (thought would fail):

Failed (should fail)
DockerSuite.TestRmiContainerImageNotFound
DockerSuite.TestRmiImageIDForceWithRunningContainersAndMultipleTags
DockerSuite.TestRunAttachDetachFromInvalidFlag
DockerSuite.TestRunWithNanoCPUs
DockerSuite.TestUpdateWithNanoCPUs
DockerSwarmSuite.TestServiceLogs
DockerSwarmSuite.TestSwarmNetworkPluginV2
DockerSwarmSuite.TestSwarmServicePsMultipleServiceIDs

Checked with: https://gist.github.com/seemethere/6128cb2391cd43b65b0ff4880af427e5

authors: seemethere vieux

repo: docker/docker-ce


text:
remove redundant install for scan-cli-plugin

This explict install was added because docker 20.10 did not have an explicit dependency on the plugin. The current release of the Docker CLI (23.0) already has a "Recommends:" dependency on the plugin, and 0095742fb5d923f876e720731073c541f47f0f79 (https://github.com/docker/docker-install/pull/335) removed the "--no-install-recommends", so fresh installs of Docker 23.0 will already install this plugin through that dependency.
Removing the explicit install, because the scan-cli-plugin has been deprecated, and will no longer receive updates, so we don't want it to be installed for the upcoming 24.0 release when using this script. see; https://github.com/docker/scan-cli-plugin/blob/b69ac460a6c23243b858326f0e26f6bd202ca781/internal/deprecation.go#L13-L15 (https://github.com/docker/scan-cli-plugin/pull/225)

@vvoland @laurazard @crazy-max ptal
/cc @cdupuis @eunomie

repo: docker/docker-install


text:
kernel 2.6

Problem description

Problem location

I saw a problem on the following URL:

I couldn't find the information I wanted. I expected to find it near the following URL:

Other:

Project version(s) affected

Suggestions for a fix

sorry

It is a  wrong operation.

It is a mistake.

sorry, I make a mistake.


text:
Update the trust sandbox documentation to use theupdateframework/notary images

Update the trust sandbox docker-compose setup to be on par with latest notary images
Proposed changes
I have updated the docker security trustsandbox documentation to use the actively developed notary server. The current example shows deprecation notices and isn't representative to what users would be using today.

Deploy preview for docsdocker ready!
Built with commit dafe6881329f3744907f7410f906274cb40bab7c
https://deploy-preview-10078--docsdocker.netlify.com

Please let me know if there are any further actions required to merge this PR.

Hey @marcofranssen , is there any chance you can remove the UCP enterprise commits from this PR as they are not related to the Notary Compose file changes. Thanks :)

@ollypom what do you mean by UCP enterprise commits?

@ollypom I have rebased my branch again on master. Should be good now.

For convenience here the particular page https://deploy-preview-10078--docsdocker.netlify.com/engine/security/trust/trust_sandbox/

authors: GordonTheTurtle marcofranssen ollypom


text:
[WIP] Get started docs updates

Proposed changes
Revamping the Get started docs to simplify Quick start content
Move content on Orchestration off Quick start to a new node
Related issues (optional)
https://docker.atlassian.net/browse/ENGDOCS-319

I had a closer look at the node version which we should update.

Thanks Stefan. I haven't pushed the updates we were looking at yesterday.

@StefanScherer I've just pushed the updates we discussed. PTAL.


text:
Reorg service management topic

Proposed changes
Move some things that were previously in the CLI ref
Some rewrites and reorganization
Add placement pref image
Fixes #3227
(will need to do a PR in moby/moby to remove the duplicated content after this is live)
cc/ @aaronlehmann @thaJeztah @sarahpark

LGTM, merging.


text:
Fedora instructions has incorrect package name

User Email thru feedback states problem
Florian via docker.com
10:15 AM (2 hours ago)
to feedback
Dear all,
First of all, Thank you for the installation instructions for docker on a Fedora system given here: https://docs.docker.com/installation/fedora/).
I think, that there is a little mistake in the instructions for Fedora 21: The package from the official Fedora21 repository is still called docker-io, not docker.
For more information, see Fedora's packages website: https://apps.fedoraproject.org/packages/docker-io
With kind regards

I think @lsm5 should be able to verify if this is correct :)

The package might still be called docker-io, but docker works as well:
$ sudo yum install docker
Loaded plugins: langpacks
Package docker-io-1.5.0-1.fc21.x86_64 already installed and latest version
Nothing to do

what @RickWieman said. docker on f21 points to the real docker-io package. This is in preparation for future releases (f22 and up) where the -io will be dropped as well and there will only be a real docker package.

@moxiegirl given the above information (thanks all!), I think this can be closed, or do you think it should be mentioned in the documentation? (it might only distract).

@thaJeztah let's close this for now. We are in the process of standardizing the installation documentation and this shortly shouldn't be a problem. Thank you for your help clarifying the situation.

authors: RickWieman lsm5 moxiegirl thaJeztah

repo: docker/docker


text:
Truncindex must be able iterate over the existing items and the ability must be used in graph.walkAll()

Now graph.walkAll() does ReadDir() and next trying to do graph.Get() for the each item in the directory. But graph.Get() does graph.idIndex.Get() and if passed ID does not exist in index then graph.Get() returns an error.
So why we need to ReadDir() if we can iterate over graph.idIndex.

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

authors: GordonTheTurtle Ramzec


text:
OOM of exec stops OOM detection for the PID1 container process and following execs

To reproduce:
rm -rf /tmp/oomeventstest && \
mkdir /tmp/oomeventstest && \
cd /tmp/oomeventstest && \
wget https://gist.githubusercontent.com/garagatyi/f3643cc448e1a7550853/raw/eb84488cf8efccf0fe1df5b47a54d28c30121183/Dockerfile && \
docker build .

Then in other terminal:
curl -s http://localhost:2375/events

When we get OOM of PID1 process only docker generates correct events.
docker run -m 64M -e "SLEEP_DELAY=10s" {imageID}
WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
sleep 10s
begin allocating memory...

{"status":"create","id":"36eb96d6c134895854e77f65ac47955b6ee08eb62bceee37cc605157f1c9482b","from":"9e0d5e071a2b","time":1449497856,"timeNano":1449497856620037149}
{"status":"attach","id":"36eb96d6c134895854e77f65ac47955b6ee08eb62bceee37cc605157f1c9482b","from":"9e0d5e071a2b","time":1449497856,"timeNano":1449497856670112786}
{"status":"start","id":"36eb96d6c134895854e77f65ac47955b6ee08eb62bceee37cc605157f1c9482b","from":"9e0d5e071a2b","time":1449497856,"timeNano":1449497856984903035}
{"status":"oom","id":"36eb96d6c134895854e77f65ac47955b6ee08eb62bceee37cc605157f1c9482b","from":"9e0d5e071a2b","time":1449497871,"timeNano":1449497871593721048}
{"status":"die","id":"36eb96d6c134895854e77f65ac47955b6ee08eb62bceee37cc605157f1c9482b","from":"9e0d5e071a2b","time":1449497871,"timeNano":1449497871701251889}

When we get OOM of exec and then get OOM of PID1 process docker doesn't send event about PID1 process OOM
docker run -m 64M -e "SLEEP_DELAY=3m" 9e0d5e071a2b
WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
sleep 3m

docker exec -it ca1f50deaca6 bash
root@ca1f50deaca6:/# export SLEEP_DELAY=1s
root@ca1f50deaca6:/# /script.sh
sleep 1s
begin allocating memory...
Killed

{"status":"create","id":"ca1f50deaca6857da867e1abe47d0c338f2c9a478716245215e99804ebe38116","from":"9e0d5e071a2b","time":1449497892,"timeNano":1449497892815550422}
{"status":"attach","id":"ca1f50deaca6857da867e1abe47d0c338f2c9a478716245215e99804ebe38116","from":"9e0d5e071a2b","time":1449497892,"timeNano":1449497892857804285}
{"status":"start","id":"ca1f50deaca6857da867e1abe47d0c338f2c9a478716245215e99804ebe38116","from":"9e0d5e071a2b","time":1449497893,"timeNano":1449497893160319023}
{"status":"exec_create: bash ","id":"ca1f50deaca6857da867e1abe47d0c338f2c9a478716245215e99804ebe38116","from":"9e0d5e071a2b","time":1449497912,"timeNano":1449497912019407009}
{"status":"exec_start: bash ","id":"ca1f50deaca6857da867e1abe47d0c338f2c9a478716245215e99804ebe38116","from":"9e0d5e071a2b","time":1449497912,"timeNano":1449497912020034187}
{"status":"oom","id":"ca1f50deaca6857da867e1abe47d0c338f2c9a478716245215e99804ebe38116","from":"9e0d5e071a2b","time":1449497946,"timeNano":1449497946969112286}
{"status":"die","id":"ca1f50deaca6857da867e1abe47d0c338f2c9a478716245215e99804ebe38116","from":"9e0d5e071a2b","time":1449498204,"timeNano":1449498204305209257}

Delay between OOM and DIE events more than 4 minutes. It's because OOM event was generated for exec OOM only.

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

authors: GordonTheTurtle garagatyi