text:
Can't compile deepstream_redaction_app

Hello,
On Jetson NANO, deepstream-4.0 is install via .deb package in the:
/opt/nvidia/deepstream/deepstream-4.0
All deepstream tests are working OK on the NANO.
When I try to compile the deepstream_redaction_app I am getting bellow error:
make
cc -c -o src/deepstream_redaction_app.o -DPLATFORM_TEGRA -I../apps-common/includes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 `pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11` src/deepstream_redaction_app.c
In file included from src/deepstream_redaction_app.c:28:0:
src/deepstream_app.h:34:10: fatal error: deepstream_app_version.h: No such file or directory
 #**include "deepstream_app_version.h**"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Makefile:56: recipe for target 'src/deepstream_redaction_app.o' failed
make: *** [src/deepstream_redaction_app.o] Error 1

Is something that I am missing?
Rgds,
Florin

The file in question should be at /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes/deepstream_app_version.h
Could you check that your Makefile includes the right paths, taking into consideration where your redaction app is placed?

Hello,
I have modified the Make to include the indicated folder like this:
CFLAGS+= -I /opt/nvidia/deepstream/deepstream-4.0/sources/includes -I /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes -I../apps-common/incl
udes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4

However now I have another error:
coredump@jetson-nano:~/projects/retinanet_for_redaction_with_deepstream$ make
cc -c -o src/deepstream_redaction_app.o -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream-4.0/sources/includes -I /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes -I../apps-common/includes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 `pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11` src/deepstream_redaction_app.c
cc -c -o src/deepstream_app_main.o -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream-4.0/sources/includes -I /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes -I../apps-common/includes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 `pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11` src/deepstream_app_main.c
cc -c -o src/deepstream_app_config_parser.o -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream-4.0/sources/includes -I /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes -I../apps-common/includes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 `pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11` src/deepstream_app_config_parser.c
src/deepstream_app_config_parser.c: In function ‘parse_config_file’:
src/deepstream_app_config_parser.c:218:12: **error**: too few arguments to function ‘parse_sink’
           !**parse_sink** (&config->
            ^~~~~~~~~~
In file included from src/deepstream_app_config_parser.c:24:0:
/opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes/deepstream_config_file_parser.h:131:1: note: declared here
 parse_sink (NvDsSinkSubBinConfig * config, GKeyFile * key_file, gchar * group, gchar * cfg_file_path);
 ^~~~~~~~~~
Makefile:56: recipe for target 'src/deepstream_app_config_parser.o' failed
make: *** [src/deepstream_app_config_parser.o] Error 1

I think this is a deepstream version issue. Please see the readme and choose the master branch if you've flashed your device with Jetpack4.2, or swithc over to the Jetpack4.3 branch.

I have flashed today the NANO with JetPack 4.3 and installed the Deepstream 4.0 from .deb file. I will try to install deepstream from tar.gz maybe will be diffrent.

If you flashed with JetPack4.3, then with DS4.0.2, please make sure to switch over to the correct branch after git clone. git checkout Jetpack4.3. The error comes from using the master branch in Jetpack4.3.

When you install JetPack4.3, there should be a tick box for installing DeepStream at the same time. That's by far the easiest way to get DeepStream on your Jetson edge device.

Deepstream is working fine with all samples.
I believe you have an issue with the deepstream_app_config_parser.c from where the error is coming.
Here is a diff with another similar file coming with deepstreem 4.0.2
coredump@jetson-nano:/opt/nvidia/deepstream/deepstream-4.0/sources/apps$ diff /opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/deepstream-app/deepstream_app_config_parser.c deepstream_app_config_parser.c
219,220c219
<           sink_bin_sub_bin_config[config->num_sink_sub_bins], cfg_file, *group,
<           cfg_file_path);
---
>           sink_bin_sub_bin_config[config->num_sink_sub_bins], cfg_file, *group);

Switching to Jetpack4.3 branch I am getting this error:
coredump@jetson-nano:/opt/nvidia/deepstream/deepstream-4.0/sources/apps$ make
cc -c -o src/deepstream_redaction_app.o -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes -I /opt/nvidia/deepstream/deepstream-4.0/sources/includes -I../apps-common/includes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 `pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11` src/deepstream_redaction_app.c
cc -c -o src/deepstream_app_main.o -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes -I /opt/nvidia/deepstream/deepstream-4.0/sources/includes -I../apps-common/includes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 `pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11` src/deepstream_app_main.c
cc -c -o src/deepstream_app_config_parser.o -DPLATFORM_TEGRA -I /opt/nvidia/deepstream/deepstream-4.0/sources/apps/apps-common/includes -I /opt/nvidia/deepstream/deepstream-4.0/sources/includes -I../apps-common/includes -I../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 `pkg-config --cflags gstreamer-1.0 gstreamer-video-1.0 x11` src/deepstream_app_config_parser.c
cc -o deepstream-redaction-app src/deepstream_redaction_app.o src/deepstream_app_main.o src/deepstream_app_config_parser.o -L/opt/nvidia/deepstream/deepstream-4.0/lib/ -lnvdsgst_meta -lnvds_meta -lnvdsgst_helper -lnvds_utils -lm -lgstrtspserver-1.0 -lgstrtp-1.0 -Wl,-rpath,/opt/nvidia/deepstream/deepstream-4.0/lib/ `pkg-config --libs gstreamer-1.0 gstreamer-video-1.0 x11`
src/deepstream_redaction_app.o: In function `watch_source_status':
deepstream_redaction_app.c:(.text+0x64): undefined reference to `reset_source_pipeline'
deepstream_redaction_app.c:(.text+0x68): undefined reference to `reset_source_pipeline'
src/deepstream_redaction_app.o: In function `bus_callback':
deepstream_redaction_app.c:(.text+0x428): undefined reference to `reset_source_pipeline'
deepstream_redaction_app.c:(.text+0x42c): undefined reference to `reset_source_pipeline'
src/deepstream_redaction_app.o: In function `bus_sync_handler':
deepstream_redaction_app.c:(.text+0x900): undefined reference to `set_source_to_playing'
deepstream_redaction_app.c:(.text+0x904): undefined reference to `set_source_to_playing'
src/deepstream_redaction_app.o: In function `add_and_link_broker_sink':
deepstream_redaction_app.c:(.text+0x17dc): undefined reference to `link_element_to_tee_src_pad'
src/deepstream_redaction_app.o: In function `create_processing_instance':
deepstream_redaction_app.c:(.text+0x1908): undefined reference to `create_sink_bin'
deepstream_redaction_app.c:(.text+0x1984): undefined reference to `create_osd_bin'
src/deepstream_redaction_app.o: In function `create_common_elements':
deepstream_redaction_app.c:(.text+0x1d5c): undefined reference to `create_secondary_gie_bin'
deepstream_redaction_app.c:(.text+0x1efc): undefined reference to `create_tracking_bin'
deepstream_redaction_app.c:(.text+0x20ac): undefined reference to `create_primary_gie_bin'
src/deepstream_redaction_app.o: In function `create_pipeline':
deepstream_redaction_app.c:(.text+0x28ec): undefined reference to `create_multi_source_bin'
deepstream_redaction_app.c:(.text+0x295c): undefined reference to `set_streammux_properties'
deepstream_redaction_app.c:(.text+0x2b2c): undefined reference to `create_tiled_display_bin'
deepstream_redaction_app.c:(.text+0x2f48): undefined reference to `create_dsexample_bin'
deepstream_redaction_app.c:(.text+0x325c): undefined reference to `enable_perf_measurement'
src/deepstream_redaction_app.o: In function `pause_pipeline':
deepstream_redaction_app.c:(.text+0x3998): undefined reference to `pause_perf_measurement'
src/deepstream_redaction_app.o: In function `resume_pipeline':
deepstream_redaction_app.c:(.text+0x3a84): undefined reference to `resume_perf_measurement'
src/deepstream_app_main.o: In function `all_bbox_generated':
deepstream_app_main.c:(.text+0x160): undefined reference to `str_replace'
deepstream_app_main.c:(.text+0x188): undefined reference to `str_replace'
deepstream_app_main.c:(.text+0x1e4): undefined reference to `str_replace'
deepstream_app_main.c:(.text+0x20c): undefined reference to `str_replace'
src/deepstream_app_config_parser.o: In function `parse_tests':
deepstream_app_config_parser.c:(.text+0x78): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x7c): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x150): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x154): undefined reference to `APP_CFG_PARSER_CAT'
src/deepstream_app_config_parser.o: In function `parse_app':
deepstream_app_config_parser.c:(.text+0x308): undefined reference to `APP_CFG_PARSER_CAT'
src/deepstream_app_config_parser.o:deepstream_app_config_parser.c:(.text+0x30c): more undefined references to `APP_CFG_PARSER_CAT' follow
src/deepstream_app_config_parser.o: In function `parse_app':
deepstream_app_config_parser.c:(.text+0x554): undefined reference to `get_absolute_file_path'
deepstream_app_config_parser.c:(.text+0x584): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x588): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x62c): undefined reference to `get_absolute_file_path'
deepstream_app_config_parser.c:(.text+0x65c): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x660): undefined reference to `APP_CFG_PARSER_CAT'
src/deepstream_app_config_parser.o: In function `parse_config_file':
deepstream_app_config_parser.c:(.text+0x7ec): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x7f0): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0x800): undefined reference to `APP_CFG_PARSER_CAT'
src/deepstream_app_config_parser.o:deepstream_app_config_parser.c:(.text+0x804): more undefined references to `APP_CFG_PARSER_CAT' follow
src/deepstream_app_config_parser.o: In function `parse_config_file':
deepstream_app_config_parser.c:(.text+0xa28): undefined reference to `parse_source'
deepstream_app_config_parser.c:(.text+0xab8): undefined reference to `parse_streammux'
deepstream_app_config_parser.c:(.text+0xb04): undefined reference to `parse_osd'
deepstream_app_config_parser.c:(.text+0xb64): undefined reference to `parse_gie'
deepstream_app_config_parser.c:(.text+0xbb4): undefined reference to `parse_tracker'
deepstream_app_config_parser.c:(.text+0xc6c): undefined reference to `parse_gie'
deepstream_app_config_parser.c:(.text+0xd74): undefined reference to `parse_sink'
deepstream_app_config_parser.c:(.text+0xe10): undefined reference to `parse_tiled_display'
deepstream_app_config_parser.c:(.text+0xe5c): undefined reference to `parse_dsexample'
deepstream_app_config_parser.c:(.text+0xed0): undefined reference to `APP_CFG_PARSER_CAT'
deepstream_app_config_parser.c:(.text+0xed4): undefined reference to `APP_CFG_PARSER_CAT'
collect2: error: ld returned 1 exit status
Makefile:59: recipe for target 'deepstream-redaction-app' failed
make: *** [deepstream-redaction-app] Error 1

Just to make sure, are you running make in the folder /opt/nvidia/deepstream/deepstream-4.0/sources/apps/retinanet_for_redaction_with_deepstream?
On Xavier, running the following command returns no difference:
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-4.0/sources/apps/retinanet_for_redaction_with_deepstream$ diff src/deepstream_app_config_parser.c ../sample_apps/deepstream-app/deepstream_app_config_parser.c
All I've done before make is git clone and git checkout.
We've only tested on Xavier so far, when time allows we will also test on Nano.

@jin-nvidia is OK now, THX

work with the jetpack4.3 branch
copy the retinanet_for_redaction_with_deepstream directory and not only the Make and /src folder in the /opt/nvidia/deepstream/deepstream-4.0/sources/apps

All good now, thank you!

authors: james-nvidia jin-nvidia ticlazau


text:
Introduce and apply a basic version of the Column abstraction

Depends on https://github.com/NVIDIA-Merlin/core/pull/163
This PR takes some early steps to integrate a single array version of a Column class inside DictArray and get the Systems tests back to passing. Since this is a partial version of Column, we're not yet reaping the benefits as far as cleaning up the code goes and there's a lot that's still gross in this version of the code. We think it's at least "not worse" though, so in theory this small PR might be merge-able. There's a lot of further refactoring to do, so we'll have to figure out what size we want the PRs to be.

rerun tests

source: gharchive/pull-request

repo: NVIDIA-Merlin/systems

license_type: permissive

license_source: github-api


text:
DIGITS ERROR: Cannot use GPU in CPU-only Caffe: check mode.

Hello,
I have been trying to search what is the reason for this error:
DIGITS ERROR: Cannot use GPU in CPU-only Caffe: check mode.
in the already submitted issues,  but I could not find the solution up to now. I apologize if I missed it if it has already been solved.
Basically I was trying to do a Semantic Segmentation experiment based on this tutorial. https://github.com/NVIDIA/DIGITS/tree/master/examples/semantic-segmentation
however, I ended up having an error like this:

I have checked my caffe installation, and I believe I have built it with GPU mode, or "CPU_ONLY : OFF" in the cmake. I also have checked the Makefile.config (if it is necessary), and the "CPU_ONLY:=1" line was also commented, meaning it is in GPU mode.

Since I still do not know what is the cause of this problem, I really appreciate any help and solution for this issue.
Thank you.

Did you tell DIGITS to use this Caffe build?
https://github.com/NVIDIA/DIGITS/blob/digits-5.0/docs/Configuration.md

If this what you meant:

I believe I did.
But now I realize that in my cmake configuration on my previous image, the CUDA dependencies is in NO status.
How can I fix this?

Now I am doing cmake by this: cmake -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-8.0 ..
And the CUDA is detected now:

Now in the recompiling status.

All right. Finally the problem solved. Thank you @lukeyeager for the help and quick response :).

source: gharchive/issue

authors: lukeyeager mrasyid

repo: NVIDIA/DIGITS


text:
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:   435 |         function(_Functor&& __f)       |

Description
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
  435 |         function(_Functor&& __f)
      |

Reproduced Steps
g++ 10.3
CUDA 11.5
Ubuntu 22.04

did you have solved it? I meet the same problem

Same issue here a well when compiling MNN 2.6.0 (https://github.com/alibaba/MNN/releases) with CUDA support.
g++ 11.4.0
CUDA 11.5
I am reporting this issue in their repo too

authors: 1773226512 cappelletto lucasjinreal

repo: NVIDIA/FasterTransformer

license_source: bigquery


text:
build failed with tf-op

Branch/Tag/Commit
v5.3
Docker Image Version
nvcr.io/nvidia/pytorch:22.12-py3
GPU name
T4
CUDA Driver
NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.8
Reproduced Steps
build with "cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DBUILD_TF2=ON -DTF_PATH=/usr/local/lib/python3.8/dist-packages/tensorflow/ .." and make
error message:
/workspace/FasterTransformer/src/fastertransformer/kernels/disentangled_attention_kernels.cu(382): error: more than one operator "=" matches these operands:
            function "__nv_bfloat16::operator=(float)"
/usr/local/cuda/include/cuda_bf16.hpp(178): here
            function "__nv_bfloat16::operator=(double)"
/usr/local/cuda/include/cuda_bf16.hpp(181): here
            operand types are: __nv_bfloat16 = int
          detected during instantiation of "void fastertransformer::disentangled_attention_kernel<TDataType,tTileSize,tBlockDimY>(TDataType *, TDataType *, const TDataType *, const TDataType *, int32_t, int32_t, int32_t) [with TDataType=__nv_bfloat16, tTileSize=32, tBlockDimY=8]"
(407): here

/workspace/FasterTransformer/src/fastertransformer/kernels/disentangled_attention_kernels.cu(382): error: more than one operator "=" matches these operands:
            function "__nv_bfloat16::operator=(float)"
/usr/local/cuda/include/cuda_bf16.hpp(178): here
            function "__nv_bfloat16::operator=(double)"
/usr/local/cuda/include/cuda_bf16.hpp(181): here
            operand types are: __nv_bfloat16 = int
          detected during instantiation of "void fastertransformer::disentangled_attention_kernel<TDataType,tTileSize,tBlockDimY>(TDataType *, TDataType *, const TDataType *, const TDataType *, int32_t, int32_t, int32_t) [with TDataType=__nv_bfloat16, tTileSize=64, tBlockDimY=4]"
(407): here

2 errors detected in the compilation of "/workspace/FasterTransformer/src/fastertransformer/kernels/disentangled_attention_kernels.cu".

@jackzhou121 ,
I had same issues, and resolved them using following patch. This code casts the value into a double, ensuring that the compiler always chooses double operands for the "=" operator.
--- a/src/fastertransformer/kernels/disentangled_attention_kernels.cu
+++ b/src/fastertransformer/kernels/disentangled_attention_kernels.cu
@@ -379,7 +379,7 @@ __global__ void disentangled_attention_kernel(TDataType*       result,
 #ifdef ENABLE_BF16
         else if constexpr (std::is_same<TDataType, __nv_bfloat16>::value) {
             // bf16
-            res = __hadd(res0, __hadd(res1, T[threadIdx.x][ty + threadIdx.y]));
+            res = static_cast<double>(__hadd(res0, __hadd(res1, T[threadIdx.x][ty + threadIdx.y])));
         }
 #endif


text:
Add nemo_curator.__version__, --version, and -v displays

Closes https://github.com/NVIDIA/NeMo-Curator/issues/132.

Thanks @ryantwolf ! Added your suggestions and tested a pip release. Everything looks okay from my end.
cc @ayushdg

repo: NVIDIA/NeMo-Curator


text:
Joint CTC Attention model not converging

I am testing the joint ctc attention model with the config available in example_configs/speech2text/jca_large_8gpus.py. I have tried a range of learning rates (1e-6 to 1e-2 - small batch size and 1 gpu) but models don't seem to converge. Any recipe on training this architecture?

The current Joint CTC - Attention model in OpenSeq2Seq was unfinished work, and hence has no documentation in the docs. So unfortunately, we do not have good suggestions for this model.
We have a similar Conv/TDNN -> RNN architecture as part of NeMo which you can find here: garnet.py  and corresponding config here: garnet.yaml.
Hopefully these parameters can help guide towards a working implementation inside OpenSeq2Seq.

authors: blisc tayciryahmed


text:
Import Error: ModuleNotFoundError: No module named 'tensorrt_llm.lora_manager'

System Info
started receiving this error after rebuilding image today - that I believe picked up #1274 . (using pip3 install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com).
Traceback (most recent call last):
  File "/TensorRT-LLM/examples/enc_dec/run.py", line 32, in <module>
    from tensorrt_llm.lora_manager import LoraManager
ModuleNotFoundError: No module named 'tensorrt_llm.lora_manager'

Who can help?
kaiyux
Information

[X] The official example scripts
[ ] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction
follow the readme here
https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/enc_dec/README.md#run
to run line:
https://github.com/NVIDIA/TensorRT-LLM/blob/4bb65f216f2f1dbddb022f4fdd8925c2856baa58/examples/enc_dec/run.py#L32
Expected behavior
import sucessfull.
actual behavior
import fails.
additional notes

same issue. is there any known workaround?

same here. trying this link.
https://console.brev.dev/notebook/streamingllm-tensorrt-llm

Could you share the tensorrt_llm version you use? I try to reproduce but fail.
bhsueh@xxx:/tmp/TensorRT-LLM$ python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorrt_llm.lora_manager import LoraManager
[TensorRT-LLM] TensorRT-LLM version: 0.9.0.dev2024031200

I'm using this version.

import tensorrt_llm
No protocol specified
[TensorRT-LLM] TensorRT-LLM version: 0.8.0

I had this issue when was using 0.9.0 version installed from pip over 0.8.0 version of this repo. Updating repo to 0.9.0 fixed it.

Hi @hademircii do u still have further issue or question now? If not, we'll close it soon.

authors: RankoR byshiue hademircii jagadeesh-k-ti luvwinnie nv-guomingz

repo: NVIDIA/TensorRT-LLM


text:
Where is download for Tensorrt 9.1.0.4 for Windows?

Hi,
in "https://github.com/NVIDIA/TensorRT-LLM/tree/release/0.5.0/windows"
you mention download "Download and unzip TensorRT 9.1.0.4" but it's not available..
also in https://github.com/NVIDIA/TensorRT/pull/3395/commits/b8adcfa894b241261f46f21d0dcc70b54a576263 we see how to download for Linux at least:
TensorRT 9.1.0.4 for CUDA 12.2, Linux x86_64
but not yet for Windows..
opening here so get notified when available..
thanks..

If you don’t mind, I’m closing that issue and we will discuss that issue in https://github.com/NVIDIA/TensorRT-LLM/issues/18.

authors: jdemouth oscarbg


text:
Fix rng_state issue and minor compiler warning

This PR fixes two issues:

[ ] rng_state being overwritten by newly created auxiliary tensors for arbitrary_seqlen backend of fused attn, affecting results in later iterations of training
[ ] variable "valid_store" being declared but never referenced

/te-ci

@ksivaman see bug 4201442

9471202 passed, should be ready to merge

repo: NVIDIA/TransformerEngine


text:
nvidia-container-cli: mount error: failed to add device rules: unable to generate new device filter program from existing programs: unable to create new device filters program: load program: invalid argument: last insn is not an exit or jmp

After I updated the following packages:
nvidia-docker2:amd64 (2.10.0-1, 2.11.0-1)
libnvidia-container1:amd64 (1.9.0-1, 1.10.0-1)
libnvidia-container-tools:amd64 (1.9.0-1, 1.10.0-1)
nvidia-container-toolkit:amd64 (1.9.0-1, 1.10.0-1)

the following command:
docker run --rm --gpus all --runtime nvidia nvidia/cuda:11.0-base nvidia-smi

started to fail with the the following error:
docker: Error response from daemon: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: mount error: failed to add device rules: unable to generate new device filter program from existing programs: unable to create new device filters program: load program: invalid argument: last insn is not an exit or jmp
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0: unknown.

Before the update I had no-cgroups = true in /etc/nvidia-container-runtime/config.toml (which I added based on this discussion).
Ater the update it says:
[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
#no-cgroups = false
#user = "root:video"
ldconfig = "@/sbin/ldconfig.real"

System info
Command line: BOOT_IMAGE=/vmlinuz-5.11.0-16-generic root=/dev/mapper/vgkubuntu-root ro quiet splash vt.handoff=7 nvidia-drm.modeset=1

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04 LTS
Release:        22.04
Codename:       jammy

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:08:00.0  On |                  N/A |
|  0%   47C    P5    25W / 370W |   2110MiB / 10240MiB |     17%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   35C    P8    15W / 220W |    205MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3080 (UUID: GPU-22...)
GPU 1: NVIDIA GeForce RTX 3060 Ti (UUID: GPU-37...)

$ dpkg -l | grep systemd
ii  systemd                                       249.11-0ubuntu3.1                           amd64        system and service manager
ii  systemd-container                             249.11-0ubuntu3.1                           amd64        systemd container/nspawn tools
ii  systemd-sysv                                  249.11-0ubuntu3.1                           amd64        system and service manager - SysV links
ii  systemd-timesyncd                             249.11-0ubuntu3.1                           amd64        minimalistic service to synchronize local time with NTP servers

Workaround
The following workaround worked for me:

restore no-cgroups = true in /etc/nvidia-container-runtime/config.toml
start docker in this way: docker run --rm --gpus all --runtime nvidia --device /dev/nvidia0 --device /dev/nvidia1 --device /dev/nvidiactl --device /dev/nvidia-modeset --device /dev/nvidia-uvm nvidia/cuda:11.0-base nvidia-smi

@DimanNe depending on how the packages are installed, the config file may be overwritten by the package manager version.
As a matter of interest, which version were you upgrading from?

@elezar

the config file may be overwritten by the package manager version

Yeah, it was overwritten, you are right. It is not a problem/questions itself. I was just trying to explain why this particular upgrade caused it.

As a matter of interest, which version were you upgrading from?

At the top of my message, according to apt logs:
nvidia-docker2:amd64 (2.10.0-1, 2.11.0-1)
libnvidia-container1:amd64 (1.9.0-1, 1.10.0-1)
libnvidia-container-tools:amd64 (1.9.0-1, 1.10.0-1)
nvidia-container-toolkit:amd64 (1.9.0-1, 1.10.0-1)

Ah, sorry, I missed that you had both versions.
I would expect the same behaviour with the 1.9.0 and no-cgroups = false. The 1.8.0, 1.8.1, and 1.9.0 releases include changes to handle cgroupv2 and would generate an error like you're seeing. There were no cgroup-related changes in the 1.10.0 release.
@klueska may have more insights into what could be causing the specific error.

I would expect the same behaviour with the 1.9.0 and no-cgroups = false

Agree. I encountered same (similar) issue with one of the previous versions/updates, and then disabled cgroups via no-cgroups = true... So, it not a problem with this particular update actually...

@klueska Any news? Am I doing something wrong?

I've been hitting an issue with the same symptoms working on enabling cgroup v2 for Bottlerocket. In this case, I found the kernel's eBPF JIT hardening to contribute to the problem. Quoting from our issue:

Bottlerocket enables eBPF JIT hardening for both privileged and unprivileged users by default. One of the hardening measures is a constant blinding pass over eBPF bytecode loaded into the kernel, which applies slight modifications to the bytecode that preserve semantics but decrease attacker control over possible instruction sequences ending up in executable kernel memory. As a side effect, programs that have been blinded cannot be dumped to user space again.
It is the inability to dump eBPF programs to user space that is causing problems for libnvidia-container-go. When allowing GPU access to a container, it prepends new filters to the existing program (I assume put in place for the cgroup by runc). If constant blinding has been applied to the program, libnvidia-container-go will prepend the new filter to a buffer of zeros. "All zeros" happens to be a valid eBPF instruction encoding (loading a constant 32-bit 0 into register 0), so when the modified program is loaded back into the kernel, the eBPF verifier will only notice there's code path that does not explicitly terminate. This results in the error seen above, with the new device filter program being rejected on the grounds of "last insn is not an exit or jmp".

@DimanNe, do you happen to have the eBPF JIT hardening measures enabled as well? You can check by running sudo cat /proc/sys/net/core/bpf_jit_harden.

@markusboehme thx for the hint.
Reducing the value of that sysctl from 2 to 1 fixed the issue for me.
I had no luck running with the cgroups off either, so this was very helpful.
@elezar looks like there's some "bug" with the ebpf generation that makes the hardened checker reject it.

The default linux-hardened kernel users are affected as well, coming from archlinux. A quick sudo sysctl -w net.core.bpf_jit_harden=1 (that won't survive a reboot, use a sysctl config for that) is a nice workaround for now, thanks to @markusboehme & @Ongy

authors: DimanNe Ongy bioluks elezar markusboehme

repo: NVIDIA/libnvidia-container


text:
Remove gitlab golang steps

Since the GitLab pipelines are triggered on pull from gitlab the golang-specific pipeline steps are not required.

Testing the changes to the pipeline here: https://gitlab.com/nvidia/cloud-native/mig-parted/-/pipelines/1155207840

repo: NVIDIA/mig-parted


text:
Recursive log dirs?

pip install runx installs v.0.0.11.
This breaks the log functionality somehow, and I'm seeing:
Copying codebase to ../logs/sweep/spectral-okapi_2021.07.12_15.16 ...
Running job sweep_spectral-okapi_2021.07.12_15.16
cd ../logs/sweep/spectral-okapi_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.01 --solver first --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/spectral-okapi_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Copying codebase to ../logs/sweep/teal-trogon_2021.07.12_15.16 ...
Running job sweep_teal-trogon_2021.07.12_15.16
cd ../logs/sweep/teal-trogon_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.01 --solver second --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/teal-trogon_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Copying codebase to ../logs/sweep/garrulous-wombat_2021.07.12_15.16 ...
Running job sweep_garrulous-wombat_2021.07.12_15.16
cd ../logs/sweep/garrulous-wombat_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.02 --solver first --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/garrulous-wombat_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Copying codebase to ../logs/sweep/lavender-prawn_2021.07.12_15.16 ...
Running job sweep_lavender-prawn_2021.07.12_15.16
cd ../logs/sweep/lavender-prawn_2021.07.12_15.16/code; PYTHONPATH=  python testrunx.py --lr 0.02 --solver second --logdir ..//logs
/bin/sh: 1: cd: can't cd to ../logs/sweep/lavender-prawn_2021.07.12_15.16/code
python: can't open file 'testrunx.py': [Errno 2] No such file or directory

Looking at ../logs, I'm seeing recursive paths like this one:
../logs/sweep/logs/sweep/logs/sweep/logs/sweep/complex-mongrel_2021.07.12_15.06/submit_cmd.sh

Here is my python script (testrunx.py)
import sys
print(sys.argv)

And here is my sweep.yml:
CMD: 'python testrunx.py'

HPARAMS:
        lr: [0.01, 0.02]
        solver: ['first', 'second']
        logdir: '..//logs'

Installing version 0.0.5 fixes the problem.

Hi, did you create a .runx file in which you define a LOGROOT? I would do so, and make the path absolute. Maybe try that.

Yep, I tried to do so, and made the path absolute now as well. When running (on v.0.0.11), I'm getting something similar to this:
 python -m runx.runx sweep.yml -i
Copying codebase to /home/.../onordesjo/git/runxtest/sweep/economic-lyrebird_2021.07.13_13.17 ...
Traceback (most recent call last):
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/site-packages/runx/runx.py", line 402, in <module>
    main()
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/site-packages/runx/runx.py", line 395, in main
    run_experiment(args.exp_yml)
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/site-packages/runx/runx.py", line 390, in run_experiment
    run_yaml(experiment, runroot)
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/site-packages/runx/runx.py", line 345, in run_yaml
    copy_code(logdir, runroot, code_ignore_patterns)
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/site-packages/runx/runx.py", line 216, in copy_code
    copytree(runroot, tgt_code_dir, ignore=code_ignore_patterns)
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/shutil.py", line 547, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/home/.../onordesjo/miniconda3/envs/sequences/lib/python3.8/shutil.py", line 504, in _copytree
    raise Error(errors)
shutil.Error: [('/home/.../onordesjo/git/runxtest/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17/code/sweep/economic-lyrebird_2021.07.13_13.17...
...
economic-lyrebird_2021.07.13_13.17/code/sweep'")]

It's not clear to me why there's ... in your LOGROOT path. Maybe you could show your .runx file?

authors: ajtao ollenordesjo

repo: NVIDIA/runx


text:
[DO NOT REVIEW] Dummy test

Signed-off-by: Peixin Li pxli@nyu.edu

build

build

build

repo: NVIDIA/spark-rapids-jni


text:
Add support to generate nested data_gen in integration tests

We need a way to quickly generate nested data. Right now we should keep it limited to 3-level nesting simply because if nesting gets too deep, the data generation could potentially take a very long time.

What exactly did you have in mind?  It is fairly simple to do something like
ArrayGen(ArraGen(ArraGen(StringGen)))

We also have some pre-built options like nested_array_gens_sample that can be extended with deeper options.

Yeah this is very straight forward and fairly simple but it gets complicated when you have structs nested within structs.
e.g.
StructGen([['child0',
   StructGen([['child0',
      StructGen([['child0',
         StructGen([['child0',
            StructGen([['child0', DecimalGen(7, 2)]])]])]])]])], ['child1', IntegerGen()]], nullable=False)

The problem gets more cumbersome if you want each level to have a different name to circumvent the SPARK-35108

That issue happens in the result of a computation.  (Spark result going back to Python) Not in the data that is generated.  What we need then is an API that can "fix" a nested dataframe so we don't run into those problems.  Not one that changes how the data is generated.

repo: NVIDIA/spark-rapids


text:
[FEA] Limit the numbers of rows being read in ParquetCachedBatchSerializer

Is your feature request related to a problem? Please describe.
It was found that PCBS tries to read in too much data which may cause an over run of the host memory
Describe the solution you'd like
Read in 10k rows at a time like the default CPU parquet implementation
Describe alternatives you've considered
We can increase the allocated memory to Spark

At the moment, the team is exploring whether Parquet or another serializer format is preferable for caching.  Once that is known we can determine the priority of this issue.

authors: razajafri sameerz


text:
[FEA] Create a BigDataChecking tool

Is your feature request related to a problem? Please describe.
https://github.com/NVIDIA/spark-rapids/pull/8706 adds a tool to generate large data sets for scale testing, and possibly some performance testing for very difficult corner cases.  But ultimately we need to also verify that the results after running that generated data through a query are correct. TPC- benchmarks provide a validation tool that can be used to check if the results are correct, but for us, our definition of correct is that it matches Spark on the CPU. Warts and all.
The other problem is that the data produced by scale testing can be very very large (both in terms of rows, but also column count, and nesting). So we need to come up with a way to validate that the results are correct in a way that is going to scale. This itself may be a really massive scale test too because we are likely to double the number of rows and columns by trying to compare two large data sets.
One way to do this is to group by all of the columns, do a count on them (so that all duplicates are removed), for both the expected data set and the test data set. Then to do two anti-joins on them. A left-anti-join to know if there are rows on the LHS that are not in the right, and a right-anti-join to know if there are rows in the RHS that are not in the left.
val expected_count = expected.groupBy(expected.schema.names: _*).count
val test_count = test.groupBy(test.schema.names: _*).count
val just_in_expected = expected_count.join(test_count, expected_count.schema.names.toSeq, "left_anti").withColumn("from", lit(0))
val just_in_test = test_count.join(expected_count, expected_count.schema.names.toSeq, "left_anti").withColumn("from", lit(1))
just_in_expected.union(just_in_test).show()

This works if all of the values produced will be bit for bit identical to those on the CPU, and are types that Spark can join on. For floating point types, especially with aggregations that is not guaranteed.  Perhaps that is good enough. We just avoid floats in our scale tests and we are good, but I am not sure we can do that especially if we want to reproduce issues from customer queries.
We might need to normalize maps to make this work and turn them into a combinations of structs and arrays with the data sorted by the key.
For floating point values or anywhere that we would want to do an approximate match, I just don't know of a good way to do it that is distributed and scales well.
If we try to normalize the floating point values by rounding of any kind there are inflection points where if we are off by a little the rounding would just exacerbate the difference. We could provide a UDF to do a fuzzy match on the floating point values as a part of the anti-join, but we could not do the group by any more to deal with duplicates because we might not even be self consistent, and aggregations don't support a fuzzy match.
The only option I can think of is to sort all the data from both data sets and produce a row number.  This takes a bit of work with monotonically_increasing_id and partition_id, along with writing out the intermediate data, but it is possible.  After that we can join on the row numbers and do a fuzzy match in a UDF for each column that contains floating point values, or maps. Not ideal but it would work.

It should be noted that something similar to this already exists in the tools repo.
https://github.com/NVIDIA/spark-rapids-tools/tree/dev/data_validation
I'm not 100% sure if we want to have two tools or just one. But the code is not that complicated. The data_validation tool is not great. It requires a primary key, which should not be a requirement. It does not handle maps, nulls, duplicate rows, or floating point approximate matches correctly.
The above example can be updated to handle nulls by using the null equals operator '<=>' as the join condition for the anti-join columns.  Maps can be replaced with a map_entries that are array_sorted. And for now we are just going to punt on floats/doubles in both cases and not include them in our scale testing suite until this can support doing a fuzzy check for them.

@revans2 I'm building pipelines to run preliminary scale test on our internal cluster and I got some numbers for this issue.
with scale_factor=100, complexity=300, I got the data at size
134.1 G  268.1 G  /data/scale-test/SCALE_100_300_parquet_1.0.0_41

GPU output size (0 means failed queries)
1.2 G     2.4 G    /data/scale-test/output/q10_1
46.2 G    92.4 G   /data/scale-test/output/q11_1
0         0        /data/scale-test/output/q12_1
46.2 G    92.4 G   /data/scale-test/output/q13_1
17.0 G    34.1 G   /data/scale-test/output/q14_1
12.9 G    25.7 G   /data/scale-test/output/q15_1
39.3 K    78.7 K   /data/scale-test/output/q16_1
9.8 G     19.7 G   /data/scale-test/output/q17_1
5.8 G     11.6 G   /data/scale-test/output/q18_1
38.5 K    77.1 K   /data/scale-test/output/q19_1
645.0 G   1.3 T    /data/scale-test/output/q1_1
38.5 K    77.1 K   /data/scale-test/output/q20_1
0         0        /data/scale-test/output/q21_1
11.1 G    22.2 G   /data/scale-test/output/q22_1
19.5 K    39.1 K   /data/scale-test/output/q23_1
16.5 M    33.1 M   /data/scale-test/output/q24_1
290.7 K   581.3 K  /data/scale-test/output/q25_1
775.5 M   1.5 G    /data/scale-test/output/q26_1
1.7 G     3.5 G    /data/scale-test/output/q27_1
1004.1 M  2.0 G    /data/scale-test/output/q28_1
1.2 M     2.4 M    /data/scale-test/output/q29_1
644.9 G   1.3 T    /data/scale-test/output/q2_1
1.4 G     2.9 G    /data/scale-test/output/q30_1
603.6 M   1.2 G    /data/scale-test/output/q31_1
841.9 K   1.6 M    /data/scale-test/output/q32_1
174.3 M   348.6 M  /data/scale-test/output/q33_1
191.7 M   383.3 M  /data/scale-test/output/q34_1
70.6 M    141.3 M  /data/scale-test/output/q35_1
26.8 M    53.5 M   /data/scale-test/output/q36_1
27.1 G    54.1 G   /data/scale-test/output/q37_1
26.5 G    53.1 G   /data/scale-test/output/q38_1
4.0 G     7.9 G    /data/scale-test/output/q39_1
644.9 G   1.3 T    /data/scale-test/output/q3_1
0         0        /data/scale-test/output/q40_1
15.7 M    31.4 M   /data/scale-test/output/q41_1
12.1 G    24.2 G   /data/scale-test/output/q4_1
12.1 G    24.2 G   /data/scale-test/output/q5_1
1.2 G     2.4 G    /data/scale-test/output/q6_1
1.2 G     2.4 G    /data/scale-test/output/q7_1
1.2 G     2.4 G    /data/scale-test/output/q8_1
1.2 G     2.4 G    /data/scale-test/output/q9_1

Lots of the output are in large size which I doubt the validation will take a very long time. Should we perform like what NDS queries do? they use LIMIT for all the queries that the output are quite small.

Limit has problems, even if there is an order by before it there can be ambiguity in the order of the results.  I would much rather do something like the following.
import org.apache.spark.sql.DataFrame

def compare(left: DataFrame, right: DataFrame): DataFrame = {
  val leftCount = left.groupBy(left.columns.map(col(_)): _*).count
  val rightCount = right.groupBy(right.columns.map(col(_)): _*).count
  val joinOn = leftCount.columns.map(c => leftCount(c) <=> rightCount(c)).reduceLeft(_ and _)
  val onlyRight = rightCount.join(leftCount, joinOn, joinType="left_anti").withColumn("_in_column", lit("right"))
  val onlyLeft = leftCount.join(rightCount, joinOn, joinType="left_anti").withColumn("_in_column", lit("left"))
  onlyRight.union(onlyLeft)
}

This should run a query that compares the two dataframes. There is a lot we can do to improve it. It does not support Maps, and floating-point is not going to be compared very well, but that should be good enough for the current scale tests and should be a good starting point.
It would also be nice to explicitly verify that the schemas match before running the query and if we see a map to throw an exception, or if you see floating point to output a warning.


text:
Try to make degenerative joins here impossible for these tests [databricks]

Fixes #10147 and fixes #10548.
This should fix the test failures for DPP when AQE is enabled and optimizes out joins that result in empty relations. First, it uses a randomly generated filter value used for all rows that prevents the filter predicate from failing, and also ensures that the ex_key predicate always uses a value that exists in the table.

build

build

authors: NVnavkumar jlowe


text:
Add GpuMapConcat support for nested type keys.

Signed-off-by: remzi 13716567376yh@gmail.com
This is a subtask of #5559.
The GpuMaxConcat could support array typed keys and struct typed keys.

build

The changes look good, but I think we should also add integration tests so that we test this on all supported platforms.


text:
wp.mesh_query_point throws access violation

I think there are some nuances to this issue and this is my best guess.
As the title stands, I could see an argument for saying its not a bug.  However, there should be a better error.  Or instead, when creating the mesh, find a way to check if it is invalid upon creation.
even though I turn on debug mode and breakpoints in vscode, the only thing I can see is access violoation on the original launch.  it was by trial and error and commenting out things until I got to where it seems to happen.

The issue was that the positions of the mesh were not on the same device as the kernel (or the point being referenced).

repo: NVIDIA/warp


text:
DSD-1636: React18 for Basic Elements

Fixes JIRA ticket DSD-1636
This PR does the following:

Updates Card, FeaturedContent, and Hero components to use the Chakra 2.8 updated API for their theme object.
No functional changes, only how the theme file is defined.

How has this been tested?

local Storybook

Accessibility concerns or updates

n/a

Checklist:

[ ] I have updated the Storybook documentation accordingly.
[ ] I have added relevant accessibility documentation for this pull request.
[ ] All new and existing tests passed.

Front End Review:

[ ] Review the Vercel preview deployment once it is ready.

@EdwinGuzman The work on the Card component is not going so smoothly and I am wondering if my struggles are stemming from the use of child components.  Do you think that would add additional complexities?  I was going to take a look at the Notification component, which also uses child components.

How is it not going smoothly? From a quick glance it looks ok but I may be missing the difficulty in  upgrading the Card styles.

repo: NYPL/nypl-design-system


text:
Pods do not update with push

When drone says the kubectl has been applied it seems my pods stay the same and never update.

I'm happy to help, but you need to provide detailed information. Based on the issues you've opened so far, it seems like you are missing basic knowledge of kubernetes. Have you read the kubernetes documentation?

I do admit that I am a beginner with kubernetes and ready most of the documentation, but also lack experience to articulate what's actually happening.  I meant to say does the plug in preform rolling updates. Deploying new software without any down time.

All the plugin does is run kubectl apply -f .kube.yml,.kube.sec.yml. Yes, that does a rolling update if you've updated a Deployment.
Please read the kubernetes documentation, and if you feel like something is wrong with this plugin, open an issue with the relevant configuration and logs.

repo: NYTimes/drone-gke


text:
Fit to CPUE plot

This plot should by default plot the posterior predictive distribution (perhaps with the posterior distribution as an inner ribbon)

This has been implemented, so keep in mind this now plots the posterior predictive distribution, not the posterior distribution for CPUE.

repo: NZRLIC/RLPlots

license: mit


text:
🛑 Damien is down

In 55adb92, Damien (https://damien-doussaud.com/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Damien is back up in 3bc24b7.

repo: Namide/upptime

license: MIT


text:
🛑 Sport API is down

In a3987b1, Sport API (https://api.sport.namide.com/api/v1) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Sport API is back up in 97f96f4.


text:
Updated build scripts

The following changes where made

Added the .dotnet folder to .gitignore
Changed build.ps1 to pass $args instead of, the invalid, $argumentList
Added #!/usr/bin/env bash -xe to build.sh
Converted CRLF to LF in build.sh

Hi @thecodejunkie, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!
The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.
TTYL, DNFBOT;

authors: dnfclas thecodejunkie


text:
Performance tweaks of the modelbinder

Introduce caching for things like property settings and lookups

Let's reopen this if/when this happens

repo: NancyFx/Nancy


text:
Set size changed during iteration -- is this a problem

Ubuntu 16.04 LTS
Thanks,
BrianR (author of Tinker chess engine)
brianr@Tinker-Ubuntu:~/alphagozero$ python3 main.py
Using TensorFlow backend.
2017-11-06 16:53:14.674061: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-11-06 16:53:14.674512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 770 major: 3 minor: 0 memoryClockRate(GHz): 1.163
pciBusID: 0000:01:00.0
totalMemory: 1.94GiB freeMemory: 1.67GiB
2017-11-06 16:53:14.674554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 770, pci bus id: 0000:01:00.0, compute capability: 3.0)
Exception in thread Thread-2:                                                                              | 1/20 [00:51<16:09, 51.02s/it]
Traceback (most recent call last):██▎                                                                    | 34/162 [00:49<03:06,  1.46s/it]
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner                                   | 6/162 [00:08<03:36,  1.39s/it]
self.run()
File "/home/brianr/.local/lib/python3.5/site-packages/tqdm/_tqdm.py", line 144, in run
for instance in self.tqdm_cls._instances:
File "/usr/lib/python3.5/_weakrefset.py", line 60, in iter
for itemref in self.data:
RuntimeError: Set changed size during iteration
BTW it is running and says:
Evaluation model_2 vs model_1 (winrate:100%): 100%|██████████████████████████████████████████████████████| 10/10 [19:11<00:00, 115.13s/it]
We found a new best model : model_2!███████████████████████▉                                             | 78/162 [02:07<02:16,  1.63s/it]

Yes, I'm having the same problem I'm not too sure what causes this. It's possible that tqdm (progressbar) does not like when a game finished early (right now it's only when the two player pass).
It does not happen every game though and does not seem to impact the results (you can just remove the tqdm decoractor in the loop over every move in self_play to disable this error message).

As it is not an AlphaGoZero issue I don't mind ignoring it.  I just wanted to make sure it seemed to be running ok in spite of the message.  It ran for a few more cycles and then the 2GB on the 1070 gpu was exceeded.  It is odd that Keras will ask for too much gpu memory and TF does not seem to.  I can always run it on another PC I have with a 1070 with 8GB, but I'd have to install Linux on that box as well.  Right now I'm using VMware Workstation (free) which does not support gpus.

I'm having the same problem on my GTX 970. It's a far greater issue, but I'm having trouble debugging it naively, the stacktrace for TF is way too large.
For me it seems to be happening after 5 or so hours.
I start to have some NaN before the actual OOM error, so I don't think debugging the dying stacktrace will help.
I'm trying to figure out who is keeping memory, but I'm not sure how to go about debugging GPU memleaks.
You can check my tensorboard http://nicolas.kwyk.fr to see the learning.
Are you sure you have a 1070 ? You stack says you have a 770 ?

Probably linked to https://github.com/tensorflow/tensorflow/issues/10408

Edited earlier comment.  Two PCs: one with a 770 and another with a 1070.

Update to comment about TF not overflowing gpu memory.  I was only running LeelaZero in playing mode and it also overflowed the 770 2GB limit in training mode with TF, so it is not just a Keras issue.

The link I provided seems to blame, the clean_session. Some placeholder are always created when initiating a session (which keras does when calling fit).
I added in my local version a K.clear_session() after each evaluation, and so far, I did not have any OOM. It might still be overflowing because I did end up with NaNs after a while.

New version seems to fix it for me. Does it fix yours ?

Trying latest version.  Will take about 5-6 hours to run until it ran out of memory last time.  Will let you know.

It has been running about 6 hours and is up to model 17.  So far none have been better than model 1.  Other than that, things seem more stable (no OOM for RAM or GPU)

Ok, so that's one problem fixed.
Did you activate SHOW_END_GAME ? you can look at the end value for each model, to look for something suspicious (if the value is high while the model was beaten badly).
I changed the value from {1, 0} to {+1, -1} in the last version you have. For me, it never stays stuck with model1, but it does seem to reach some plateau after a few models(something like 6-10), and looking at end games it's still not playing quite properly, but nothing really suspicious (values are sometimes off but for both models).
I also have added more self play games in my current configuration. Because the algorithm only learns from these self play games, if they are not enough, it's unlikely to progress far enough. It also shows from deepmind parameters
self play games: 25k  in deepmind but here the conf has 20.
I moved to 50 and it's not as stuck anymore but I feel it would need more like 250 or more in order to avoid being stuck. It's a costly check, so for now I'm focusing on adding the missing features of the algo.
I've got a feeling we could add evaluations games to the training set of games in order to improve almost freely this "plateau" effect.
Also I'm currently at .4s/move but for 16 MCTS simulations instead of 1600. So there's two orders of magnitude to work on. batch exploration could get 8x, and I'm hoping quantization could get another improvement. With this I think it would be easier to increase the number of self play games.

Let's close this.

authors: Narsil brianprichardson

repo: Narsil/alphagozero


text:
provide more info when SBCF are different

When two SeekableByteChannelFactory objects are found to not be the same, the differences string is currently appended with "<data parameter name> not equal. 1.cs=<byte channel 1 size> 2.cs=<byte channel 2 size>".
This doesn't account for channels of the same size with different data. Finding these differences currently means stepping through the program with the debugger to manually convert each channel's input stream back into a string. Appending the data directly from each object makes this issue much easier to debug.
differences was also removed as the error message for assertNull in RegressionTestUtil because it already prints when the assert fails.

Line breaks and payload string removed from log messaging. Returned to original formatting with slightly adjusted wording for improved clarity.
Before:
[ERROR]   ExamplePlaceTest>RegressionTest.testExtractionPlace:266->ExtractionTest.testExtractionPlace:114->RegressionTest.checkAnswers:337 com.burrito.transform.ExamplePlace: PDiff: data not equal. 1.cs=64 2.cs=64 ==> expected: <null> but was: <com.burrito.transform.FfmpegPlace: PDiff: data not equal. 1.cs=64 2.cs=64>
After:
[ERROR]   ExamplePlaceTest>RegressionTest.testExtractionPlace:266->ExtractionTest.testExtractionPlace:114->RegressionTest.checkAnswers:337 expected: <null> but was: <com.burrito.transform.ExamplePlace: parent_differences: <data> elements are not equal. SeekableByteChannelFactory_1=64_byte_stream : SeekableByteChannelFactory_2=64_byte_stream >

Line breaks and payload string removed from log messaging. Returned to original formatting with slightly adjusted wording for improved clarity.
Before:
[ERROR]   ExamplePlaceTest>RegressionTest.testExtractionPlace:266->ExtractionTest.testExtractionPlace:114->RegressionTest.checkAnswers:337 com.example.transform.ExamplePlace: PDiff: data not equal. 1.cs=64 2.cs=64 ==> expected:  but was: <com.example.transform.FfmpegPlace: PDiff: data not equal. 1.cs=64 2.cs=64>
After:
[ERROR]   ExamplePlaceTest>RegressionTest.testExtractionPlace:266->ExtractionTest.testExtractionPlace:114->RegressionTest.checkAnswers:337 expected:  but was: <com.example.transform.ExamplePlace: parent_differences:  elements are not equal. SeekableByteChannelFactory_1=64_byte_stream : SeekableByteChannelFactory_2=64_byte_stream >

I have lost track of this change. @drivenflywheel or @jdcove2 do you think it is something we still want to do?

Putting this aside.

authors: cfkoehler rupert-griffin

repo: NationalSecurityAgency/emissary


text:
Decompilation broken for MIPS:BE:32:default

Describe the bug
Decompilation fails with error
 Low-level Error: Could not find op at target address:
To Reproduce

have MIPS:BE:32:default asm
opcodes clearly gets disassembled
functions are found, but
on decompilation it fails with above mentioned message

Screenshots

Environment:

OS: Win7
Java Version: 12.0.2
Ghidra Version: 9.1-BETA

Additional context
I have developed a loader for N64 Roms and want to upgrade it for ghidra 9.1 (the code in that repo was already upgraded for 9.1 but the dist was left for 9.0.4)

Can you post the decompile debug .xml output file for this function?  The issue may not be at the instruction, as I can't duplicate the issue with a similar set of instructions.

Sure, here 4 examples
debug.zip

For what it's worth, I'm seeing the same issue with the SH2 processor and 9.1-Beta, so probably not CPU related.
This is from a database upgraded from 9.0.4 and I'm also using a bunch of overlays. Let me know if you need extra info.

Possilbe overlay related.  It may be related to the new syscalls features, or the upgrade from 9.0.4.

@zeroKilo are any of your memory blocks overlays?

well im not sure what you mean by overlayed. I create an InputStream for most blocks from a single byte[] array, the rom, and the 2 .spXXX blocks that are zeroed byte arrays. these blocks dont overlap in memory, the layout is like following:

here the bytestring of fun_800007ec
here the disassembly of fun_800007ec
here the actual rom file
here the function making blocks
here is how the makeblock function is used

I reproduced the issue on my end with a clean new project in 9.1-dev and 2 segments (still with SH2)
Should I just attach the .gar file here? (1.7mb)

This is definitely an issue with overlays.  Your second parameter to MemoryBlockUtil.createInitialedBlock is actually creating an overlay block.
There appears to be a lookup issue with the decompiler and overlays, possibly associated with the new reference overrides for syscalls.

yes you seem to be right, changing the second parameter to false lets the decompilation finish

is there anywhere to read about what overlays mean? I still dont understand the difference, whats it used for normally?

@zeroKilo overlays are used when you have multiple code segments that can be loaded at the same address in memory. Their address would overlap in memory, so you need to create them a separate overlays.
The root issue remains, as overlays are basically unusable in 9.1 due to this.

Agreed.  The overlay issue is a regression related to system call override changes, it is being diagnosed and will be fixed before release.

@yaz0r thanks for the explanation

I would hardly call this issue resolved. @emteere do you need a new issue to track the overlay problem or want to reopen this?

Fixed by bf4a17b67dff93b6a2555ceb4eed9ba85d019e05

authors: emteere ryanmkurtz yaz0r zeroKilo

repo: NationalSecurityAgency/ghidra


text:
Unable to add another badge prerequisite to Learning Path; the error message is incorrect at well

PR: https://github.com/NationalSecurityAgency/skills-service/pull/2833

:+1:

authors: dwalizer sudo-may

repo: NationalSecurityAgency/skills-service


text:
App rejected by Apple because "using Safari" to log FB

Today I got my app rejected:
"We noticed that the user is taken to Safari to sign in or register for an account, which provides a poor user experience."
Is anything I can change from my side? It seems crazy ...
Thanks!

Although I'd like to know how to set the behavior on Angular Typescript,  a fast fix is in
login.manager.ios.js
On Line 9
change
if (fbLoginBehavior === void 0) { fbLoginBehavior = login_behavior_1.LoginBehavior.LoginBehaviorBrowser; }
to
if (fbLoginBehavior === void 0) { fbLoginBehavior = login_behavior_1.LoginBehavior.LoginBehaviorNative; }

Hi @arcovoltaico,
Just pass the desired LoginBehavior as a second parameter to the init method of the plugin (LoginBehaviorBrowser is the default value).
...
import { LoginBehavior } from "nativescript-facebook";
...
nsFacebook.init("fb_id", LoginBehavior.LoginBehaviorNative);
...

Thanks so much !

authors: DimitarTachev arcovoltaico

repo: NativeScript/nativescript-facebook


text:
Unable to debug when project's app directory is renamed

NativeScript CLI allows you to change the name of the app directory in the project by adding a nsconfig.json with the following content:
{
    "appPath": "src"
}

After that you can safely rename the app dir to src and CLI will build the project and livesync and debug will still work. However, debugging in VSCode is not working as the mapping between files on device and the local paths is not correct.
This breaks debugging of newly created NativeScript Angular projects (tns create myapp --ng), which already have nsconfig.json and have src dir instead of app.

Scenarios that should be tested on both iOS and Android:

[ ] Debug application that does not have nsconfig.json and the app directory is called app
[ ] Debug application that has nsconfig.json, but there's no appPath property in it.
[ ] Debug application that has nsconfig.json with appPath set to different dir, for example "src" and ensure you've renamed "app" dir to "src".
[ ] Debug application that does not have nsconfig.json. Stop the debugging. Add nsconfig.json, add "appPath" property in it, rename the "app" directory. Start the debugging again.


text:
NHM taxonomy

Parse taxonomy into a tree structure.
Add as dataset / tree UI for browsing data.

We could use GBIF data for this?

repo: NaturalHistoryMuseum/ckanext-nhm


text:
[vanilla-store] createVanillaSelect 구현

Related Issue
#112
Describe your changes

creatVanillaSelect를 구현합니다.
useSetStore, useStore, useStoreSelector에 함수오버라이드를 적용하여 select의 set은 사용못하도록 합니다.

Request

[x] 로컬 테스트 완료 (myasset-web)

@2-one-week 한주님 요기도 혹시 changeset action 수정해주실 수 있을까요~?

const useSyncStore = <State>(store: VanillaStore<State> | VanillaSelect<State>, initialValue?: State) => {
    const value = useSyncExternalStore(store.subscribe, store.get, () => initialValue || store.get())
    useSyncPersistStore(store, value)
    return value
}

요거 따로 빼서 그냥 재활용할까여?

아놔 ㅋㅋㅋ 잠시만요..

이제됐따

감사합니다 슈웃해보겠습니다!

authors: yceffort-naver yujeong-jeon

repo: NaverPayDev/pie


text:
🛑 Becpl Angular UI is down

In 13883fb, Becpl Angular UI ($BECPL_ANGULAR_URL) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Becpl Angular UI is back up in c9929ee after 17 minutes.

repo: NehalDamania/becpl-uptime


text:
🛑 Becpl Angular UI is down

In 381bf38, Becpl Angular UI ($BECPL_ANGULAR_URL) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Becpl Angular UI is back up in 308b00d after 23 minutes.


text:
🛑 Becpl Angular UI is down

In cf450e7, Becpl Angular UI ($BECPL_ANGULAR_URL) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Becpl Angular UI is back up in ed3ef49 after 5 minutes.


text:
when install on ubuntu 14.04 /usr/bin/ld: cannot find -lippicv

already install all the dependency by apt-get:
sudo apt-get install libhdf5-dev libyaml-dev libopencv-dev pkg-config
sudo apt-get install python python-dev python-pip python-virtualenv
next,
git clone https://github.com/NervanaSystems/neon.git
cd neon
make
then error occurs:
Finished processing dependencies for neon==1.3.0
######################
Setup complete.  Type:
. '.venv/bin/activate'
to work interactively
######################
Building loader.so...
/usr/bin/ld: cannot find -lippicv
collect2: error: ld returned 1 exit status
make[1]: *** [loader.so] Error 1
make: [neon/data/loader] Error 2 (ignored)

This may be an issue with OpenCV - see https://github.com/opencv/opencv/issues/5852
Can you retry with a more recent version of OpenCV and neon (1.5.3 is latest).

I solved this by downloading the ippicv lib
https://sourceforge.net/projects/opencvlibrary/files/3rdparty/ippicv/
and copying the appropriate files like so
/usr/local/opencv/libippicv.a
/usr/local/include/opencv/ippicv_types.h
/usr/local/include/opencv/ippicv_redefs.h
/usr/local/share/OpenCV/3rdparty/lib/libippicv.a
I also added the line
/usr/local/opencv/
to /etc/ld.so.conf.d/opencv.conf (had to create this).
After that, run sudo ldconfig. The creates a new location for that static library and tells the linker to look for that in the opencv dir.
Hope this helps!

authors: Cpruce fucusy jennifermyers

repo: NervanaSystems/neon


text:
Codegen mode

I want to use ngraph in codegen mode to generate C/C++ code from a pre-trained ONNX model that can link to my project in C. I used the tutorial below to create an ng_function and do a computation with some input, however I couldn't find any new files created with the optimized low level code. Do I need to change something in the way I ran the model or is this to do with how nGraph is compiled?
https://www.ngraph.ai/tutorials/onnx-tutorial

Our pre-built package is DEX only. If you want CODEGEN, you need to build nGraph from source using that tutorial. Please add  -DNGRAPH_USE_LEGACY_MKLDNN=FALSE to cmake option.
After nGraph is build, stay in build directory.

use serialization tool to convert onnx model to serialized model.
$ src/tools/serialize_onnx/serialize_onnx -i <.onnx> -o <.json>

use nbench to run the serialized model.
$ NGRAPH_CODEGEN=1 src/tools/nbench/nbench <*.json> -i 1 -b CPU

You will get Function_0_codegen.cpp in build/cpu_codegen.

Thanks, I now have the Function_0_codegen.cpp file following the steps you described.
In order to compile and link it against some main file I assume we need to create CPURuntimeContextCG with init_cg_ctx API? But how do we create cpu:CPURuntimeContext struct? We can not find the relevant API.

This is not a supported feature yet. You can take a look at setup_runtime_context in cpu_call_frame.cpp. For CODEDEN, we don't need to setup everything since we have CPURuntimeContextCG. You can try this:
auto ctx = new CPURuntimeContext;
    // num_inputs is the number of Parameters to the model, set the value of ctx->p_en[i] to false if that Parameter does not change from iteration to iteration, otherwise, set it to true.
    ctx->p_en = new bool[num_inputs];

    ctx->first_iteration = true;

    // Create temporary buffer pools
    size_t alignment = 4096; // you can change this number.
    // modify compile() at https://github.com/NervanaSystems/ngraph/blob/master/src/ngraph/runtime/cpu/cpu_external_function.cpp#L763  in cpu_external_function.cpp to write the definition of "buffer_size_vec" to the Function_0_codegen.cpp file before Function_0
    // the value of buffer_size_vec comes from m_memory_buffer_sizes.

    for (auto buffer_size : buffer_size_vec)
    {
        auto buffer = new AlignedBuffer(buffer_size, alignment);
        ctx->memory_buffers.push_back(buffer);
    }

Hi, I'm Elad and I'm working with Gal.
I figured it is not supported any longer ,but the generated code in Function_0_codegen.cpp is referring to this struct the following way
size_t pool_base_ptr = (size_t) ctx->memory_buffers[0]->get_ptr();
The parameter 'pool_base_ptr' is later used to set the data handler of the mkldn_memories in the CPURuntimeContextCG.
cg_ctx->set_memory_ptr(1, ((float*)(pool_base_ptr + 0)));
....
cg_ctx->set_memory_ptr(30, ((float*)(pool_base_ptr + 124112896)));
it is done for all primitives in net.
The mkldn_memories in the CPURuntimeContextCG are created in 'deserialize_memory_descs_and_build_memory' with NULL handle, so the actual data location is only set by set_memory_ptr API.
Since the setting of the data handler is done by pool_base_ptr + some offset, I assume that we need only one big buffer in memory_buffers[0].
My questiona are:

how do I know the required size of this buffer?
why not to re-use the buffer as input/output between layers?

As I said before, the buffer size comes from m_memory_buffer_sizes of CPU_ExternalFunction. You need to modify compile() in cpu_external_function.cpp to write that value out to the Function_0_codege.cpp:  https://github.com/NervanaSystems/ngraph/blob/master/src/ngraph/runtime/cpu/cpu_external_function.cpp#L763.
Something like this:
writer << "size_t buffer_size = " << to_string(m_memory_buffer[0] << ";\n";
Then you use buffer_size to allocate buffer:
size_t alignment = 4096; // you can change this number.
auto buffer = new AlignedBuffer(buffer_size, alignment);
ctx->memory_buffers.push_back(buffer);
We allocate one large buffer for all the intermediate values at the beginning. This is for performance reason. Input/output of layers would use that buffer with an offset. I am not sure what kind of re-use you're referring to since every layer is using the same buffer with different offsets.

Hi, i'll get the value the way you said. Meanwhile i just calculated it
manually and built exe.
There are several place holders in generated code for layers param (W+B)
that point to fixed addresses, using these as appear in code leads to
seg-fault.
I guess i need to replace them with valid addresses and load params, right?
Thanks
Elad
בתאריך יום ב׳, 18 בנוב׳ 2019, 20:30, מאת Amy Zhuang <
notifications@github.com>:

As I said before, the buffer size comes from m_memory_buffer_sizes of
CPU_ExternalFunction. You need to modify compile() in
cpu_external_function.cpp to write that value out to the
Function_0_codege.cpp:
https://github.com/NervanaSystems/ngraph/blob/master/src/ngraph/runtime/cpu/cpu_external_function.cpp#L763
.
Something like this:
writer << "size_t buffer_size = " << to_string(m_memory_buffer[0] << ";\n";
Then you use buffer_size to allocate buffer:
size_t alignment = 4096; // you can change this number.
auto buffer = new AlignedBuffer(buffer_size, alignment);
ctx->memory_buffers.push_back(buffer);
We allocate one large buffer for all the intermediate values at the
beginning. This is for performance reason. Input/output of layers would use
that buffer with an offset. I am not sure what kind of re-use you're
referring to since every layer is using the same buffer with different
offsets.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/NervanaSystems/ngraph/issues/3870?email_source=notifications&email_token=ADYLNCPZWFTK5IDWUM6V63DQULNMLA5CNFSM4JLXOG2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEELOBIA#issuecomment-555147424,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADYLNCNALTHOYTNQHYEHNM3QULNMLANCNFSM4JLXOG2A
.

Those under the comment "// Declare all constants"? Yes, you need to replace those.

Thank you @ayzhuang. wityh your guidance this works well. I am nowtrying to install this setup on SUSE (rather than Ubuntu) - is there any guidelines of how to go about installing the dependencies and main software?

This issue is very helpful, thanks! It does appear that since https://github.com/NervanaSystems/ngraph/pull/4679 the code generation step mentioned above is now:
NGRAPH_CODEGEN_DUMP_SOURCE=1 src/tools/nbench/nbench -f <*.json> -i 1 -b CPU:CODEGEN

@ayzhuang in your estimate, would it be hard to implement code generation for the GCPU backend? Seems like a great feature, enabling smooth integration of ngraph with, among others, IoT and Edge targeted code.

authors: GalMoore ayzhuang eladdidy robinvanemden

repo: NervanaSystems/ngraph


text:
Can't attach volume when the trident DaemonSet can't connect to trident-csi for the first time.

Trident 19.07.0 with CRD can't attach volume when the trident DaemonSet can't connect to trident-csi for the first time.
I think trident DaemonSet pod must crash(or retry forever) when it cant't register itself.
The details are as below.
(I modified the node name my-worker1 for security reasons)

I found below pods in my cluster which mount Trident's Volume.
$ kubectl get pods
NAME            READY   STATUS              RESTARTS   AGE
pod-pvc-block   0/1     ContainerCreating   0          25m
pod-pvc-file    0/1     ContainerCreating   0          58m

I checked trident's log.
trident-csi
$ kubectl logs -n trident trident-csi-cbfc4489f-t6mlf trident-main -f
time="2019-09-03T07:25:18Z" level=error msg="Node info not found." node=my-worker1
time="2019-09-03T07:25:18Z" level=error msg="GRPC error: rpc error: code = NotFound desc = node my-worker1 was not found"

This log said node Not Found but the node existed.
$ kubectl get nodes my-worker1
NAME                                           STATUS   ROLES    AGE    VERSION
my-worker1   Ready    <none>   168m   v1.15.1

I found there are no tridentnodes.
$ kubectl get tridentnodes.trident.netapp.io --all-namespaces
No resources found.

I checked DaemonSet Pod.
$ kubectl logs -n trident trident-csi-pz5w7 trident-main
time="2019-09-03T04:43:34Z" level=info msg="Running Trident storage orchestrator." binary=/usr/local/bin/trident_orchestrator build_time="Wed Jul 31 14:23:00 UTC 2019" version=19.07.0
time="2019-09-03T04:43:34Z" level=info msg="Initializing plain CSI helper frontend."
time="2019-09-03T04:43:34Z" level=info msg="Added frontend." name=plain_csi_helper
time="2019-09-03T04:43:34Z" level=info msg="Initializing CSI frontend." name=my-worker1 version=19.07.0
time="2019-09-03T04:43:34Z" level=info msg="Enabling node service capability." capability=STAGE_UNSTAGE_VOLUME
time="2019-09-03T04:43:34Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_WRITER
time="2019-09-03T04:43:34Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_READER_ONLY
time="2019-09-03T04:43:34Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_READER_ONLY
time="2019-09-03T04:43:34Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_SINGLE_WRITER
time="2019-09-03T04:43:34Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_MULTI_WRITER
time="2019-09-03T04:43:34Z" level=info msg="Added frontend." name=csi
time="2019-09-03T04:43:34Z" level=warning msg="Couldn't retrieve volume transaction logs: Unable to find key"
time="2019-09-03T04:43:34Z" level=info msg="Trident bootstrapped successfully."
time="2019-09-03T04:43:34Z" level=info msg="Activating plain CSI helper frontend."
time="2019-09-03T04:43:34Z" level=info msg="Activating CSI frontend."
time="2019-09-03T04:43:34Z" level=info msg="Listening for GRPC connections." name=/plugin/csi.sock net=unix
time="2019-09-03T04:45:38Z" level=error msg="Could not communicate with controller after 90.00 seconds; could not log into the Trident CSI Controller: error communicating with Trident CSI Controller; Get https://trident-csi:34571/trident/v1/node: dial tcp 10.27.3.141:34571: connect: connection refused" node=my-worker1
time="2019-09-03T04:45:39Z" level=error msg="Error registering node my-worker1 with controller; could not log into the Trident CSI Controller: error communicating with Trident CSI Controller; Put https://trident-csi:34571/trident/v1/node/my-worker1: dial tcp 10.27.3.141:34571: connect: connection refused"

This was the cause of this trouble!
I delete the pod.
$ kubectl delete pods -n trident trident-csi-pz5w7
pod "trident-csi-pz5w7" deleted

and checked re-created Pod works well.
$ kubectl logs -n trident trident-csi-s4bwv trident-main
time="2019-09-03T07:49:25Z" level=info msg="Running Trident storage orchestrator." binary=/usr/local/bin/trident_orchestrator build_time="Wed Jul 31 14:23:00 UTC 2019" version=19.07.0
time="2019-09-03T07:49:25Z" level=info msg="Initializing plain CSI helper frontend."
time="2019-09-03T07:49:25Z" level=info msg="Added frontend." name=plain_csi_helper
time="2019-09-03T07:49:25Z" level=info msg="Initializing CSI frontend." name=my-worker1 version=19.07.0
time="2019-09-03T07:49:25Z" level=info msg="Enabling node service capability." capability=STAGE_UNSTAGE_VOLUME
time="2019-09-03T07:49:25Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_WRITER
time="2019-09-03T07:49:25Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_READER_ONLY
time="2019-09-03T07:49:25Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_READER_ONLY
time="2019-09-03T07:49:25Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_SINGLE_WRITER
time="2019-09-03T07:49:25Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_MULTI_WRITER
time="2019-09-03T07:49:25Z" level=info msg="Added frontend." name=csi
time="2019-09-03T07:49:25Z" level=warning msg="Couldn't retrieve volume transaction logs: Unable to find key"
time="2019-09-03T07:49:25Z" level=info msg="Trident bootstrapped successfully."
time="2019-09-03T07:49:25Z" level=info msg="Activating plain CSI helper frontend."
time="2019-09-03T07:49:25Z" level=info msg="Activating CSI frontend."
time="2019-09-03T07:49:25Z" level=info msg="Listening for GRPC connections." name=/plugin/csi.sock net=unix

checked tridentnodes.
$ kubectl get tridentnodes.trident.netapp.io --all-namespaces
NAMESPACE   NAME         AGE
trident     my-worker1   67s

my pods working !
$ kubectl get pods  -w
NAME            READY   STATUS    RESTARTS   AGE
pod-pvc-block   1/1     Running   0          32m
pod-pvc-file    1/1     Running   0          65m

I think trident DaemonSet pod must crash(or retry forever)  when it cant't register itself.

I saw this with K8S 1.15.3 + T 19.07.0 last week

Does the fix also resolve scenarios where connection is lost after some time and not just on startup? I've now experience both issues.

@clintonk understood. Something eles must be up then. I initially ran into this issue when the trident pod was starting up. I just had another issue where a (non trident) pod never started because the trident pod on the node lost it's connection to the controller after running for several days. The pod was sitting in an init state indefinitely because the mount never happened for the pv. This was for nfs but I get your message about iscsi/iqn as well. So it must matter in some shape or form after the fact. I simply deleted the trident pod on the node and then everything moved on as expected.
I essentially want to make sure that it continually monitors and tries to connect over time.

Also note, if I understand correctly the iscsi/iqn stuff happens at attach time so a connection the the controller would still need to function beyond startup of the pod. I may have misunderstood how/when that info got added though. Also that assumes that the node pods don't make those api calls but rather the controller...which may not be the case, not sure.

@travisghansen Prior to this fix, if the CSI Trident node was restarted for whatever reason, and even if it had previously registered with the controller, if it could not contact the controller after the most recent restart, then it would give up after 90 seconds and no longer respond to attachment requests.  This fix continues trying the registration indefinitely until it succeeds, and it does not block other node operations while doing so.
FWIW, while attaching a volume to a pod, a CSI provisioner is first called at the controller followed by the node.  The controller must know the node details, because only the controller interacts with the storage management interfaces, and knowing (for example) the node IQN allows us to update igroups as needed at attach time.

@clintonk until it succeeds is what I'm concerned about. What happens after it succeeds and then fails at some point? Will it start a reconnect sequence in that scenario as well (or exit the pod so a new gets recreated I suppose)?

After discussion on slack with @clintonk  it became clear I misread some logs. Connection after startup is irrelevant. Thanks for the help! Looking forward to the fix.

authors: clintonk inajob scaleoutsean travisghansen

repo: NetApp/trident


text:
0.4.x

Why are the changes needed?

Brief change log
merge from master
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before making a pull request

Documentation

Does this pull request introduces a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

I will close this PR since branch 0.4.x had merged master.

repo: NetEase/arctic


text:
网盘链接不存在

您好数据集的网盘链接不存在，可以麻烦再发一份吗

我得找时间再找找这个数据。应该是因为毕业了，学校直接把我的网盘账号给关了。

这个数据集获取不了 All datasets are available at Tsinghua Cloud: https://cloud.tsinghua.edu.cn/d/aa4102a5d1614e57bc36/

https://zenodo.org/record/8153849

authors: JasonPeng1310 lizeyan zhanpengjie

repo: NetManAIOps/PSqueeze


text:
Default .env file does not include required config

The default .env file in the repository does not include all the configuration parameters required for a successful installation such as DISPATCH_UI_URL. Therefore .env file should be removed and rely on providing sane defaults in .env.example and reply on 'ensure_file_from_example' to take care of creating proper .env file

Makes sense. Thanks for smoothing this out.

authors: kevgliss shumbashi

repo: Netflix/dispatch-docker


text:
New feature: get connected applications

It would be a great feature to know who is connected to who.
When a service retrieve information about another one (discovery, GET /apps/appid), it could be interesting to register this in order to know that the service is connected to the other.
Thus, we could build the complete graph of service dependencies. Eureka have all the information, It would be a great pity not to take advantage of it.
I know this is not based on real connections (logs) but it would provide pretty good information for nothing.
What do you think ?

It would be nice to have :).

authors: jeromevdl qiangdavidliu

repo: Netflix/eureka


text:
Improve style sheet adoption ergonomics.

One common pattern for element authors (now that import attributes enable folks to import css files directly) is to adopt imported style sheets into a shadow root at custom element initialization time.
This adds two static getters — shadowRootInit and styleSheets. The goal is to make shadow root initialization as declarative as possible. Note that we still expose createRenderRoot so we don’t get in the way if folks need to do something more advanced (it’s also still the only way to forgo the creation of a shadow root at all).
Closes #52.

@klebba — This is more-or-less what I figured we’d do here. It’s pretty straightforward, just a way to declare things in isolation as an ergonomic benefit.

Note… if we did want to get rid of createRenderRoot. We could consider a flag like this:
static get useShadowRoot() {
  return true;
}

We would have to do something more to allow the use-case where folks will want to create a custom element without a shadow root by using the host itself as the render root.

Looks awesome, nice work!

authors: klebba theengineear

repo: Netflix/x-element


text:
Improve logging

Try https://learn.microsoft.com/en-us/dotnet/core/extensions/high-performance-logging approach
Clean up logging
Add documentation about log usage and when to use specific log levels

I would like to propose the implementation of an EventId system for our logs, which will consist of six digits.
The first three digits will be related to a specific project:

1XX: Core projects such as Libp2p, Libp2p.Core
2XX: Protocols, for example:

200: Ping
201: Plaintext
and so forth

The last three digits will determine the log level and the log's ID:

000..099: Trace
100..199: Debug
200..299: Information
300..399: Warning
400..499: Error
500..599: Critical

For example, the complete EventId for the Ping protocol could be 200100.
This system will provide a clear and structured way to identify events in our logs, making it easier to track and troubleshoot issues.
Please feel free to share your thoughts and feedback on this proposal.
I will prepare a PR for the Ping protocol to make it more clear.
@flcl42

@chertby it's good to have some structure but it seems like complication for potential contributors, another rule to keep in mind. Let's start by having just incremental numbers for the messages which is already a lot.

I still suggest leaving different event IDs for different protocols. But then use a simple increment. Use the name of the function itself as an event hire.
It’s simpler than the previous option, but we keep good logging, which should help with debugging.
I updated PR
private const int EventId = 200_000;

[LoggerMessage(
    EventId = EventId + 1,
    EventName = nameof(ReadingPong),
    Message = "Reading pong",
    Level = LogLevel.Trace)]
internal static partial void ReadingPong(
    this ILogger logger);

[LoggerMessage(
    EventId = EventId + 2,
    EventName = nameof(VerifyingPong),
    Message = "Verifying pong",
    Level = LogLevel.Trace)]
internal static partial void VerifyingPong(
    this ILogger logger);


text:
Enhance Error Handling for Unexisting Tx Hash

This a new version of PR of #1062
@joshklop

@aminsato Would love to get this merged, but we aim for a linear commit history without merge commits. That means git rebase main should succeed without needing to manually resolve conflicts. Do you mind doing that?

@aminsato Would love to get this merged, but we aim for a linear commit history without merge commits. That means git rebase main should succeed without needing to manually resolve conflicts. Do you mind doing that?

@joshklop
Thanks for your tips
After hard work 😓, I made my commit history linear and resolved conflicts 😞. Would you mind taking a look?

authors: aminsato joshklop

repo: NethermindEth/juno


text:
"Warning: No sync-checkpoint received for quite a long time"

according saw in a comment does not affect but in reality if it is assumed that the stake lasts 1.6 days? I have four good days with the message on my client QT and no stake, look ua quick fix to this but the network will be vunerable and nobody will want to stake

I'm sorry but I'm not sure I have correctly understood your issue. If it's about the warning, as your title suggests, then it's right, it's not an issue. It only means that we didn't published sync-checkpoints yet.
Sync-checkpoints are a PoS feature that allow the coin operators to manually enforce a blockchain if need be. Since this feature is by nature quite centralizing, we promised to not use it until its usefulness exceed its drawback.

repo: NeuCoin/neucoin


text:
Alternative way to get offset before first "epoch" in axon protocol conversion

At the moment Neo uses a slightly "magic" equation to work out the (undocumented) time spent at holding potential before the first "epoch" of a protocol starts:
https://github.com/NeuralEnsemble/python-neo/blob/6ce00dc9c13e85c2b2cbf6dacb6d66f93b0099c6/neo/rawio/axonrawio.py#L338-L340
Digging into the File Support Pack 1 for Windows for the old ABF 1.x format, there's a method inside AxonDev/Comp/AxAbfFio32/ABFHWAVE.CPP (which I guess deals with "waveforms", a.k.a. D/A protocol outputs) called _GetHoldingLength():
//===============================================================================================
// FUNCTION: GetHoldingLength
// PURPOSE:  Get the duration of the first/last holding period.
//
static int _GetHoldingLength(int nSweepLength, int nNumChannels)
{
   ASSERT((nSweepLength % nNumChannels)==0);

   // Calculate holding count.
   int nHoldingCount = nSweepLength / ABFH_HOLDINGFRACTION;

   // Round down to nearest sequence length.
   nHoldingCount -= nHoldingCount % nNumChannels;

   // If less than one sequence, round up to one sequence.
   if (nHoldingCount < nNumChannels)
      nHoldingCount = nNumChannels;

   return nHoldingCount;
}

where ABFH_HOLDINGFRACTION is a macro defined as 64 and where I'm guessing nNumChannels is the number of AD (not DA) channels.
I've tried for a few files and it seems to give the same result as the current code. Don't want to fix what isn't broken, but perhaps good to try out?
The next function down, ABFH_GetHoldingDuration, suggests a difference for "old" versions of the format (but given that these were "old" in 2000, maybe that's ok)
Note that the file starts with

//    Copyright (c) 1993-2000 Axon Instruments.
//    All rights reserved.
//    Permission is granted to freely to use, modify and copy the code in this file.

Hi @MichaelClerx. Thanks for digging into the Axon format and looking for potential bugs in Neo.
If the current code is already loading the file correctly I don't think it's necessary to change it. If you find it important, feel free to open a PR and include a corresponding test in the our code base here.

authors: JuliaSprenger MichaelClerx

repo: NeuralEnsemble/python-neo


text:
nullreference exception on instantiating neutronium

Hi @David-Desmaisons
Neutronium\Neutronium.Core\Binding\Listeners\ListenerUpdater.cs
private void OnCommandCanExecuteChanged(object sender, EventArgs e)
{
var updater = _JsUpdaterFactory.GetUpdaterForExecutionChanged(sender);
ScheduleChanges(updater);
}
throw exception because for some magic reason sender is null. i cant figure out whats wrong with my code. however adding a check on sender is null will make the application work as expected. Nothing stop working if the check is there! Would it be possible if you could add that conditino there?

In fact, this is command implementation that send an event with null sender.
So you should check your code base to correct this problem.
That said Neutronium should be resilient to interface implementation error, so I will implement the null check.

thanks im not exactly sure, why thise has surface now. Ive been using those RelayCommands for years wihout issues. I did update Neutronium to last version, but cant be sure if that triggered it.  I have another one! but this one is a bit more tricky not sure if this is because of the scenarion im implementing
Basically replace UserControl with another on without closing main window.
If i set this.DataContext = new model() then i get exception
    private void ChromiumFxControl_Loaded(object sender, RoutedEventArgs e)
    {
        this.Loaded -= ChromiumFxControl_Loaded;
        _Window = Window.GetWindow(this); > returns null
        _WindowHandle = new WindowInteropHelper(_Window).Handle;
        _Window.Closed += Window_Closed;
        _Window.LocationChanged += _Window_LocationChanged;
        _Window.StateChanged += Window_StateChanged;
        UpdateDpiMatrix();
    }

Im not really sure what that means however, if i dont set the datacontext than everything works fine. If i also check for null and return
_Window = Window.GetWindow(this);
if (_Window == null)
{
return;
}
Everything loads and works properly. Can you also implement this check?

=> This is a differenet issue/question than the previous one so open an new issue, I will comment on the new one.

yep, sorry about that

authors: David-Desmaisons jmecosta

repo: NeutroniumCore/Neutronium


text:
Compare value type performance vs ref type performance

rsl::call_traits::param_type
a bit verbose, I know
Originally posted by @RisingLiberty in https://github.com/New-Creta/Rex/pull/79#discussion_r1186902175

results are negligible in my quick tests, so I guess it's fine

authors: Dyronix RisingLiberty

repo: New-Creta/Rex


text:
Match fiss_site forms to hab con template

the fields for feature times are missing from the form_fiss_site_tidy
the fields "morphology", "l_bank_texture_dominant_2" and "r_bank_texture_dominant_2" are either blank or have a value of "FALSE"

latest version of the tidy_fiss script is in https://github.com/NewGraphEnvironment/fish_passage_moti_2022_reporting/tree/main/scripts (currently private repo but not for long).
Noticed that the gazetted_name column is named gazetted_names in the site information page.  This causes a lack of names to come through in form_loc


text:
Normalize Address State field across forms

Fixes #440
Creator

[x] Build relevant tests if any apply
[x] Ensure there are no new warnings
[x] Upload an animated GIF showcasing the solution (if it applies)
[x] Set a relevant reviewer

Reviewer

[ ] Run test and make sure all tests pass
[ ] Review function and form on iOS
[ ] Review function and form on Android
[ ] Review function and form on Web
[ ] Review new test logic

I added an input picker for state in the Profile AddressForm. With this change, I created a new data component, withAddressState, within withUser that can be used in BillingAddressForm so that changes made to Profile Address can be used elsewhere and not have to have withCheckout in the ProfileAddressForm.

While I would love to approve this, since I worked on it we should probably get someone else to review it. @burkeshartsis, @IsaacHardy, or @richarddubay have at it.

@cdanwards Let's try to solve the last issue in issue #440 in this PR. I will create an enhancement issue for pre-filling the new payment form with existing user address.

So that it's written down...what was decided was that we would pull out the changes that we merged for splitting the states and territories because it hardcoded values. What @IsaacHardy would like to do is to hardcode the values and do the splitting in Heighliner and send back the data separately. Then we can make another change to get that data and correct the dropdown once that is changed.

authors: IsaacHardy cdanwards conrad-vanl richarddubay

repo: NewSpring/Apollos


text:
Mcs 1298 stitch top down plotter

This depends on a fix to MCS-1301 that removes the image folder.  That folder is also probably going to be renamed and this will have to be updated, but that should be easy.

Should we add to this PR to provide a config way to switch between the two top down image generators or even provide a way to produce both?  This might ease the transition and mitigate risks.

@kdrumm We can definitely do that. Do you have a suggestion for how to approach this?

@kdrumm We can definitely do that. Do you have a suggestion for how to approach this?

I added some more config options that we can hopefully usually ignore.

@kdrumm We can definitely do that. Do you have a suggestion for how to approach this?

I added some more config options that we can hopefully usually ignore.

Awesome - I meant to ask, will this require any changes to either the Unity or Python PR for capturing the top-down images?

will this require any changes to either the Unity or Python PR for capturing the top-down images?

Nope!

I had to make some changes here.  I think at least one person should reapprove.

authors: alexcraneunity kdrumm

repo: NextCenturyCorporation/MCS


text:
linux arm builds

Feature details
We could modify our github action to produce arm64 builds for linux to support raspberry pis, Rockchip boards like nanopc-t6 and Radxa. The challenge is to make it work seamlessly with the build tools that currently work out-of-the-box within a single github action workflow. Additionally, these boards typically have a flavor of linux that uses the 5.10 kernel. There is a lot of work to get Rockchip boards onto the mainline linux kernel (6.6), but I'm not sure of the eta on that. I do not think NiceNode works well on kernal 5.10 (uname -r to check)
To build a dev file in ubuntu arm:
sudo apt install ruby ruby-dev binutils
sudo gem install fpm
export USE_SYSTEM_FPM=true

# build
npm run package -- --linux deb --arm64

# install (check version)
sudo dpkg -i ./release/build/nice-node_4.7.2-alpha_arm64.deb

# users may have to install this
sudo apt install libxss1

(mac) asahi fedora39 arm:
sudo dnf install ruby glibc qemu fedora-packager rpmdevtools
sudo gem install fpm
export USE_SYSTEM_FPM=true

# build
npm run package -- --linux deb --arm64

sudo dnf install ./release/build/nice-node-4.7.2-alpha.aarch64.rpm
# error
Error: Transaction test error:
  file /usr/lib/.build-id/b6/4138acadfa57e4e7682b5591ebe4081878438c from install of nice-node-4.7.2~alpha-1.aarch64 conflicts with file from package python3-3.12.1-1.fc39.aarch64

# works but probably not the best solution
sudo rpm -i --replacefiles ./release/build/nice-node-4.7.2-alpha.aarch64.rpm

# run in foreground
nice-node

I have been working on the underlying repos to fix this:
https://github.com/electron-userland/electron-builder-binaries/issues/49#issuecomment-2008211462
https://github.com/electron-userland/electron-builder/issues/5154

v6+ will have linux, mac, and windows arm builds. bring on the arm chips :)

repo: NiceNode/nice-node