text:
On the feasibility of replacing N4BiasFieldCorrection with SPM Segment

Hi guys,
I've noticed anecdotally (though I've heard this from others as well) that for better FreeSurfer surface reconstruction, the choice of bias field correction makes a huge difference. In this regard, the SPM bias field correction tends to perform much better than the N4 algorithm.
In order to use the SPM bias field correction and still benefit from smriprep, I'm thus left with 2 options:

run SPM bias field correction BEFORE smriprep

This should work, but requires that I replace the original T1 with a bias corrected in my experiment-datasets BIDS directory, which is OK but goes against the principles of reproducibility I suppose.
Once N4 comes along, there should be no bias to estimate, and so N4 should hopefully act as no-op. Somewhat hacky and not something I'd like to rely on.

replace N4 with SPM in smriprep

Replacing N4 with SPM in workflows/anatomical.py seems trivial, and I'd be willing to provide a PR if there is interest. However, given that SPM requires Matlab, I don't know how this pans out for docker/singularity containers.
And even if N4 were not to be replaced with SPM in master, I would like to be able to use my own fork. Is it currently possible to pass a patched smriprep directory for an fmriprep singularity image? And how does one go about having the container access the external Matlab?
Any tips of suggestions are most welcome.
As always, thanks for this awesome project.
Possibly related #7

@dangom any updates about that? I've randomly ended up on that issue and was wondering about the "flexibility" of the sMRIPrep pipeline in general (I'm a new sMRIPrep user).

For end users, flexible as a diamond by design. The goal is to be robust, not malleable.
I moved away from sMRIPrep because I wanted to switch to FS 7, and at that time sMRIPrep still relied on FS 6. I haven't checked whether that's changed already - if so maybe it's time to give it a try again.
With that said, there is no problem at all in just running the SPM bias field correction before sMRIPrep. That works without issues.
Replacing N4 for SPM is not the way to go since it requires Matlab. Ideally someone would bite the bullet and reimplement the SPM algo outside of matlab, and then replacing N4 could be a project worth pursuing (if NNs don't replace the SPM bias field correction by then, that is).

authors: dangom mafortin

repo: nipreps/smriprep


text:
Estimated ETH value instead of BTC?

Would it be possible to make the extension ETH-centered? Many users want to see the value of their entire wallet and specific coins in ETH instead of BTC.

Or at least make it show value in ETH when the pair you are trading is ETH. As of now, it uses the amount of ETH and shows the $ value in BTC, making a 1 ETH sell look like a $6700 order.

source: gharchive/issue

repo: nirvanatikku/bittrex-enhanced

license: mit

license_type: permissive

license_source: bigquery


text:
Branch tracking detection fails for different system language

svn2git parses the git error messages to detect whether branch tracking is enabled, see https://github.com/nirvdrum/svn2git/blob/master/lib/svn2git/migration.rb#L360. This fails if the messages are localized. A workaround is to use LANG=C before calling the script. This was already mentioned in #132

I've logged this issue in my new implementation. Hopefully I can fix it other than depending on a workaround.

OK I've resolved this issue within a new release in svn2git.net. In case anyone wants it, just download the latest release from https://github.com/mazong1123/svn2gitnet/releases


text:
error: While opening repository /home/*/.local/share/flatpak/repo: opening repo: opendir(objects): No such file or directory

$ flatpak --version;lsb_release -d;uname -r;kde
Flatpak 1.10.7
Description:    Debian GNU/Linux 11 (bullseye)
5.10.0-11-amd64

KDE desktop
"sudo flatpak install qbittorrent"
"sudo flatpak list"
"flatpak run org.qbittorrent.qBittorrent"
    error: While opening repository /home/user/.local/share/flatpak/repo: opening repo: opendir(objects): No such file or directory
some Debian/flatpak bug, that can be fixed by command "mkdir /home/$(whoami)/.local/share/flatpak/repo/objects"
can this be made the way so this objects is created and user is not bothered by that error?

Well, this doesn't really sound like something that should be fixed here, so I'll go ahead an close the issue.
If you persist /home/user/.local/share/flatpak/repo it should be created if it doesn't already exist. I'm guessing you want to persist the whole flatpak directory, though? In that case you could add something like
system.activationScripts.createFlatpakRepo = {
  deps = [ "createPersistentStorageDirs" ];
  text = ''
    mkdir -p /persistent/home/<user>/.local/share/flatpak/repo
    chown <user>:<group> /persistent/home/<user>/.local/share/flatpak/repo
  ''
};

for NixOS or
home.activation.createFlatpakRepo =
  config.lib.dag.entryAfter
  [ "createAndMountPersistentStoragePaths" ]
  ''
    mkdir -p ~/.local/share/flatpak/repo
  ''

for home-manager.

doesn't really sound like something that should be fixed here

and You do not have idea what is the right bug tracker?

If it's a Flatpak bug, I'm guessing https://github.com/flatpak/flatpak/issues/, but I'm not sure.

authors: slrslr talyz

repo: nix-community/impermanence

license: MIT

license_source: github-api


text:
Using --build-on-remote targets i686-linux

Describe the bug
When trying to build my x86_64-linux machine from my M1 Macbook, I get the following error:
Failed to find a machine for remote build!
derivation: bm173dm4why0gsh6h485v6q9dgz1r9gb-nv-codec-headers-11.1.5.2.drv
required (system, features): (i686-linux, [])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([x86_64-linux], 1, [], [])

Previously I was able to build this system successfully if running locally. The x86_64-linux machine in question is a local bare metal PC in my office, running the latest NixOS installer (23.05).
I'm unsure why i686-linux is being targeted here.
To Reproduce
Steps to reproduce the behavior:
I ran this on my local LAN:
nix run github:dustinlyons/nixos-anywhere -- root@192.168.0.134 --flake .#felix --build-on-remote -L
Expected behavior
I expect the system closure to be built. I'm unsure why ffmpeg headers are being targeted for i686-linux.
System information
Here's my code: https://github.com/dustinlyons/nixos-config
Additional context
Logs:
nix run github:numtide/nixos-anywhere -- root@192.168.0.134 --flake .#felix --build-on-remote -L
### Uploading install SSH keys ###
/Users/dustin/.nix-profile/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/tmp/tmp.6LIk7zZn2o/nixos-anywhere.pub"
/Users/dustin/.nix-profile/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/Users/dustin/.nix-profile/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
Warning: Permanently added '192.168.0.134' (ED25519) to the list of known hosts.
(root@192.168.0.134) Password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'ConnectTimeout=10' -o 'UserKnownHostsFile=/dev/null' -o 'StrictHostKeyChecking=no' 'root@192.168.0.134'"
and check to make sure that only the key(s) you wanted were added.

### Gathering machine facts ###
Warning: Permanently added '192.168.0.134' (ED25519) to the list of known hosts.
### Building disko script ###
warning: Git tree '/Users/dustin/.local/share/src/nixos-config' is dirty
warning: not writing modified lock file of flake 'git+file:///Users/dustin/.local/share/src/nixos-config':
• Removed input 'nixos'
### Formatting hard drive with disko ###
Warning: Permanently added '192.168.0.134' (ED25519) to the list of known hosts.
Warning: Permanently added '192.168.0.134' (ED25519) to the list of known hosts.
umount: /mnt/boot unmounted
umount: /mnt unmounted
++ realpath /dev/nvme0n1
+ disk=/dev/nvme0n1
+ lsblk --output-all --json
++ dirname /nix/store/bxq686x89kkmkaamh9qnk8178gqvr4vn-disk-deactivate/disk-deactivate
+ jq -r --arg disk_to_clear /dev/nvme0n1 -f /nix/store/bxq686x89kkmkaamh9qnk8178gqvr4vn-disk-deactivate/disk-deactivate.jq
+ set -fu
+ wipefs --all -f /dev/nvme0n1p1
/dev/nvme0n1p1: 8 bytes were erased at offset 0x00000036 (vfat): 46 41 54 31 36 20 20 20
/dev/nvme0n1p1: 1 byte was erased at offset 0x00000000 (vfat): eb
/dev/nvme0n1p1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa
+ wipefs --all -f /dev/nvme0n1p2
/dev/nvme0n1p2: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
++ zdb -l /dev/nvme0n1
++ sed -nr 's/ +name: '\''(.*)'\''/\1/p'
+ zpool=
+ [[ -n '' ]]
+ unset zpool
+ wipefs --all -f /dev/nvme0n1
/dev/nvme0n1: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/nvme0n1: 8 bytes were erased at offset 0x1d1c1115e00 (gpt): 45 46 49 20 50 41 52 54
/dev/nvme0n1: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
++ mktemp -d
+ disko_devices_dir=/tmp/tmp.9CsDqk3PLa
+ trap 'rm -rf "$disko_devices_dir"' EXIT
+ mkdir -p /tmp/tmp.9CsDqk3PLa
+ device=/dev/nvme0n1
+ imageSize=2G
+ name=nvme0n1
+ type=disk
+ device=/dev/nvme0n1
+ type=gpt
+ sgdisk --new=1:0:+100M --change-name=1:disk-nvme0n1-ESP --typecode=1:EF00 /dev/nvme0n1
Creating new GPT entries in memory.
The operation has completed successfully.
+ udevadm trigger --subsystem-match=block
+ udevadm settle
+ device=/dev/disk/by-partlabel/disk-nvme0n1-ESP
+ extraArgs=()
+ declare -a extraArgs
+ format=vfat
+ mountOptions=('defaults')
+ declare -a mountOptions
+ mountpoint=/boot
+ type=filesystem
+ mkfs.vfat /dev/disk/by-partlabel/disk-nvme0n1-ESP
mkfs.fat 4.2 (2021-01-31)
+ sgdisk --new=2:0:-0 --change-name=2:disk-nvme0n1-root --typecode=2:8300 /dev/nvme0n1
The operation has completed successfully.
Discarding device blocks: + udevadm trigger --subsystem-match=block
+ udevadm settle
+ device=/dev/disk/by-partlabel/disk-nvme0n1-root
+ extraArgs=()
+ declare -a extraArgs
+ format=ext4
+ mountOptions=('defaults')
+ declare -a mountOptions
+ mountpoint=/
+ type=filesystem
+ mkfs.ext4 /dev/disk/by-partlabel/disk-nvme0n1-root
mke2fs 1.47.0 (5-Feb-2023)
done
Creating filesystem with 488352785 4k blocks and 122093568 inodes
Filesystem UUID: e59daada-f9ac-4699-871e-14606c0afc86
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
	102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

+ set -efux
+ findmnt /dev/disk/by-partlabel/disk-nvme0n1-root /mnt/
+ mount /dev/disk/by-partlabel/disk-nvme0n1-root /mnt/ -t ext4 -o defaults -o X-mount.mkdir
+ findmnt /dev/disk/by-partlabel/disk-nvme0n1-ESP /mnt/boot
+ mount /dev/disk/by-partlabel/disk-nvme0n1-ESP /mnt/boot -t vfat -o defaults -o X-mount.mkdir
+ rm -rf /tmp/tmp.9CsDqk3PLa
### Building the system closure ###
warning: Git tree '/Users/dustin/.local/share/src/nixos-config' is dirty
warning: not writing modified lock file of flake 'git+file:///Users/dustin/.local/share/src/nixos-config':
• Removed input 'nixos'
trace: warning: optionsDocBook is deprecated since 23.11 and will be removed in 24.05
trace: warning: optionsDocBook is deprecated since 23.11 and will be removed in 24.05
trace: warning: optionsDocBook is deprecated since 23.11 and will be removed in 24.05
trace: warning: The option `fonts.fonts' defined in `/nix/store/xb426qj1fkj48rw3fwx6mqh6vj0488mk-source/nixos' has been renamed to `fonts.packages'.
Failed to find a machine for remote build!
derivation: bm173dm4why0gsh6h485v6q9dgz1r9gb-nv-codec-headers-11.1.5.2.drv
required (system, features): (i686-linux, [])
1 available machines:
(systems, maxjobs, supportedFeatures, mandatoryFeatures)
([x86_64-linux], 1, [], [])
error: a 'i686-linux' with features {} is required to build '/nix/store/bm173dm4why0gsh6h485v6q9dgz1r9gb-nv-codec-headers-11.1.5.2.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

I get similar issues when calling --build-on-remote when building for x86_64-linux (gdrn here https://github.com/multivac61/nixos-config)
Note my user is a trusted
error:
       … while calling the 'head' builtin

         at /nix/store/q43321jn4spvsf1vzmiv56cgm4wgiss3-source/lib/attrsets.nix:1575:11:

         1574|         || pred here (elemAt values 1) (head values) then
         1575|           head values
             |           ^
         1576|         else

       … while evaluating the attribute 'value'

         at /nix/store/q43321jn4spvsf1vzmiv56cgm4wgiss3-source/lib/modules.nix:821:9:

          820|     in warnDeprecation opt //
          821|       { value = addErrorContext "while evaluating the option `${showOption loc}':" value;
             |         ^
          822|         inherit (res.defsFinal') highestPrio;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: a 'x86_64-linux' with features {} is required to build '/nix/store/s4hs4qxwr3snrcfg5f6dsp41nhb5fczi-converted.json.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

repo: nix-community/nixos-anywhere


text:
WIP plugins/indent-o-matic: init

Adds indent-o-matic will resolve: https://github.com/nix-community/nixvim/issues/1143

require('indent-o-matic').setup {
    -- Global settings (optional, used as fallback)
    max_lines = 2048,
    standard_widths = { 2, 4, 8 },

    -- Disable indent-o-matic for LISP files
    filetype_lisp = {
        max_lines = 0,
    },

    -- Only detect 4 spaces and tabs for Rust files
    filetype_rust = {
        standard_widths = { 4 },
    },

    -- Don't detect 8 spaces indentations inside files without a filetype
    filetype_ = {
        standard_widths = { 2, 4 },
    },
}

Anyone got any tips on how to handle the per filetype configs they have in this plugin?

You are not forced to create nix options for every upstream plugin option. You can simply add it to the example to show that it can be done. settings is a freeformType, this means that users can add arbitrary keys even if they are not defined in nix.

Right I think this is good to go now if you are happy with it.

@GaetanLepage @traxys any further tweaks required or is this good to go?

source: gharchive/pull-request

authors: alisonjenkins traxys

repo: nix-community/nixvim


text:
Removes SIGSTKFLT when cross-compiling to MIPS.

This is the only failing error when cross compiling to Tessel, which is a MIPS-based architecture.
Apparently libc does not have SIGSTKFLT (Term Stack fault on coprocessor) on this system, and indeed it does not have a coprocessor.
This commit should preserve the semantics for all other architectures. Potentially this can be inverted in the future if more platforms are added to only whitelist ARM, x86, x86_64, but I thought this conditional would suffice.

@fiveop Looks like this condition is not possible on MIPS as an architecture, so this change seems to make sense.  https://www.linux-mips.org/archives/linux-mips/2006-05/msg00345.html

I wish there was a better way to manage this (and there might be through macro magic), but I'm good with this change today in order to fix mips @homu r+
I still need to revisit getting the CI tests for MIPS to pass.  Until then, I would not say we officially support MIPS.  I would like to, of course.

:pushpin: Commit 1f2d896 has been approved by posborne

:zap: Test exempted - status

Thanks for finding the reference for this @posborne. If I can help with fixing the MIPS CI, lmk what I should look at.

If I can help with fixing the MIPS CI, lmk what I should look at.

Sure, that task is basically getting  CI passing.  The infrastructure built on top of docker is already present and you can run CI for mips by doing the following (also read https://github.com/nix-rust/nix/tree/master/ci):
 $ DOCKER_IMAGE=posborne/rust-cross:mips \
      RUST_VERSION=1.7.0 \
      RUST_TARGET=mips-unknown-linux-gnu ci/run-docker.sh

authors: homu posborne tcr

repo: nix-rust/nix


text:
Simulation Render scale isn't right

This may be an artifact of the ghost images in the simulation, but when you change the display scale of the render any previous simulations run have increased vector sizes that are disproportionate to their relative size.  however active objects have the proper vector displayed (at an appropriate scale).
However when large scales are used, the velocity vector may not be seen.  The solution for this may be to translate the vector to the surface of the object (although that may require a fair bit of calculations? ie which direction to translate it)  Another solution may be to create a unit vector in the velocity direction and scale that to be of length (Radius + actual velocity)

No longer relevant.

repo: njewsbury/gravitational-simulation


text:
Press page turn to open next comic does not work anymore (comic view)

When you press page turn at the end of a comic Bubble asks if it should open the previous comic not the next. Swipe page turn at the end works correctly as does press page turn in manga view.

Sorry, my bad. The bug has already been fixed, will work properly in the next release.

fixed in v1.4.0

repo: nkanaev/bubble


text:
Are there not annotated data for lesion segmentation or lesion detection in this dataset?

Hi, it's a great dataset!
I have downloaded the dataset and extracted it. However, there is only a DR_grading folder. Are there not annotated data for lesion segmentation or lesion detection in this dataset? or just I should re-download it by some file error?

I'm sorry. It's my fault. After I re-downloaded and extracted it, the '''lesion_detection''' and '''lesion_segmentation''' folder are shown.

repo: nkicsl/DDR-dataset


text:
Doesn't work with bluetooth headphones

Doesn't record when you have bluetooth headphones (like airpods or beats) connected. Keeps printing out "Could not write audio data, recording aborted" from the didFailToRecord delegate method. Do you have any idea how I can include support for airpods or other bluetooth devices?
I am currently using this in an app and would love to use it in production, but I can't release it until this is fixed.
Thank you so much, any help is appreciated!

Replace  "try AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .videoRecording, options: [.mixWithOthers, .defaultToSpeaker, .allowBluetooth, .allowAirPlay, .allowBluetoothA2DP])"   this line with   "try AVAudioSession.sharedInstance().setCategory(.playAndRecord, mode: .videoRecording, options: [])"
In Capture.swift file.

repo: nkmrh/FilterCam


text:
I'm stuck with u8g2

Hi,
I would like to use 8ug2 in SPI mode. I have copied Your files into clean project but it crashes during SPI initialization. I have tried VSPI and HSPI, I'm pretty sure my PIN's are ok because this board works when I compile it for Arduino project.
On Arduino I'm using this board:
U8G2_SH1106_128X64_VCOMH0_1_4W_HW_SPI u8g2(U8G2_R0, /* cs=*/ 0, /* dc=*/21, /* reset=*/ 5);
This is my main.c
// CLK - GPIO14
#define PIN_CLK 18

// MOSI - GPIO 13
#define PIN_MOSI 23

// RESET - GPIO 26
#define PIN_RESET 5

// DC - GPIO 27
#define PIN_DC 21

// CS - GPIO 15
#define PIN_CS 0

static char tag[] = "test_SSD1306";

void task_test_SSD1306(void *ignore) {
        u8g2_esp32_hal_t u8g2_esp32_hal = U8G2_ESP32_HAL_DEFAULT;
        u8g2_esp32_hal.clk   = PIN_CLK;
        u8g2_esp32_hal.mosi  = PIN_MOSI;
        u8g2_esp32_hal.cs    = PIN_CS;
        u8g2_esp32_hal.dc    = PIN_DC;
        u8g2_esp32_hal.reset = PIN_RESET;
        u8g2_esp32_hal_init(u8g2_esp32_hal);

        u8g2_t u8g2; // a structure which will contain all the data for one display
        u8g2_Setup_sh1106_128x64_vcomh0_f(
                &u8g2,
                U8G2_R0,
                u8g2_esp32_spi_byte_cb,
                u8g2_esp32_gpio_and_delay_cb);  // init u8g2 structure

        u8g2_InitDisplay(&u8g2); // send init sequence to the display, display is in sleep mode after this,

        u8g2_SetPowerSave(&u8g2, 0); // wake up display
        u8g2_ClearBuffer(&u8g2);
        u8g2_DrawBox(&u8g2, 10,20, 20, 30);
        u8g2_SetFont(&u8g2, u8g2_font_ncenB14_tr);
        u8g2_DrawStr(&u8g2, 0,15,"Hello World!");
        u8g2_SendBuffer(&u8g2);

        ESP_LOGD(tag, "All done!");
        vTaskDelete(NULL);
}
void app_main(void)
{

        //vTaskDelay(1000/portTICK_PERIOD_MS);

        xTaskCreatePinnedToCore(&task_test_SSD1306, "task_simple_tests_ssd1306", 8048, NULL, 5, NULL, 0);
}

This is output during start on VSPI and HSPI
rst:0xc (SW_CPU_RESET),boot:0x16 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:4504
load:0x40078000,len:0
ho 12 tail 0 room 4
load:0x40078000,len:12976
entry 0x40078d88
I (53) cpu_start: Pro cpu up.
I (53) cpu_start: Single core mode
I (53) heap_init: Initializing. RAM available for dynamic allocation:
D (56) heap_init: New heap initialised at 0x3ffae6e0
I (61) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
D (67) heap_init: New heap initialised at 0x3ffb22a8
I (72) heap_init: At 3FFB22A8 len 0002DD58 (183 KiB): DRAM
I (79) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (85) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
D (91) heap_init: New heap initialised at 0x4008864c
I (96) heap_init: At 4008864C len 000179B4 (94 KiB): IRAM
I (102) cpu_start: Pro cpu start user code
D (114) clk: RTC_SLOW_CLK calibration value: 3619085
D (49) intr_alloc: Connected src 46 to int 2 (cpu 0)
D (49) intr_alloc: Connected src 57 to int 3 (cpu 0)
D (50) intr_alloc: Connected src 24 to int 9 (cpu 0)
I (55) cpu_start: Starting scheduler on PRO CPU.
D (60) heap_init: New heap initialised at 0x3ffe0440
D (64) heap_init: New heap initialised at 0x3ffe4350
D (69) intr_alloc: Connected src 16 to int 12 (cpu 0)
D (75) u8g2_hal: gpio_and_delay_cb: Received a msg: 40, arg_int: 0, arg_ptr: 0x0
I (82) gpio: GPIO[0]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 1| Intr:0
I (91) gpio: GPIO[5]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 1| Intr:0
I (100) gpio: GPIO[21]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 1| Intr:0
D (110) u8g2_hal: spi_byte_cb: Received a msg: 20, arg_int: 0, arg_ptr: 0x0
E (117) err: esp_err_t = 257
assertion "0 && "spi_bus_initialize(VSPI_HOST, &bus_config, 1)"" failed: file "/home/franki/apps/qt/examples/SPI-sh1106/main/./u8g2_esp32_hal.c", line 56, function: u8g2_esp32_spi_byteb
abort() was called at PC 0x400d2267 on core 0

Backtrace: 0x40086c14:0x3ffb68f0 0x40086d13:0x3ffb6910 0x400d2267:0x3ffb6930 0x400d2083:0x3ffb6960 0x400dd135:0x3ffb69d0 0x400dc742:0x3ffb69f0 0x400dcfdd:0x3ffb6a10 0x400e1f4d:0x3ffb6a0

Rebooting...

rst:0xc (SW_CPU_RESET),boot:0x16 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:4504
load:0x40078000,len:0
ho 12 tail 0 room 4
load:0x40078000,len:12976
entry 0x40078d88
I (53) cpu_start: Pro cpu up.
I (53) cpu_start: Single core mode
I (53) heap_init: Initializing. RAM available for dynamic allocation:
D (56) heap_init: New heap initialised at 0x3ffae6e0
I (61) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
D (67) heap_init: New heap initialised at 0x3ffb22a8
I (72) heap_init: At 3FFB22A8 len 0002DD58 (183 KiB): DRAM
I (79) heap_init: At 3FFE0440 len 00003BC0 (14 KiB): D/IRAM
I (85) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
D (91) heap_init: New heap initialised at 0x4008864c
I (96) heap_init: At 4008864C len 000179B4 (94 KiB): IRAM
I (102) cpu_start: Pro cpu start user code
D (114) clk: RTC_SLOW_CLK calibration value: 3613862
D (49) intr_alloc: Connected src 46 to int 2 (cpu 0)
D (49) intr_alloc: Connected src 57 to int 3 (cpu 0)
D (50) intr_alloc: Connected src 24 to int 9 (cpu 0)
I (55) cpu_start: Starting scheduler on PRO CPU.
D (60) heap_init: New heap initialised at 0x3ffe0440
D (64) heap_init: New heap initialised at 0x3ffe4350
D (69) intr_alloc: Connected src 16 to int 12 (cpu 0)
D (75) u8g2_hal: gpio_and_delay_cb: Received a msg: 40, arg_int: 0, arg_ptr: 0x0
I (82) gpio: GPIO[0]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 1| Intr:0
I (91) gpio: GPIO[5]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 1| Intr:0
I (100) gpio: GPIO[21]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 1| Intr:0
D (110) u8g2_hal: spi_byte_cb: Received a msg: 20, arg_int: 0, arg_ptr: 0x0
E (117) err: esp_err_t = 257
assertion "0 && "spi_bus_initialize(HSPI_HOST, &bus_config, 1)"" failed: file "/home/franki/apps/qt/examples/SPI-sh1106/main/./u8g2_esp32_hal.c", line 56, function: u8g2_esp32_spi_byteb
abort() was called at PC 0x400d2267 on core 0

Backtrace: 0x40086c14:0x3ffb68f0 0x40086d13:0x3ffb6910 0x400d2267:0x3ffb6930 0x400d2083:0x3ffb6960 0x400dd135:0x3ffb69d0 0x400dc742:0x3ffb69f0 0x400dcfdd:0x3ffb6a10 0x400e1f4d:0x3ffb6a0

Rebooting...

I would be grateful if You can give my any tpis why it refuses to work ;)
It looks like an memory error:
#define ESP_ERR_NO_MEM          0x101
Best Regards
Marek

I am using this HAL with the ESP_IDF.  I am getting the same error as Marek.  However I noticed that I am only getting this error when the ESP32 logging component is set to anything other than Info.  When the Log level is set to Info, the code runs great.
Does anyone have any idea why this is happening and what can be done to fix it?

Hi Jim
I will try to use U8G2 library again, because due to this problem we had to use something else in our project. Can You share your code that works eg. on a github.
Best Regards
Marek

Here is a turnkey project based on Neil's example:
 git clone --recursive https://github.com/JimGaluska/u8g2-esp_idf-example.git

This will run as is, however if you change the log verbosity to anything other than 'Info' the example will crash and burn at the same point, and in the same way as your issue.
Good luck.
Jim

I think I have found out what the issue is and have a fix or workaround (however you want to look at it).
First off, I found out the the issue only happens when I have set the ESP32 clock speed down to the slowest setting; 80Mhz.  At higher clock speeds the issue goes away.
The following is a snippet from the file test_SSD1306.c:
void task_test_SSD1306(void *ignore) {
    u8g2_esp32_hal_t u8g2_esp32_hal = U8G2_ESP32_HAL_DEFAULT;
    u8g2_esp32_hal.clk   = PIN_CLK;
    u8g2_esp32_hal.mosi  = PIN_MOSI;
    u8g2_esp32_hal.cs    = PIN_CS;
    u8g2_esp32_hal.dc    = PIN_DC;
    u8g2_esp32_hal.reset = PIN_RESET;
    u8g2_esp32_hal_init(u8g2_esp32_hal);

    u8g2_t u8g2; // a structure which will contain all the data for one display
    u8g2_Setup_ssd1306_128x64_noname_f(
            &u8g2,
            U8G2_R0,
            u8g2_esp32_spi_byte_cb,
            u8g2_esp32_gpio_and_delay_cb);  // init u8g2 structure

    u8g2_InitDisplay(&u8g2); // send init sequence to the display, display is in sleep mode after this,

Function u8g2_esp32_hal_init is implement in the hal as follows::
void u8g2_esp32_hal_init(u8g2_esp32_hal_t u8g2_esp32_hal_param) {
    u8g2_esp32_hal = u8g2_esp32_hal_param;
} // u8g2_esp32_hal_init

As you can see, u8g2_esp32_hal_init copies the structure that is passed in by value to a local copy.  My theory is that function u8g2_esp32_hal_init returns before the entire contents of the passed in structure is copied to the local structure..  When function u8g2_InitDisplay is called, it makes a callback into the hal where the structure is used to initialize the spi interface before it is fully copied.
My fix/workaround is to add a delay before between when functions u8g2_esp32_hal_init and u8g2_InitDisplay is called as follows:
void task_test_SSD1306(void *ignore) {
    u8g2_esp32_hal_t u8g2_esp32_hal = U8G2_ESP32_HAL_DEFAULT;
    u8g2_esp32_hal.clk   = PIN_CLK;
    u8g2_esp32_hal.mosi  = PIN_MOSI;
    u8g2_esp32_hal.cs    = PIN_CS;
    u8g2_esp32_hal.dc    = PIN_DC;
    u8g2_esp32_hal.reset = PIN_RESET;
    u8g2_esp32_hal_init(u8g2_esp32_hal);
    vTaskDelay(1);                                           // <== Insert delay here

    u8g2_t u8g2; // a structure which will contain all the data for one display
    u8g2_Setup_ssd1306_128x64_noname_f(
            &u8g2,
            U8G2_R0,
            u8g2_esp32_spi_byte_cb,
            u8g2_esp32_gpio_and_delay_cb);  // init u8g2 structure

    u8g2_InitDisplay(&u8g2); // send init sequence to the display, display is in sleep mode after this,

On my system, this make the problem go away in all cases that I have tried.
One interesting observation is that placing the delay inside the function u8g2_esp32_hal_init does not work:
void u8g2_esp32_hal_init(u8g2_esp32_hal_t u8g2_esp32_hal_param) {
    u8g2_esp32_hal = u8g2_esp32_hal_param;
    vTaskDelay(100);                                          // <== Large delays don't work here
} // u8g2_esp32_hal_init

Perhaps vTaskDelay takes away CPU time to the structure copy operation when it is used inside the hal init function and it doesn't remove CPU time when it is used at the higher level.

Jim, many thanks for Your valuable help. We are using u8g2 now in our project.
Best Regards
Marek

@JimGaluska Your fix did it for me as well. I spent a quarter day on this before adding vTaskDelay(1) after calling init and fixing it all. Thanks. Unsure why it works , but your explanation seems good.

Another thing to check: are all your struct elements properly set or cleared?
I ran into a similar crash using the i2c version. I found that u8g2_esp32_i2c_byte_cb() (in u8g2_esp32_hal.c) created a struct i2c_config_t conf, but didn't initialize the clk_flags member. When the call succeeded, clk_flags was 0x0; when it failed, clk_flags was something odd like 0x3FFB55F0. Setting clk_flags to 0 let me do away with cTask delays and printfs that had mysteriously been preventing crashes (apparently by pushing stuff around on the stack enough that, when it worked, clk_flags wound up in a clean area).
I saw a similar fix for the SPI version in a pull request: https://github.com/nkolban/esp32-snippets/pull/884

Another thing to check: are all your struct elements properly set or cleared?
I ran into a similar crash using the i2c version. I found that u8g2_esp32_i2c_byte_cb() (in u8g2_esp32_hal.c) created a struct i2c_config_t conf, but didn't initialize the clk_flags member. When the call succeeded, clk_flags was 0x0; when it failed, clk_flags was something odd like 0x3FFB55F0. Setting clk_flags to 0 let me do away with vTaskDelays and printfs that had mysteriously been preventing crashes (apparently by pushing stuff around on the stack enough that, when it worked, clk_flags wound up in a clean area).
I saw a similar fix for the SPI version in a pull request: #884

Thanks Adam - this was exactly the problem I had and setting conf.clk_flags=0 as per your suggestion has fixed it

authors: JimGaluska adamwilt erichiller frankipl monepic


text:
BLE Gateway - Impossible to use WiFi and BLE at the same time

Hi, as in the title I'm trying to create a BLE gateway. The problem is that if I am already connected to the WiFi the BLE part doesn't work.
Here is the main code, see also my FlowerCare library if necessary.
The following is the terminal output I'm able to obtain:
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:812
load:0x40078000,len:0
load:0x40078000,len:11404
entry 0x40078a28
Create flora 1
Create flora 2
Create flora 3
Connecting to FASTWEB-Brunelli
..Connected!
Setup done!
Get data flora 1
E (35770) BT: btc_gattc_call_handler()

After some testing I located the "error". The program blocks inside FC_1->getData(), at the call to _BLEClient->connect(*_addr) inside FlowerCare_BLE.cpp.
Going deeper I found that it blocks inside BLEClient.cpp when calling m_semaphoreOpenEvt.wait("connect"). The problem is that despite I located the error I don't know what to do, can you help me?

I would start with commenting out this line and see what would happend:
m_semaphoreOpenEvt.wait("connect")
This wont fix the issue, maybe even cause some other issues but it can give you more info.

I don't know if it is a good idea, but I can try it. I was hoping that someone else encountered this problem before and resolved it. I know that there is some applications that use BLE and Wifi at the same time (like this) but with arduino core I'm unable to get it working without modify the "standard" BLE library for arduino core

https://github.com/nkolban/esp32-snippets/blob/master/cpp_utils/tests/BLETests/SampleClientWithWiFi.cpp

I did run into an issue with BLE+WiFi and found that ensuring my BLE connection was complete before connecting to WiFi avoided the problem.  That was a while ago (6 months perhaps) and I haven't retested since.  Doing that, my BLE->WiFi gateway now runs happily for hours on end!

authors: Brunez3BD chegewara johnhmacleod


text:
Implicit operator for option type

Would it be very awful if Option type had implicit operator for converting value from T => Option<T>?
public static implicit operator Option<T>(T value)
{
     return Option.Some(value);
}

It would allow to use shortcut for assigning option values. Instead of Option<bool> opt = Option.Some(true); I could use Option<bool> opt = true;

Hi Janno,
I will consider this. In general I am not a big fan of implicit conversions, and it does leave some questions to what the conversion behavior should be, e.g. should it return value.Some() or maybe even value.SomeNotNull() etc.
Usually I simply use the extension methods, e.g. var opt = true.Some();, which leave very little syntactic overhead.
In my opinion, the only situation in which an implicit conversion "looks nicer", is when converting the return value (e.g. return true vs return true.Some()). However, this breaks the symmetry between .Some() and .None(), with very little benefit. As such, I do like explicitly choosing the conversion (.Some(), .None(), .SomeNotNull(), .SomeWhen(...), and so on).
Nevertheless, I will give it some thoughts. You are welcome to come up with more pros/cons.

Nils

Ok. This is all reasonable and those extension methods for conversion really help a lot. Since there are many possible interpretations for this implicit behavior (some might prefer null conversion to None, others might expect Some(null)), then it's hard to make a preference. And when one choice is made it might introduce some unexpected behaviors for end users.
Considering that, I agree, that explicit conversion is much safer way to go and this question is resolved for me.

Waiting for .NET Core compatible NuGet packages to be released :smile:

@janno-p Just FYI, .NET Core support has just been released. Enjoy :)

Great! I will.

authors: janno-p nlkl

repo: nlkl/Optional


text:
The MSVC team recently test JSON project failed to run test on release configuration on windows_x64.

Description
Hi All,
JSON failed to run test 'test-diagnostics_cpp11','test-json_pointer_cpp11' and 'test-unicode1_cpp11'on release configuration with MSVC on windows x64. Could you please help look at this issue or provide some workarounds? Thanks in advance.
Reproduction steps

git clone https://github.com/nlohmann/json.git F:\gitP\nlohmann
set VSCMD_SKIP_SENDTELEMETRY=1 & "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=amd64 & set CL= /fsanitize=address /GS- /wd5072 & set LINK= /InferASanLibs /incremental:no /debug
mkdir&cd F:\gitP\nlohmann\json\build_amd64
cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.18362.0 -DJSON_BuildTests=On .. 2>&1
msbuild /m /p:Platform=x64 /p:Configuration=Release nlohmann_json.sln /t:Rebuild 2>&1
ctest -C Release --output-on-failure 2>&1

Expected vs. actual results
Expected test pass
Actual results :
The following tests FAILED:
8 - test-bson_cpp11 (Failed)
25 - test-diagnostics_cpp11 (Failed)
38 - test-json_pointer_cpp11 (Failed)
61 - test-unicode1_cpp11 (Failed)
Errors while running CTest
Minimal code example
1.C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\link.exe /ERRORREPORT:QUEUE /OUT:"F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe" /INCREMENTAL:NO /NOLOGO /LIBPATH:"F:\gitP\bitcoin\tools\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\lib" /LIBPATH:"F:\gitP\bitcoin\tools\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\lib\manual-link" /NATVIS:F:\gitP\nlohmann\json\nlohmann_json.natvis kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib "F:\gitP\bitcoin\tools\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\lib\*.lib" /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /PDB:"F:/gitP/nlohmann/json/build_amd64/tests/Release/test-diagnostics_cpp11.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"F:/gitP/nlohmann/json/build_amd64/tests/Release/test-diagnostics_cpp11.lib" /MACHINE:X64  /machine:x64 "test-diagnostics_cpp11.dir\Release\unit-diagnostics.obj" F:\gitP\nlohmann\json\build_amd64\tests\test_main.dir\Release\unit.obj

2.test-diagnostics_cpp11.exe

Error messages
==86508==ERROR: AddressSanitizer: container-overflow on address 0x00e159afb444 at pc 0x7ffc23ec68fc bp 0x00e159afadb0 sp 0x00e159afa530
WRITE of size 2 at 0x00e159afb444 thread T0
    #0 0x7ffc23ec6929  (C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x180046929)
    #1 0x7ff6ed53b9dc in std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>::replace(unsigned __int64, unsigned __int64, char const *const, unsigned __int64) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x1400bb9dc)
    #2 0x7ff6ed53ad42 in std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>::replace(unsigned __int64, unsigned __int64, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> const &) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x1400bad42)
    #3 0x7ff6ed4ebefd in nlohmann::detail::replace_substring<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>(class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> &, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> const &, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> const &) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x14006befd)
    #4 0x7ff6ed4d7077 in nlohmann::detail::escape<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>(class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x140057077)
    #5 0x7ff6ed4bc83a in std::accumulate<class std::reverse_iterator<class std::_Vector_iterator<class std::_Vector_val<struct std::_Simple_types<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>>>>, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, class <lambda_0b07af80a1b5e289a8e1fd0031570a68>>(class std::reverse_iterator<class std::_Vector_iterator<class std::_Vector_val<struct std::_Simple_types<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>>>>, class std::reverse_iterator<class std::_Vector_iterator<class std::_Vector_val<struct std::_Simple_types<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>>>>, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, class <lambda_0b07af80a1b5e289a8e1fd0031570a68>) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x14003c83a)
    #6 0x7ff6ed4d40af in nlohmann::detail::exception::diagnostics<class nlohmann::basic_json<class std::map, class std::vector, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, bool, __int64, unsigned __int64, double, class std::allocator, struct nlohmann::adl_serializer, class std::vector<unsigned char, class std::allocator<unsigned char>>>>(class nlohmann::basic_json<class std::map, class std::vector, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, bool, __int64, unsigned __int64, double, class std::allocator, struct nlohmann::adl_serializer, class std::vector<unsigned char, class std::allocator<unsigned char>>> const *) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x1400540af)
    #7 0x7ff6ed4ca21a in nlohmann::detail::type_error::create<class nlohmann::basic_json<class std::map, class std::vector, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, bool, __int64, unsigned __int64, double, class std::allocator, struct nlohmann::adl_serializer, class std::vector<unsigned char, class std::allocator<unsigned char>>> const *, 0>(int, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> const &, class nlohmann::basic_json<class std::map, class std::vector, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, bool, __int64, unsigned __int64, double, class std::allocator, struct nlohmann::adl_serializer, class std::vector<unsigned char, class std::allocator<unsigned char>>> const *) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x14004a21a)
    #8 0x7ff6ed4d7642 in nlohmann::detail::from_json<class nlohmann::basic_json<class std::map, class std::vector, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, bool, __int64, unsigned __int64, double, class std::allocator, struct nlohmann::adl_serializer, class std::vector<unsigned char, class std::allocator<unsigned char>>>>(class nlohmann::basic_json<class std::map, class std::vector, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, bool, __int64, unsigned __int64, double, class std::allocator, struct nlohmann::adl_serializer, class std::vector<unsigned char, class std::allocator<unsigned char>>> const &, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> &) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x140057642)
    #9 0x7ff6ed4d8aa1 in nlohmann::basic_json<class std::map, class std::vector, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, bool, __int64, unsigned __int64, double, class std::allocator, struct nlohmann::adl_serializer, class std::vector<unsigned char, class std::allocator<unsigned char>>>::get_impl<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, 0>(struct nlohmann::detail::priority_tag<0>) const (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x140058aa1)
    #10 0x7ff6ed485740  (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x140005740)
    #11 0x7ff6ed55dc93 in doctest::Context::run(void) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x1400ddc93)
    #12 0x7ff6ed58b627 in main (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x14010b627)
    #13 0x7ff6ed5dd20f in __scrt_common_main_seh D:\a01\_work\2\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #14 0x7ffc55c57973  (C:\Windows\System32\KERNEL32.DLL+0x180017973)
    #15 0x7ffc58a0a2f0  (C:\Windows\SYSTEM32\ntdll.dll+0x18005a2f0)

Address 0x00e159afb444 is located in stack of thread T0 at offset 164 in frame
    #0 0x7ff6ed4bbfaf in std::accumulate<class std::reverse_iterator<class std::_Vector_iterator<class std::_Vector_val<struct std::_Simple_types<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>>>>, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, class <lambda_0b07af80a1b5e289a8e1fd0031570a68>>(class std::reverse_iterator<class std::_Vector_iterator<class std::_Vector_val<struct std::_Simple_types<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>>>>, class std::reverse_iterator<class std::_Vector_iterator<class std::_Vector_val<struct std::_Simple_types<class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>>>>>, class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>, class <lambda_0b07af80a1b5e289a8e1fd0031570a68>) (F:\gitP\nlohmann\json\build_amd64\tests\Release\test-diagnostics_cpp11.exe+0x14003bfaf)

  This frame has 8 object(s):
    [32, 40) '_UFirst'
    [48, 56) '_ULast'
    [64, 96) 'compiler temporary'
    [80, 88) '_Right'
    [96, 104) '_Right'
    [112, 144) 'compiler temporary'
    [128, 160) 'compiler temporary'
    [144, 145) 'compiler temporary' <== Memory access at offset 164 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp, SEH and C++ exceptions *are* supported)
HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x180046929)
Shadow bytes around the buggy address:
  0x02f027bdf630: 00 00 f1 f1 f1 f1 f8 f8 f8 f8 f2 f2 f2 f2 f8 f8
  0x02f027bdf640: f8 f8 f2 f2 f2 f2 03 fc 00 00 f2 f2 f2 f2 02 fc
  0x02f027bdf650: 00 00 f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00
  0x02f027bdf660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x02f027bdf670: 00 00 00 00 f1 f1 f1 f1 00 f2 00 f2 00 00 00 00
=>0x02f027bdf680: f2 f2 f2 f2 00 f2 00 f2[04]fc 00 00 f2 f2 f2 f2
  0x02f027bdf690: 00 00 00 00 f2 f2 f2 f2 01 f3 f3 f3 f3 00 00 00
  0x02f027bdf6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x02f027bdf6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x02f027bdf6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x02f027bdf6d0: 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==86508==ABORTING

Compiler and operating system
Visual Studio 2019, Window10
Library version
git commit : af34396
Validation

[X] The bug also occurs if the latest version from the develop branch is used.
[X] I can successfully compile and run the unit tests.

JSON_amd64.zip
build and test  log file

Thanks for reporting. Can you please try Release mode just to be sure?

The error seems to be happening in replace_substring which I've recently modified. Could you re-run your test with the latest release version (3.10.5) and let me know if the error exists there as well?

Thanks for reporting. Can you please try Release mode just to be sure?

Do you mean Debug?

There's a high likelihood that this is a false positive.
Are you sure what you're doing is supported? I can find documentation that specifically states only x86 is supported (but that may have changed in subsequent releases).

Thanks for reporting. Can you please try Release mode just to be sure?

Do you mean Debug?

No, Release. We sometimes had issues that MSVC took too much memory or even had a stack overflow when executing certain tests in Debug mode.

Thanks for reporting. Can you please try Release mode just to be sure?

Do you mean Debug?

No, Release. We sometimes had issues that MSVC took too much memory or even had a stack overflow when executing certain tests in Debug mode.

I ask because this bug is specific to an amd64 Release build. Also, MSVC ASAN is apparently only supported by i368 Release builds. I'm just not sure if that has changed in subsequent VS 2019 releases (I suspect not, because the MS blog post was last updated in 2021).
Since ASAN is not finding any issues on Linux and container-overflow is known to produce false positives because of missing ASAN annotations or mixing standard library code built with and without ASAN due to inlining, I'm confident that this bug report is invalid.
Additionally, the code in question is not particularly complicated and some of the parameters are literals. I don't see how it could fail.
With StringType = std::string :
template<typename StringType>
inline void replace_substring(StringType& s, const StringType& f,
                              const StringType& t)
{
    JSON_ASSERT(!f.empty());
    for (auto pos = s.find(f);                // find first occurrence of f
            pos != StringType::npos;          // make sure f was found
            s.replace(pos, f.size(), t),      // replace with t, and
            pos = s.find(f, pos + t.size()))  // find next occurrence of f
    {}
}

template<typename StringType>
inline StringType escape(StringType s)
{
    replace_substring(s, StringType{"~"}, StringType{"~0"});
    replace_substring(s, StringType{"/"}, StringType{"~1"});
    return s;
}

Thanks for reporting. Can you please try Release mode just to be sure?

Do you mean Debug?

No, Release. We sometimes had issues that MSVC took too much memory or even had a stack overflow when executing certain tests in Debug mode.

Thanks for your reply and suggestion.
This error was found when compiling this project in the latest development version of VS, and is only reproduced in ASAN mode. the same error is reported in debug mode, I apologize for not clarifying this issue, I am trying to solve the problem by setting ASAN_OPTIONS=detect_container_overflow=0.

Please try also in Release mode.

Please try also in Release mode.

Niels, he did. It's in the title. :-) That's why I had to ask for clarification the last time.

Oh boy, sorry for the confusion…

The error seems to be happening in replace_substring which I've recently modified. Could you re-run your test with the latest release version (3.10.5) and let me know if the error exists there as well?

@falbrechtskirchinger  I've tried the latest release version b5364fa of release/3.10.5 branch. But the error still exist.  And the failed error are same.
As you said 'The error seems to be happening in replace_substring which I've recently modified.' Can you provide a link to your commit for your changes? We want to try to see if the commit before this modification has this problem.
Thanks,
Lin

@falbrechtskirchinger I've tried the latest release version b5364fa of release/3.10.5 branch. But the error still exist. And the failed error are same.
As you said 'The error seems to be happening in replace_substring which I've recently modified.' Can you provide a link to your commit for your changes? We want to try to see if the commit before this modification has this problem.

If you've tested 3.10.5, you've already done so. My change is unreleased and also immaterial. I still think this bug report is (probably) invalid for the reasons stated above and I'm only qualifying my response because I haven't personally checked the MSSTL/libstdc++ sources for ASan annotations.

The only thing I can see is that asan may be tripping on replacing the last character of the string with a two-character string.  If it's not detecting that the string container has already been expanded to accept that, it would detect that as an overflow error.

But that would still be an issue outside the library, right? AFAICT, std::string::replace should deal with resizing the string.
I did grep through libstdc++ trunk, and std::vector is still the only annotated container. Maybe I'll check MSSTL next.

Correct, I don't see anything wrong with the library code, just trying to figure out what it might be detecting.

authors: Vincent9802 falbrechtskirchinger gregmarr nlohmann spacelg

repo: nlohmann/json


text:
Unhandled response

I'm working with this lib in a project using it for the websockets RTM API implementation, and when receiving an error response from the api this line https://github.com/nlopes/slack/blob/master/websocket.go#L184 Is making the app crash.
The error responses from slack are also going here and I think it should handle the error gracefully.
For instance when sending a message with empty text, slacks answer is: {"ok":false,"reply_to":1,"error":{"code":2,"msg":"message text is missing"}}
Making the Unmarshal fail and exiting the app.
I think that the unmarshal error could be relaxed and at least reach https://github.com/nlopes/slack/blob/master/websocket.go#L194-L195
Said that, agree that error should not end up in that bucket.

@victorcoder Thanks for the report, I will be taking a look shortly

@victorcoder  This commit (https://github.com/nlopes/slack/commit/b8c4312a0b54fc89302947fabe02e83c766bda50) should fix the issue. I also made sure errors are being passed down to you. Example in https://github.com/nlopes/slack/blob/master/examples/websocket/websocket.go#L46

@nlopes, tested it and it fixes the error! Thanks
Just to let you know, I'm using https://github.com/danryan/hal and I used your API implementation to power the new RTM adaptor https://github.com/danryan/hal/pull/42
It's working like a charm

Awesome!

authors: nlopes victorcoder

repo: nlopes/slack


text:
Percentages in FreqDist.plot()

Figure 1-4 displays percentages, even though this was dropped in r6689.
Reinstate this functionality somewhere (e.g. as a flag for FreqDist.plot(),
or perhaps in ProbDist; though its not clear how to plot smoothed probdists).
http://groups.google.com/group/nltk-users/browse_thread/thread/8f3e3f2b531836a4
http://groups.google.com/group/nltk-users/browse_thread/thread/85ba122d808147c
Migrated from http://code.google.com/p/nltk/issues/detail?id=457earlier comments
StevenBird1 said, at 2009-12-10T23:48:08.000Z:
The figure has been corrected not to display percentages.
StevenBird1 said, at 2009-12-11T06:35:18.000Z:
Deleting "Book" tagzhai-run@163.com said, at 2010-01-15T10:14:57.000Z:
But how to change it back to percentage (now it is count, and different from the
example in the book). More over, while running FreqDist.plot, a warning prompts:
Warning (from warnings module):
  File "C:\Python26\lib\site-packages\matplotlib\dates.py", line 91
    import pytz.zoneinfo
ImportWarning: Not importing directory 'C:\Python26\lib\site-packages\pytz
\zoneinfo': missing __init__.py
Why

Closing, as we don't have capacity to take this on.

authors: alexrudnick stevenbird

repo: nltk/nltk


text:
Better Treebank tokenizer

Added detokenization abilities by reversing the regexes operations
Post-hoc tokens mapping to original sentence; might not be 100% correctly aligned, c.f. #1190
(moonshot) keeping explicit offset during tokenization.

[CI: retest]

[CI: retest]

@leondz Sorry for hijacking the PR, I've add the code from #1190 with a regex detokenizer. The credits for the align_tokens function should go to you though.

The PR is good for review now =)

Thanks for resolving all these long-outstanding problems @alvations.

authors: alvations stevenbird


text:
It will be very slow when I add 30600 512-dim features into it

I have init a HierarchicalNSW withm maxelements=40000, M=128, ef=100. and then call addPoint to add feature, at first, it is very fast, but when it goes to 30600, it will be stuck at there

BTW, python is ok, I am using c++

repo: nmslib/hnswlib


text:
OVS: Support OVS netdev datapath

The OVS netdev datapath will use TUN interface for OVS internal
interface, nmstate should merge this kernel TUN interface state with
user space OVS interface state, otherwise verification failure will
happen.
The netdev datapath is used for OVS DPDK.  Manually tested with OVS
DPDK on Intel E810-C card.
Integration test case included.

The rpmbuild failure of epel-8 and 9 is caused by outdated rust. It will be automatically resolved once RHEL 9.2 released.

A nit, not sure if it is possible to provide a querying example.

Nothing special, the tun interface is still show as ovs-interface.

repo: nmstate/nmstate


text:
Code of Conduct

There is a code of conduct in the repository, but it is not mentioned in the Readme or the Contributing instructions. I think this should be fixed to meet the requirements of openjournals/joss-reviews#1738.

Fixed in https://github.com/nnadeau/pybotics/commit/bcb62d974a14327af7eaaa48fe4033adc1fb52da

authors: CameronDevine nnadeau

repo: nnadeau/pybotics


text:
Manager must use copy constructor than shared_ptr

The manager must use copy constructor than a shared_ptr.
Using shared_ptr with manager means that the manager between the two models is intertwined. However, the two models.
It is allowed for models to use copy constructor to create a copy of the model, which would have them share the same manager.
In this case, modifying the model A would change the memory in the manager, invalidating model B and its manager.
Rather, if the manager is copy constructed, that would give the flexibility to use the same memory for the manager of both models A and B internally with same Tensor/shared_ptr but allow changes to the model/its manager without invalidating the other model.

:octocat: cibot: Thank you for posting issue #831. The person in charge will reply soon.

Making clear from the offline discussion:
Manager can share weights to be used across model (but not gradient, in&outs which can be seen temporary) thus allowing reusing memory.
 Model A = Backbone;
 Model B = Backbone;
 Model C; (c is classifier)
 Model D; (d is another classifier)
 A.append(C);
 B.append(D);

 A.train();
 B.train();

Showerthoughts:

having a lock_guard style idiom to manage temporaries (gradient, in&outs).
gradient optimization can't be done at compile time in this case

authors: kparichay taos-ci zhoonit

repo: nnstreamer/nntrainer


text:
Custom roles module

users are able to add a custom role with a custom name and color in a specific channel
custom roles are listed below the default role but are the user's highest colored role
each custom role may only be assigned to one user
the custom role is deleted when a user removes it

Basic role management in 18faa08.

Superseded by #74

repo: noahkw/botw-bot


text:
Having trouble getting any characteristic.read to work.

I have looked at the sample code for reading a characteristic and just can't seem to get it to work.  There is no error when I read however the data is always empty.  The following is the code that I am using to try and read.
var disconnectPeripheral = function (peripheral) {
    peripheral.disconnect(function(error) {
        console.log('Disconnected from device: ' + peripheral.uuid);
    });
};`

var findCharacteristic = function (characteristics, id) {
    for (var i  in characteristics) {
        if (characteristics[i].uuid === id) {
            return characteristics[i];
        }
    }
};

peripheral.connect(function (error) {
            if (error) {
                console.error(error);
            }

            console.log("Connected to device " + peripheral.uuid);

            peripheral.discoverAllServicesAndCharacteristics(function(error, services, characteristics) {
                if (error) {
                    console.error(error);
                    disconnectPeripheral(peripheral);
                }

                var versionCharacteristic = findCharacteristic(characteristics, '1005');

                if (versionCharacteristic) {

                    versionCharacteristic.on('data', (data, isNotification) => {
                        console.log(data.toString());
                    });

                    versionCharacteristic.read((error, versionData) => {
                        if (error) {
                            console.error(error);
                            disconnectPeripheral(peripheral);
                        }

                        console.log("Version: " + versionData.toString('utf8'));
                        disconnectPeripheral(peripheral);
                    })
                } else {
                    disconnectPeripheral(peripheral);
                }
            });
        });

My application logs out Version: with nothing after it.  I am using a Raspberry Pi 3 Model B running Raspbian Stretch, Noble 1.8.1, and NodeJS 8.11.1

I'm dumb

repo: noble/noble


text:
OHMR works unreliably

In rare cases ohmr executes without administrator permissions, thus triggering "no temperatures was found among CPUs".
Any ideas why?

That was ohmr bug. Figuring out a workaround.
Open Hardware Monitor Report

--------------------------------------------------------------------------------

Version: 0.8.0.2

--------------------------------------------------------------------------------

Common Language Runtime: 4.0.30319.34209
Operating System: Microsoft Windows NT 6.1.7601 Service Pack 1
Process Type: 32-Bit

--------------------------------------------------------------------------------

Ring0

Status: Extracting driver failed

Open Hardware Monitor Report

--------------------------------------------------------------------------------

Version: 0.8.0.2

--------------------------------------------------------------------------------

Common Language Runtime: 4.0.30319.34209
Operating System: Microsoft Windows NT 6.1.7601 Service Pack 1
Process Type: 32-Bit

--------------------------------------------------------------------------------

Ring0

Status: Installing driver "C:\OpenHardwareMonitorReport\OpenHardwareMonitorLib.sys" failed and file exists
First Exception: OpenSCManager returned zero.
Second Exception: OpenSCManager returned zero.

repo: nobodysu/zabbix-mini-IPMI

license: Unlicense


text:
Add test for onnx.HammingWindow

Adds e2e test for the onnx.HammingWindow op.

cc @renxida.

repo: nod-ai/SHARK-TestSuite


text:
Fix 215

A temporary fix for #215.
Sorry, had to create a new pull request because the old one was from my master branch. Oops.

Easy, because nan 2.0 support PR almost done. Just a few minutes, I have to test it on Linux. On Window all tests pased.

Of course it's failing on Linux. :) Maybe because I have iojs 2.5 there. Tomorow I can continue.

This will help for people trying to install right now. Thanks!

authors: TooTallNate feldgendler unbornchikken

repo: node-ffi/node-ffi


text:
Flow export Inconsistency with API

Hey All,
I have recently been using the export function in the UI of node-red while also using the HTTP API,
I noticed that the JSON format of the flow export is different than requesting the same flow via REST.
Moreover, one cant not use an exported flow from the UI in JSON and POST it to another node-red instance because it expects the REST format.
Is there a specific reason of why their representation are different?
First Representation
 [
  {
    "id": "e251ec3c.9a5c6",
    "type": "tab",
    "label": "Flow 2"
  },
  {
    "id": "a9aef78e.f25878",
    "type": "inject",
    "z": "e251ec3c.9a5c6",
    "name": "",
    "topic": "",
    "payload": "",
    "payloadType": "date",
    "repeat": "",
    "crontab": "",
    "once": false,
    "x": 114.5,
    "y": 253,
    "wires": [
      [
        "4ec97390.2daf9c"
      ]
    ]
  },
  {
    "id": "4ec97390.2daf9c",
    "type": "debug",
    "z": "e251ec3c.9a5c6",
    "name": "",
    "active": true,
    "console": "false",
    "complete": "false",
    "x": 272.5,
    "y": 218,
    "wires": [

    ]
  }
]

The Second representation
  "id": "e251ec3c.9a5c6",
  "label": "Flow 2",
  "nodes": [
    {
      "id": "a9aef78e.f25878",
      "type": "inject",
      "z": "e251ec3c.9a5c6",
      "name": "",
      "topic": "",
      "payload": "",
      "payloadType": "date",
      "repeat": "",
      "crontab": "",
      "once": false,
      "x": 114.5,
      "y": 253,
      "wires": [
        [
          "4ec97390.2daf9c"
        ]
      ]
    },
    {
      "id": "4ec97390.2daf9c",
      "type": "debug",
      "z": "e251ec3c.9a5c6",
      "name": "",
      "active": true,
      "console": "false",
      "complete": "false",
      "x": 272.5,
      "y": 218,
      "wires": [

      ]
    }
  ]
}```

Hi - this is a limitation of the current flow api that needs some rethinking and reworking. The individual flow api needed a representation that encapsulated a single flow. But changing the main /flows format which has existed since the very start was not something we could undertake lightly - too many things depend on it.
This leaves us in the current state caught between the two - this is one of the reasons we've not widely publicised the individual flow api; it exists and can be used but isn't the most useable api.
The design of the new flow format, and some of the thinking behind it, is here: https://github.com/node-red/node-red/wiki/Design%3A-Flow-file-format-v2
We'll revisit this api at some point - have captured a task on the whiteboard for it: https://trello.com/c/pmwMcnFk/86-tidy-up-flow-api

authors: SalehAly knolleary

repo: node-red/node-red


text:
[CSV node] Add support for parsing empty strings and null values

[ ] Bugfix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)

Proposed changes
Database replication use case
Current databases state
Database 1

name
e-mail
Language
Country

tmdoit
tmdoit@doit.pl
PL
Poland

Database 2

name
e-mail
Language
Country

tmdoit
tmdoit@doit.pl
PL
Poland

After "Database 1" update
Database 1

name
e-mail
Language
Country

tmdoit
""
NULL
NULL

Database 2

name
e-mail
Language
Country

tmdoit
tmdoit@doit.pl
PL
Poland

To reflect changes in "Database 2" I use CSV string: '§,"",,' where:

§: means field doesn't changed
"": means field is set to empty string
,,: means field is set to NULL value
,: means field is set to NULL value

Now using CSV node it's impossible to get null values and empty strings - in both cases there is no value returned.

Checklist

[ ] I have read the contribution guidelines
[x ] For non-bugfix PRs, I have discussed this change on the forum/slack team.
[ ] I have run grunt to verify the unit tests pass
[ ] I have added suitable unit tests to cover the new/changed functionality

Coverage decreased (-0.02%) to 77.498% when pulling 305b2609fa22a0a339ef7e386db38d0104ead02d on digitaloak:dev into 6912dec166e23e38d9fc1e0a626512b4ca988358 on node-red:dev.

Thanks - there was no need to change all the variable names - just the labels/translations - but thanks !

authors: coveralls dceejay tmdoit


text:
Module not found: Error: Can't resolve '../build/Debug/nodegit.node'

I am getting these errors when using this library
The Debug folder isn't even there. Anyone have any idea?
I am using Node 6.10.2 and npm 3.10.10:
ERROR in ./~/nodegit/dist/nodegit.js
Module not found: Error: Can't resolve '../package' in 'C:\Users\Dolan\Documents\Git-Proton\node_modules\nodegit\dist'
 @ ./~/nodegit/dist/nodegit.js 979:18-39
 @ ./src/app/common/git/git.service.ts
 @ ./src/app/common/git/git.module.ts
 @ ./src/app/app.module.ts
 @ ./src/main.ts
 @ multi ./src/main.ts

ERROR in ./~/nodegit/dist/nodegit.js
Module not found: Error: Can't resolve '../build/Debug/nodegit.node' in 'C:\Users\Dolan\Documents\Git-Proton\node_modules\nodegit\dist'
 @ ./~/nodegit/dist/nodegit.js 18:11-49
 @ ./src/app/common/git/git.service.ts
 @ ./src/app/common/git/git.module.ts
 @ ./src/app/app.module.ts
 @ ./src/main.ts
 @ multi ./src/main.ts

Anyone got any ideas? I am really stuck

Having the same issue in Mac OSX 10.12.6, using Electron.
Log:
┏ Renderer Process -----------

  Hash: 5e2f705eccaf53da0ab1
  Version: webpack 2.7.0
  Time: 8372ms
        Asset     Size  Chunks  Chunk Names
  renderer.js  3.09 MB       0  renderer
   styles.css   142 kB       0  renderer

  WARNING in ./~/promisify-node/index.js
  107:14-27 Critical dependency: the request of a dependency is an expression

  ERROR in ./~/nodegit/dist/nodegit.js
  Module not found: Error: Can't resolve '../build/Debug/nodegit.node' in '/Users/christian/Desktop/projects/FreshProject/node_modules/nodegit/dist'
   @ ./~/nodegit/dist/nodegit.js 18:11-49
   @ ./~/babel-loader/lib!./~/vue-loader/lib/selector.js?type=script&index=0!./src/renderer/components/Index.vue
   @ ./src/renderer/components/Index.vue
   @ ./src/renderer/router/index.js
   @ ./src/renderer/main.js
   @ multi ./.electron-vue/dev-client ./src/renderer/main.js
  Child html-webpack-plugin for "index.html":
           Asset     Size  Chunks  Chunk Names
      index.html  1.45 MB       0

┗ ----------------------------

@CrixuAMG I am using Electron too...
Perhaps thats the reason?

I've only recently picked up Electron, so I'm not really well known yet with process specific issues.
I'll take a look when I'm home and see if I can somehow solve the issue or provide some more information.

No luck so far, going to try compiling it locally sometime this week and see if I can make it run, going to let you know how it goes!

It seems like electron does support native modules though:
https://github.com/electron/electron/blob/master/docs/tutorial/using-native-node-modules.md
Now I am confused

Looks you are running a debug build of the node engine, and this will cause debug builds of modules.

How do you run a production build of nodegit?

I installed nodegit using npm install --dev nodegit, only adding --dev to the command listed on the README.md file.
Installing without --dev (after removing it and the node_modules folder) does not fix the issue.

I have that problem too.
This is strange, since there is no "Debug/nodegit.node", there is only "Release/nodegit.node" which makes a lot of sense. I have no idea though why it tries to include any debug things.

Oh, I managed to fix it.
The problem exist at all because I use Webpack (and Electron, but it wasn't the problem), which had some problems with resolving relative modules inside node_modules. What I need to do was, in Webpack conf:
const fs = require('fs')
// I use relative ../../ path to node_modules, because I use Yarn workspaces!!
// In your case probably You need to get rid of ../../
const nodeModules = fs.readdirSync('../../node_modules').filter((x) => (x !== '.bin'))

// webpack conf
{
   ...,
   externals: nodeModules,
   ...,
}

Looks like electron says it is a Debug build of node?

@saper in my case it was Webpack problem. But I'm still not sure about that Debug thing. I could look into nodegit source, but don't have time for that, since it imported succesfuly in typical (not Webpack) node environment.

It is not related to node-git and only a bit to node-gyp. What does process.config.target_defaults.default_configuration say from within node?

'Release'

As I said, this is Webpack import problem, which his also described on other pages.

I don't understand your solution. Probably there is some deep trouble there. Worst case you end up using two different node engines, I don't know...

Is there some new information?
My app works in dev mode, builds fine and then throws this error once I start up. I've tried electron-rebuild but that doesn't help.

We do not support running node-sass on Electron. You can compile your sass using traditional node engine.

For what it's worth I switched to dugite which builds and runs fine with electron.

I got this to work with Electron by attaching it to the window from the preload.js instead of requiring it from a rendering thread:
preload.js:
window.NodeGit = require("nodegit");

electron.js:
let mainWindow = new BrowserWindow({
    ..
    webPreferences: {
        nodeIntegration: false,
        preload: __dirname + '/preload.js',
    },
    ....
});

From the rest of the application I then access it from window.NodeGit.

Looking at the file in question in nodegit dist path, here is the block that appears to not see the Release build even when it is there.
try {
  rawApi = require("../build/Release/nodegit.node");
} catch (ex) {
  /* istanbul ignore next */
  if (ex.code !== "MODULE_NOT_FOUND") {
    throw ex;
  }

  rawApi = require("../build/Debug/nodegit.node");
}

Does anybody know why it can't see the Release build and falls back to the Debug release that isn't there?

Part of me wonders if that code block is even a good way of doing a debug vs. release check.  Really it should be controlled explicitly by an environment variable during build.

Has anyone managed to make this work from electron?
I do have an angular (CLI) app and I'm not able to run nodegit at all

After much debugging and trying different approaches, I ended up upgrading the version of Electron to the latest and taking advantage of the Electron Rebuild CLI.
In the package.json I use -f to force it even if the build exists, and -b is the flag for debug, which is the environment we are looking for above.
"rebuild": "electron-rebuild -f -b"

I also encountered this error message when using webpack to pack up (not during run-time though).
I figured out that commenting out the line importing debug version of nodegit.node helps webpack bundle successfully.
But I wonder what is the purpose of loading debug version of nodegit.node and if it is the right thing to ignore it in webpack.
Project Configuration
Here is part of my package.json:
  "dependencies": {
    "nodegit": "^0.26.5",
  },
  "devDependencies": {
    "electron": "^8.2.5",
    "node-loader": "^1.0.2",
    "webpack": "^4.46.0",
  },

Not sure if it matters, my node version is 12.20 and I installed nodegit with prebuilt binaries targeting windows x64.
You may see that I installed electron, webpack, and a package named node-loader. The usage of the last package will be explained later.
And here is my webpack configuration for the main process of electron:
const main = {
  entry: path.resolve(__dirname, './main.js'),
  output: {
    filename: 'main.js',
    path: distPath,
  },
  target: 'electron-main',
  node: {
    __dirname: false,
  },
  module: {
    rules: [
      {
        test: /\.node$/,
        loader: 'node-loader',
      },
    ],
  },
};

I require('nodegit'); in main.js to force webpack to bundle it to test if it can be bundled successfully for main process of electron.
Problem Description
Before the node-loader package is installed, webpack fails to pack release version of nodegit.node with the following error message:
WARNING in ./node_modules/nodegit/build/Release/nodegit.node 1:2
Module parse failed: Unexpected character '�' (1:2)
    You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
    (Source code omitted for this binary file)
     @ ./node_modules/nodegit/dist/nodegit.js 12:11-51
     @ ./main.js

Because line 12 of nodegit.js is actually catched, this exception further leads to another exception when trying to load debug version of nodegit.node:
ERROR in ./node_modules/nodegit/dist/nodegit.js
    Module not found: Error: Can't resolve '../build/Debug/nodegit.node' in 'C:\my\project\path\node_modules\nodegit\dist'
     @ ./node_modules/nodegit/dist/nodegit.js 19:11-49
     @ ./main.js

Here the node-loader comes to rescue.
Since webpack told me to search for loaders to solve the error for release version of nodegit.node, I installed node-loader.
It eliminates the first error message but the second one continues to show.
Since there is no debug version, currently what I could do is commenting out rawApi = require("../build/Debug/nodegit.node");.
After doing so, webpack could finally bundle nodegit and it finally works. (launched the bundled JS with electron)
Hope this help everyone trying to use nodegit together with electron and webpack.

@zedaizd Thank you so much! :tada:
Adding the following to my webpack config for main solved the issue:
import { IgnorePlugin } from "webpack";

// Inside the config object, add:

plugins: [
  new IgnorePlugin({ resourceRegExp: /build\/Debug\/nodegit.node$/i }),
],
externals: {
  nodegit: "commonjs nodegit",
},

authors: CrixuAMG SAPessi atrauzzi burningTyger dolanmiu klappy maxime1992 nikwen saper soanvig zedaizd

repo: nodegit/nodegit


text:
Will (ambient) loaders have access to synchronous import.meta.resolve?

This is a potential gotcha I have on my to-do list to ask about.
Loaders cannot block their own thread with import.meta.resolve calls, so that will somehow need to be addressed in the design.
A loader author may be perfectly happy to deal with this limitation.  However, if a loader attempts to import a (transitive) dependency that wants to call import.meta.resolve, that code will fail.
I'm not saying I have a magical solution to this problem, just that it probably should be called out in the design as a known limitation.

I guess at least at first, calls to import.meta.resolve from within a loaders context could just throw an error.

This is apparently not possible as it creates a viscous cycle.
Closing, but feel free to re-open if there's something to be done about that 🙂

repo: nodejs/loaders


text:
How will hot reloading be accomplished?

I see in the use cases documentation that it mentions hot reloading as a use case that you're hoping to support - is there a proposal for how that would be accomplished?
I am interested in it for the various node loaders that I help maintain at https://github.com/node-loader, such as an import maps loader, babel loader, http loader, postcss loader, etc. I've been using a user-land implementation of loader chaining to combine these, but am interested in also hot reloading them. I explored it a bit with adding query parameters to the module urls during resolve(), which succeeds in reinstantiating the module but does not delete or replace the older, existing module.
I also help maintain SystemJS, which has its own loader implementation, so I'm interested to see if the nodejs implementation has any similarities.

Not totally sure, but this article that explains how I implemented "proxyquire" like functionality for ESM may help. Mocking libraries such as these need to enable importing different module implementations on every import, which they do by using the fact that module cache key is a URL (with query parameters that can be manipulated). This is probably similar to what you'd want to do with hot reloaders?
https://dev.to/giltayar/mock-all-you-want-supporting-es-modules-in-the-testdouble-js-mocking-library-3gh1

Thanks for the response - the approach in the quibble article is similar to what I've experimented with that adds query strings to specifiers during the resolve hook. However, the problem with that approach is that, as far as I know, there's no current way of replacing a module in the nodejs registry - when you add or modify the query parameter during resolve, it just instantiates a new module rather than replacing the module at the file path. So Quibble has to expose a global module api since using the nodejs module loader isn't sufficient.
The use case I've been exploring is a project similar to nodemon or node-dev, except hot reloading the ES modules that changed rather than restarting the nodejs process. For that, I think that an api would be needed to tell the nodejs module loader that it should re-instantiate the module (including calling getSource and transformSource again for it, and then updating live bindings for the exports to those of the reinstantiated module rather than the original module). In other words, the internal ModuleMap would set a new Module object for a URL.
The other part of it would be exposing an api for individual modules to clean up any memory when they are deleted from the module map. The import.meta.hot proposal is similar to this, except I believe (not sure) that it's intended use case is to update live bindings for a module without replacing it in the ModuleMap, rather than for cleaning up memory.
I'm sure there are lots of use cases to consider and lots of varying approaches that could be proposed. And I'm not sure if a full replacement of the Module within the ModuleMap would be feasible (eg its unclear what it would do if the new module has different exports than the previous one). If the approach I suggested above is interesting to you, I'm happy to propose an API for it. And if not, I am interested to hear others thoughts on what would be a good api.

I’m currently working on a new Node.js server framework for the Small Web that is based on ESM loaders and also interested in this.
Just like everyone else, I’m currently  just overloading modules using a random query string fragment but this is, of course, a memory leak. While this is perhaps tolerable during development, I’d also like to explore hot reloading in production where updates of the app are made possible without entirely restarting the server.
I’m pretty sure you’re far more aware of the considerations around this than I am but if there’s any feedback or assistance I can provide, just let me know and I’d be happy to.

there is an upcoming PR that implements this functionality and has a test case that allows replacing the global specifiers resolution https://github.com/nodejs/node/pull/39240 ; the JS spec itself does not allow for replacement of values, only new values so indeed you must generate new module source texts and cannot remove references to old source texts.

there is an upcoming PR that implements this functionality

I read through https://github.com/nodejs/node/pull/39240, but it seems related more to communication between loaders and mocking, rather than with hot reloading? It's quite possible I'm missing something or that we're using different definitions of hot reloading.

the JS spec itself does not allow for replacement of values, only new values so indeed you must generate new module source texts and cannot remove references to old source texts.

I wasn't aware of this - could you share a link to the part of the spec that forbids this? I was under the impression that an API for manipulating the module registry was mostly left undefined rather than forbidden by current specs.

Just like everyone else, I’m currently just overloading modules using a random query string fragment but this is, of course, a memory leak. While this is perhaps tolerable during development, I’d also like to explore hot reloading in production where updates of the app are made possible without entirely restarting the server.

@aral this is also what I would look to solve for. I think that a module should be modifiable / replaceable without restarting the NodeJS process, since the memory leak associated with the query string approach is not ideal.
Do you have any ideas for an API that would allow for this? Here are some of the thoughts I had when thinking about possible solutions:
// One idea
import.meta.reload('./some-file.js').then(() => {
  console.log("All loader hooks, including getSource/transformSource were re-run. The resulting module replaced the old module entirely. Live bindings for dependent modules have been updated")
});

// Another idea
import.meta.delete('./some-file.js')
import('./some-file.js').then(() => {
  console.log("All loader hooks, including getSource/transformSource were re-run. The resulting module replaced the old module entirely. Live bindings for dependent modules have been updated")
})

since the query string approach results in memory leaks and also requires reloading all dependent modules (not just the changed module) so that they get the new version of the hot reloaded module.

@joeldenning maybe I'm missing something but doesn't the query string approach fail to reload dependent modules since they are cached?
My use case is to reload a local module (along with its dependencies) every time it or its dependencies are modified. While this works for direct changes to the module, I'm not sure how to handle modifications to one of the module's local dependencies (simply re-importing the module doesn't work since it's dependencies are cached, and I don't want to have to require all imports of the module to use query strings for cache busting).
Basically I have code that looks like this:
import module from "module";
import dependencyTree from "dependency-tree";
import chokidar from "chokidar";

let impl = new (await import("../impl")).Impl();
const deps = dependencyTree.toList({
  directory: ".",
  filename: module.createRequire(import.meta.url).resolve("../impl"),
  filter: (path) => !path.includes("node_modules"),
});
chokidar.watch(deps).on("change", async () => {
  try {
    impl = new (await import(`../impl.ts#${Math.random()}`)).Impl(); // use query string for cache busting
  } catch (e) {
    console.error("Failed to reload:", e);
  }
});

This works for direct edits to impl.ts but not for edits to one of impl.ts's dependencies.
Once chained loaders are implemented maybe I can use something like https://github.com/vinsonchuong/hot-esm to solve this

maybe I'm missing something but doesn't the query string approach fail to reload dependent modules since they are cached?

Yes, the dependent modules are not reinstantiated unless you add a query string to them, too.
This is part of why I do not consider the query string approach to be hot reloading - it's just adding more modules to the registry without reloading existing modules. As far as I know, there is no current method of achieving real hot reloading where a module and its dependencies are updated.

Hi @joeldenning,

I am interested in it for the various node loaders that I help maintain at https://github.com/node-loader, such as an import maps loader, babel loader, http loader, postcss loader, etc.

Thank you for organizing a set of use-cases as you have. I have not had a moment to review all of what you have collected, but I have another loader (loader339) that I have been meaning to transfer to the @nodejs organization after some improvements are made. It is intended to support the APM use case. We have sort of been using it to help guide the business use-cases (just made it public for your viewing pleasure, but might keep it public since it is already receiving stars in private).

How will hot reloading be accomplished?

You may need to make your own module graph. That is not really a case I have been working on, but thanks for bringing it up — surely someone will want to get that solved as well.

I implemented a full-featured hot reloading loader:
https://www.npmjs.com/package/dynohot
It supports live bindings like Webpack, and the full .accept(specifier)-style hot interface. Great for running an API server without nodemon or similar nuclear solutions.

authors: DerekNonGeneric aral bmeck giltayar hpx7 joeldenning laverdet