text:
Cannot install sharp on ARM64 Odroid Ubuntu

Hi ,
I've been trying to install sharp on Ubuntu 16.04 Xenial installed on an Odroid but am not able to get the dependency libvips installed on it. Which obviously is causing installation of sharp to fail.
I'm running node v7.9.x and npm is v4.2.0.
Here is the output when I try to install sharp "sudo  npm install sharp"

sharp@0.17.3 install /home/pankaj/node_modules/sharp
node-gyp rebuild

ERROR: Download of https://dl.bintray.com/lovell/sharp/libvips-8.4.2-linux-arm64.tar.gz failed: Response code 404 (Not Found)
gyp: Call to 'LDD_VERSION="ldd (Ubuntu GLIBC 2.23-0ubuntu3) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper." node -e "require('./binding').download_vips()"' returned exit status 1 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:308:16)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:194:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Linux 3.14.29
gyp ERR! command "/usr/bin/nodejs" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/pankaj/node_modules/sharp
gyp ERR! node -v v7.9.0
gyp ERR! node-gyp -v v3.5.0
gyp ERR! not ok
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"arm64"})
npm WARN enoent ENOENT: no such file or directory, open '/home/pankaj/package.json'
npm WARN pankaj No description
npm WARN pankaj No repository field.
npm WARN pankaj No README data
npm WARN pankaj No license field.
npm ERR! Linux 3.14.29
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "sharp"
npm ERR! node v7.9.0
npm ERR! npm  v4.2.0
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! sharp@0.17.3 install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the sharp@0.17.3 install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the sharp package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs sharp
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls sharp
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /home/pankaj/.npm/_logs/2017-04-17T08_03_38_451Z-debug.log
I followed the steps on sharp.dimens.io:
Added repository: sudo add-apt-repository -y ppa:lovell/trusty-backport-vips
gpg: keyring /tmp/tmp33esnaad/secring.gpg' created gpg: keyring /tmp/tmp33esnaad/pubring.gpg' created
gpg: requesting key 21794330 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmp33esnaad/trustdb.gpg: trustdb created
gpg: key 21794330: public key "Launchpad PPA for Lovell Fuller" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
then did - sudo apt-get update
Here I get the following error in output:
Hit:1 http://deb.odroid.in/c2 xenial InRelease
Ign:2 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial InRelease
Ign:3 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial Release
Ign:4 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main arm64 Packages
Ign:5 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main armhf Packages
Ign:6 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main all Packages
Hit:7 https://deb.nodesource.com/node_7.x xenial InRelease
Ign:8 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en_US
Ign:9 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en
Ign:10 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-ru
Ign:4 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main arm64 Packages
Ign:5 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main armhf Packages
Ign:6 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main all Packages
Ign:8 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en_US
Ign:9 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en
Ign:10 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-ru
Ign:4 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main arm64 Packages
Ign:5 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main armhf Packages
Ign:6 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main all Packages
Ign:8 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en_US
Ign:9 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en
Ign:10 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-ru
Ign:4 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main arm64 Packages
Ign:5 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main armhf Packages
Ign:6 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main all Packages
Ign:8 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en_US
Ign:9 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en
Ign:10 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-ru
Ign:4 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main arm64 Packages
Ign:5 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main armhf Packages
Ign:6 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main all Packages
Ign:8 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en_US
Ign:9 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en
Ign:10 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-ru
Err:4 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main arm64 Packages
404  Not Found
Err:5 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main armhf Packages
404  Not Found
Ign:6 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main all Packages
Ign:8 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en_US
Ign:9 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-en
Ign:10 http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial/main Translation-ru
Hit:11 http://ports.ubuntu.com/ubuntu-ports xenial InRelease
Hit:12 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease
Hit:13 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease
Hit:14 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease
Hit:15 http://oph.mdrjr.net/odrobian xenial InRelease
Hit:16 http://ports.ubuntu.com/ubuntu-ports xenial-proposed InRelease
Reading package lists... Done
W: The repository 'http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://oph.mdrjr.net/odrobian/dists/xenial/InRelease: Signature by key F3230DD84F39A606727D3D5D1B881FD777439C3B uses weak digest algorithm (SHA1)
E: Failed to fetch http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu/dists/xenial/main/binary-arm64/Packages  404  Not Found
E: Failed to fetch http://ppa.launchpad.net/lovell/trusty-backport-vips/ubuntu/dists/xenial/main/binary-armhf/Packages  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
My sources contains: /etc/apt/sources.list
deb http://ports.ubuntu.com/ubuntu-ports/ xenial main universe restricted multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial main universe restricted multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main universe restricted multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main universe restricted multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-backports main restricted universe multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-backports main restricted universe multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted universe multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted universe multiverse
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-proposed main restricted universe multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-proposed main restricted universe multiverse
uname -a
Linux odroid64 3.14.29 #25 SMP PREEMPT Fri Apr 22 16:24:15 EDT 2016 aarch64 aarch64 aarch64 GNU/Linux
I've spent hours searching and trying different things but none have worked. Please advice how to fix this. Thanks in advance!

Hello, it looks like the value for node's process.arch attribute on 64-bit ARM chips has now settled on (the undocumented) "arm64" rather than "arch64" as currently checked (which I think should be "aarch64", so that check is probably wrong anyway).
Are you able to make the following change from arch64 to arm64 in binding.js#L45-L46 and see if that works?
-  if (arch === 'arm' || arch === 'armhf' || arch === 'arch64') {
-    const armVersion = (arch === 'arch64') ? '8' : process.env.npm_config_armv || process.config.variables.arm_version || '6';
+  if (arch === 'arm' || arch === 'armhf' || arch === 'arm64') {
+    const armVersion = (arch === 'arm64') ? '8' : process.env.npm_config_armv ||  process.config.variables.arm_version || '6';

I searched on my installation and did not find the file "binding.js" anywhere on it. Since sharp is not yet installed I do not have "node_modules/sharp" created yet.
Are you suggesting I download sharp code into my "node_modules" folder and then modify this file and try again ?

I cloned the repo into my "node_modules" directory and modified the file "binding.js" with the changes you'd suggested. Then I tried to install sharp again and here is what I get :
pankaj@odroid64:~$ sudo npm install sharp
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"arm64"})
npm WARN enoent ENOENT: no such file or directory, open '/home/pankaj/package.json'
npm ERR! Linux 3.14.29
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "sharp"
npm ERR! node v7.9.0
npm ERR! npm  v4.2.0
npm ERR! path /home/pankaj/node_modules/sharp
npm ERR! code EISGIT
npm ERR! git /home/pankaj/node_modules/sharp: Appears to be a git repo or submodule.
npm ERR! git     /home/pankaj/node_modules/sharp
npm ERR! git Refusing to remove it. Update manually,
npm ERR! git or move it out of the way first.
npm ERR! Please include the following file with any support request:
npm ERR!     /home/pankaj/.npm/_logs/2017-04-17T18_40_04_505Z-debug.log

I've made the change on a temporary branch. Please can you test this "version" in your package.json by placing a dependency on it using:
  "sharp": "lovell/sharp#arm64"

Thank you !! The install successfully completed. I'll test it out in another few.
How do I go about for future such devices ?
pankaj@odroid64:~$ sudo npm install sharp

sharp@0.17.3 install /home/pankaj/node_modules/sharp
node-gyp rebuild

make: Entering directory '/home/pankaj/node_modules/sharp/build'
TOUCH Release/obj.target/libvips-cpp.stamp
CXX(target) Release/obj.target/sharp/src/common.o
CXX(target) Release/obj.target/sharp/src/metadata.o
CXX(target) Release/obj.target/sharp/src/operations.o
CXX(target) Release/obj.target/sharp/src/pipeline.o
CXX(target) Release/obj.target/sharp/src/sharp.o
CXX(target) Release/obj.target/sharp/src/utilities.o
SOLINK_MODULE(target) Release/obj.target/sharp.node
COPY Release/sharp.node
TOUCH Release/obj.target/win_copy_dlls.stamp
make: Leaving directory '/home/pankaj/node_modules/sharp/build'
privatebox@1.0.0 /home/pankaj
└─┬ sharp@0.17.3  (git://github.com/lovell/sharp.git#fc1a3562c3d6b3f9cab2554bbed9c6dfacdbd03f)
├─┬ caw@2.0.0
│ ├─┬ get-proxy@1.1.0
│ │ └─┬ rc@1.2.1
│ │   ├── deep-extend@0.4.1
│ │   ├── minimist@1.2.0
│ │   └── strip-json-comments@2.0.1
│ └── tunnel-agent@0.4.3
├─┬ color@1.0.3
│ ├─┬ color-convert@1.9.0
│ │ └── color-name@1.1.2
│ └─┬ color-string@1.5.2
│   └─┬ simple-swizzle@0.2.2
│     └── is-arrayish@0.3.1
└─┬ got@6.7.1
├─┬ create-error-class@3.0.2
│ └── capture-stack-trace@1.0.0
├── duplexer3@0.1.4
├── get-stream@3.0.0
├── is-redirect@1.0.0
├── is-retry-allowed@1.1.0
├── is-stream@1.1.0
├── lowercase-keys@1.0.0
├── timed-out@4.0.1
├── unzip-response@2.0.1
└─┬ url-parse-lax@1.0.0
└── prepend-http@1.0.4
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"arm64"})

I've completed setting up the environment and during testing of sharp/libvips I get a warning that is supposed to be harmless and after that the node application crashes. Here is the output:
------ error from my application -------
INFO: Entered 'POST/downloadImage'
vips warning: VipsJpeg: error reading resolution
*** Error in `node': free(): invalid size: 0x0000007f60001230 ***
Aborted

-------  end of error message, application crashed  --------
Here is the line of code just prior to the log message "INFO: Entered POST/downloadImage":
      sharp(fileToDownload)
         .resize(maxSize, maxSize)
         .toBuffer(function(err, outputBuffer) {
           if (err) {
              logger.log(logger.levelInfo, "sharp.read -error occurred : " + err);
           }
           else{

              res.writeHead(200, {'Content-Type': 'image/jpeg' });
              res.end( outputBuffer );
           }
      });

Sharp version that got installed after the workaround you put in place  - 0.17.3
libvips42/xenial,now 8.2.2-1 arm64 [installed]
The same code worked when node was running on an X86 or even another ARM board (Allwinner). Please advice how to resolve this?

I came across issue # 567
It has similar error message:
*** Error in `node': free(): invalid size: 0x0000007f60001230 ***
Aborted

Your advice there was to install libvips via npm and not apt-get, so I uninstalled libvips-dev and libgsf-1-dev  and then installed libvips using npm successfully.
However now that same point in code aborts with the following error message:
INFO: Entered 'POST/downloadImage'
node: malloc.c:2928: __libc_malloc: Assertion `!victim || chunk_is_mmapped (mem2chunk (victim)) || ar_ptr == arena_for_chunk (mem2chunk (victim))' failed.
Aborted

Please advice.

Glad to hear the "arm64" fix allowed you to install - I'll add this change to the v0.18.0 release.

malloc.c:2928: __libc_malloc: Assertion `!victim || chunk_is_mmapped (mem2chunk (victim)) || ar_ptr == arena_for_chunk (mem2chunk (victim))' failed.

This looks like a memory allocation failure, or possibly memory corruption. Are you able to share the smallest possible code sample that fails in this way?

sure. Here is the code that gets called and crashes the app:
ImageSizeOriginal is set to very high value - 999999999
app.get('/downloadImage', function(req, res) {

    logger.log(logger.levelInfo,"Entered '/downloadImage' for GET method ");

    var maxSize = req.query.maxSize;  // used has height/width of image
    var fileToDownload = mediaParentFolder + req.query.filename;

    if(maxSize < ImageSizeOriginal){
      logger.log(logger.levelDebug, "will send reduced size of " + maxSize + " from file at location : " + fileToDownload);
      sharp(fileToDownload)
         .resize(maxSize, maxSize)
         .toBuffer(function(err, outputBuffer) {
           if (err) {
              logger.log(logger.levelInfo, "sharp.read -error occurred : " + err);
           }
           else{
              res.writeHead(200, {'Content-Type': 'image/jpeg' });
              res.end( outputBuffer );
           }
      });
    }
    else {  // This will even catch maxsize as 'undefined'
      logger.log(logger.levelDebug, "will send Original size from file at location : " + fileToDownload);
      fs.readFile(fileToDownload, function read(err, data) {
         if (err) {
               logger.log(logger.levelInfo, "error occurred : " + err);
               logger.log(logger.levelInfo, "data holds : '" + data + "'");
         }
         res.writeHead(200, {'Content-Type': 'image/jpeg' });
         res.end(data);
      });
    }
});

Are you able to reduce this example further, without all the HTTP stuff? Ideally this will be self-contained code that can be run at the command line yet still produce the error.

Yes. I've done that now and here it is:
var sharp = require("sharp");

      sharp("DDD89B0C-FB3C-4F32-A4FD-186CE6C6D290.jpg")
         .resize(150, 150)
         .toBuffer(function(err, outputBuffer) {

           if (err) {
              logger.log(logger.levelInfo, "sharp.read -error occurred : " + err);
           }
           else{

              console.log("Sharp optimization successful");
              console.log("sending bytes : " + outputBuffer.length);
           }
      });

Here is the complete set of logs and error it generates:
pankaj@odroid64:~/test$ node testsharp.js
vips warning: VipsJpeg: error reading resolution
*** Error in `node': free(): invalid size: 0x0000007f98000a60 ***
Aborted

Great, thank you, that's a very straightforward example so hopefully I'll be able to reproduce/confirm this. (But probably won't get around to this for a few days though.)

Sure, no worries.
I appreciate your time and effort in resolving this.
I’m going to continue trying a few things to see if I can provide more information on this. Also, I’m working on making it accessible via SSH so you can access this installation from your end (even if you break it, it’ll be OK). Once it’s available over internet, I’ll let you know.
If there is anything I can do from my end then let me know.
On Apr 18, 2017, at 5:25 PM, Lovell Fuller <notifications@github.commailto:notifications@github.com> wrote:
Great, thank you, that's a very straightforward example so hopefully I'll be able to reproduce/confirm this. (But probably won't get around to this for a few days though.)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/lovell/sharp/issues/772#issuecomment-294987352, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ANIxL3CF8argpjsF2E2Yw1CEsox2nY_Eks5rxSpkgaJpZM4M-9-S.

I tested the same code on another ARM platform and it worked just fine.
uname -a
Linux FriendlyARM 3.4.39-h3 #26 SMP PREEMPT Sat Feb 4 16:59:41 CST 2017 armv7l GNU/Linux
It is only with this S905 ARM that we see the crash.

I added a temporary commit https://github.com/lovell/sharp/commit/88b9f4ba87d72739682de1178337a570fb633f49 on the arm64 branch that I think might help, please can you try installing from it again.

Sure. I’ll try it right now and revert shortly.
On Apr 19, 2017, at 2:30 AM, Lovell Fuller <notifications@github.commailto:notifications@github.com> wrote:
I added a temporary commit 88b9f4bhttps://github.com/lovell/sharp/commit/88b9f4ba87d72739682de1178337a570fb633f49 on the arm64 branch that I think might help, please can you try installing from it again.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/lovell/sharp/issues/772#issuecomment-295118634, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ANIxL6qWJLZMbwtAtMWnH2i-CsLYePJHks5rxaoNgaJpZM4M-9-S.

Awesome. That fixed it.
It works beautifully now. Thank you!! I appreciate your prompt help and fix on this.

Great, thanks for confirming. 64-bit ARM (ARMv8) support will be fixed properly in the v0.18.0 release. (I'm also expanding the ARM packaging tests to help prevent further regression.)

Thanks!
So for future installs I can simply install sharp without having to do following in dependencies:
"sharp" : "lovell/sharp#arm64"

Correct?

@ajiratech2 Correct. This issue is tagged as v0.18.0 so you'll get a notification when it is available via npm.

v0.18.0 now available, thanks for reporting this.

Thank you!
I appreciate your help in getting this resolved and adding it to mainline.
On May 30, 2017, at 3:41 AM, Lovell Fuller <notifications@github.commailto:notifications@github.com> wrote:
v0.18.0 now available, thanks for reporting this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/lovell/sharp/issues/772#issuecomment-304800072, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ANIxL79iRpbU-jwE6Moprt0YZ38YG1FDks5r-8hEgaJpZM4M-9-S.

Sorry for commenting from a closed issue.  But is there any precompiled ARMHF (32 bit) binaries available?
We recently switched to sharp (from external imagemagick / graphicmagik), but our Raspberry Pi snapcraft builds will only pull in the ARM64 binaries - resulting in wrong ELF class broken builds 😞   Thanks.

@Sing-Li Pre-compiled binaries for libvips and its dependencies are provided for ARMv7-A.
https://github.com/lovell/sharp-libvips/blob/master/linux-armv7/Dockerfile#L21
Is npm install being run on the same device as used at runtime?
The sharp.node binary will still need to be compiled, either on the device it will be used at runtime or cross-compiled to match.
If the problem persists, please open a new issue with the result of running npm install sharp on the device used at runtime.

authors: Sing-Li ajiratech2 lovell


text:
sms test file done

please revew accpted pr

ok

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
Cap v4

Hey, what a great idea this is! Will you update this to Capacitor v4? Cant wait to try it out and see all the new iOS 16 possibilities

Let me know if something doesn't work or if you have any additional feature requests

Hey Timo, sure thing! Upon building with NextJS I am getting an error. I will try to describe it:
Describe the bug
Our project's web assets (webpack) build is failing if the following import { SiriShortcuts } from 'capacitor-plugin-siri-shorts is defined somewhere in the code.
The error is the following:
triggerUncaughtException(err, true /* fromPromise */);
            ^
ReferenceError: capacitorExports is not defined
    at Object.<anonymous> (/node_modules/capacitor-plugin-siri-shorts/dist/plugin.js:12:8)```

I am using the NextJS's next build command to run build.

**To Reproduce**
Steps to reproduce the behavior:

1. Build a project with (NextJS and Capacitor)[https://github.com/mlynch/nextjs-tailwind-ionic-capacitor-starter]
2. Follow installation steps of `capacitor-plugin-siri-shorts`
3. Run yarn build

Possible solution
Change   "main": "dist/plugin.js" to  "main": "dist/plugin.cjs.js", in the package.json
This morning I was looking for a solution for us. I have to say that I have no experience with Swift or capacitor plugins. But I managed to get next build working  if I made this change.
I read about another (plugin)[https://github.com/capacitor-community/bluetooth-le/issues/12] that had this issue too.
Let me know if this is the correct solution! I haven't managed to get the plugin to work, but I'll continue to try this Sunday.
Thanks so much for replying @lovetodream

Even through I cannot reproduce your issue, I've found out that the boilerplate for the package.json for capacitor plugins has changed to the cjs-way you're suggesting https://github.com/ionic-team/create-capacitor-plugin/blob/main/assets/plugin-template/package.json.mustache#L5
I'll release an update including the changes as a breaking change with v3.0.0 of the plugin, because I do not know how this affects existing projects.

Even through I cannot reproduce your issue, I've found out that the boilerplate for the package.json for capacitor plugins has changed to the cjs-way you're suggesting https://github.com/ionic-team/create-capacitor-plugin/blob/main/assets/plugin-template/package.json.mustache#L5 I'll release an update including the changes as a breaking change with v3.0.0 of the plugin, because I do not know how this affects existing projects.

Cheers man! Will let you know after the update how things go.

source: gharchive/issue

authors: RobSchilderr lovetodream

license: MIT


text:
[bazel] can't build //sw/device/silicon_creator/rom/e2e:sigverify_mod_exp_rma_otbn_fpga_cw310_rom --define bitstream=vivado

./bazelisk.sh test //sw/device/silicon_creator/rom/e2e:sigverify_mod_exp_rma_otbn_fpga_cw310_rom --define bitstream=vivado
INFO: Invocation ID: 0251e26b-5b41-4983-b723-346e6efdb0f6
INFO: Build option --define has changed, discarding analysis cache.
ERROR: /usr/local/google/home/drewmacrae/review/hw/bitstream/BUILD:90:10: in filegroup rule //hw/bitstream:otp_mmi: non-test target '//hw/bitstream:otp_mmi' depends on testonly target '//hw/bitstream/vivado:otp_mmi' and doesn't have testonly attribute set

Originally posted by @drewmacrae in https://github.com/lowRISC/opentitan/issues/16916#issuecomment-1362077540
"filegroup rule //hw/bitstream:otp_mmi: non-test target '//hw/bitstream:otp_mmi' depends on testonly target '//hw/bitstream/vivado:otp_mmi' and doesn't have testonly attribute set"

There's an open question why we didn't hit this before. Presumably it only applies to tests that exercise the OTP so only SW work demands them, and vivado only gets invoked when people are updating hardware so it's only the paranoid folks that end up seeing it?

repo: lowRISC/opentitan


text:
[dv/xprop] Address remaining xprop warnings

Description
There are a few remaining xprop warnings for constructs that block instrumentation. They happen in hw/vendor/pulp_riscv_dbg/src/dm_csrs.sv and hw/vendor/pulp_riscv_dbg/src/dm_mem.sv, and have the following warnings:
../src/pulp-platform_riscv-dbg_0.1_0/pulp_riscv_dbg/src/dm_csrs.sv:288 NO "a range in a case-inside" (289)
../src/pulp-platform_riscv-dbg_0.1_0/pulp_riscv_dbg/src/dm_csrs.sv:362 NO "a range in a case-inside" (363)
../src/pulp-platform_riscv-dbg_0.1_0/pulp_riscv_dbg/src/dm_mem.sv:254 NO "a range in a case-inside" (257)
../src/pulp-platform_riscv-dbg_0.1_0/pulp_riscv_dbg/src/dm_mem.sv:298 NO "a range in a case-inside" (257)
../src/pulp-platform_riscv-dbg_0.1_0/pulp_riscv_dbg/src/dm_mem.sv:386 NO "a range in a case-inside" (387)
These are due to VCS not handling a range as a label for a "case inside" statement. Since this code is used for hardware implementation it would be ideal to refactor it to enable xprop instrumentation. This may be done by expanding the ranges into individual labels. It may require macro trickery or even template expansion (as in using .tpl files).
The least desirable option is to explicitly disable xprop instrumentation of this, which would mean adding the (* xprop_off *) attribute to the enclosing "always" processes.

Thanks for flagging this, @matutem. In addition to the two code change options that you mentioned, I see a third option where these cases are handled with an if (addr >= first_addr && addr <= last_addr) block under the default label of a unique case (non-inside) statement. For example:
-      unique case (dm_csr_addr) inside
+      unique case (dm_csr_addr)
-        [(dm::Data0):DataEnd]: begin
-          /* handle accesses to data addresses ... */
-        end
         /* handle accesses to other addresses ... */
-        default:;
+        default: begin
+          if (dm_csr_addr >= dm::Data0 && dm_csr_addr <= DataEnd) begin
+            /* handle accesses to data addresses ... */
+          end
+        end
       end

VCS should be able to instrument this code for xprop, right?
This would have the advantage that no preprocessing (or even external tools) are required. One disadvantage would be that if there are multiple such ranges, the circuit generated is not exactly identical to the one currently described: multiple ranges would be handled with an if ... else if ... chain in the default statement, which defines a priority, whereas unique case does not define a priority. But I think that isn't a problem for this use case. WDYT?

I tried the code changes I suggested, and they indeed remove the remaining warnings in xprop.log. I've created PR #17310 to get it done.

Triaged for rv_dm. Assigning to M2.5 with https://github.com/lowRISC/opentitan/labels/Priority%3AP2.

@andreaskurth is this fixed by https://github.com/lowRISC/opentitan/pull/17733?

@andreaskurth is this fixed by #17733?

Yes, this is resolved in #17733 through xprop_off attributes since, as described in https://github.com/lowRISC/opentitan/pull/17310, current EDA tool versions support neither unique case inside nor unique if constructs for X propagation.

authors: GregAC andreaskurth matutem


text:
[otbn] Can the last instruction of a loop be a jump?

When the last instruction in a loop body is a JAL, the address that has to be pushed to the call stack is different in the last cycle of the loop. Do we want to support this, or do we want to mandate a nop following such an instruction?

I think we should specify that the address pushed into the call stack is pc + 4 regardless of the current loop iteration. Writers of software might well want to avoid putting a JAL at the end of a loop body, but I don't think it makes sense to worry about that in hardware.
As a separate note, I think that @imphil was wondering about whether to prohibit jump-and-link (and branches?) in loops entirely. If we do that, I guess this point is moot.

"No" :-D We now raise an error if you do that.

"No" :-D We now raise an error if you do that.

authors: felixmiller rswarbrick


text:
[aes] shadow reg storage and update error status question

Hi Pirmin,
I would like to confirm the AES status register behavior as below:
1). Write two shadow registers with different value to trigger an update error
2). Check status register -> ALERT_RECOV_CTRL_UPDATE_ERR is set to 1.
3). Probe design to make a storage error in shadow reg's committed val
4). Check status register -> ALERT_FATAL_FAULT is set to 1, but ALERT_RECOV_CTRL_UPDATE_ERR is reset to 0.
Is that intended to reset ALERT_RECOV_CTRL_UPDATE_ERR field back to 0?
I can align on DV side, but just want to double check and see if you want to make this field persistent.

Thanks,
Cindy

should we try to make this behavior standard? ie if an update error
happens, we don't clear it unless software does it?
On Thu, Sep 30, 2021 at 2:04 PM cindychip @.***> wrote:

Hi Pirmin,
I would like to confirm the AES status register behavior as below:
1). Write two shadow registers with different value to trigger an update
error
2). Check status register -> ALERT_RECOV_CTRL_UPDATE_ERR is set to 1.
3). Probe design to make a storage error in shadow reg's committed val
4). Check status register -> ALERT_FATAL_FAULT is set to 1, but
ALERT_RECOV_CTRL_UPDATE_ERR is reset to 0.
Is that intended to reset ALERT_RECOV_CTRL_UPDATE_ERR field back to 0?
I can align on DV side, but just want to double check and see if you want
to make this field persistent.
[image: image]
https://user-images.githubusercontent.com/11466553/135529951-3ad16cbb-3cdb-497b-859e-d70160338b1f.png
Thanks,
Cindy
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/lowRISC/opentitan/issues/8460, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAH2RSWWY3SBEZAZ3THJU3DUETGHNANCNFSM5FDHBQGQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

It would be ideal to standardize it (if not too much trouble on design side). Also aes status fields are all "RO" instead of "RW1C", that means SW cannot clear it.

Hey @cindychip ,
yes I can confirm that this is intended behavior. The idea is that whenever a fatal alert is occurring (no matter whether it's a storage error in the shadow register or some other fault inside the design), all status bits are cleared to 0.
The motivation for this behavior is that we don't want to give away potentially useful information to an adversary to refine an attack. For example, if a fault is injected while the cipher core is idle and the module locks up with the idle bit high, the adversary knows that the timing was wrong and needs to change that.
As for the clearing: The update error is cleared whenever the shadow register is written.

hmmmm Pirmin since you have multiple fault error possibilities inside AES,
does that mean there's no way to dis-ambiguate which fault caused the error?
I think this is something we should discuss in upcoming silicon / security
meetings. I'm still not convinced that we need to "hide" software error
readbacks.
I still maintain that if an attacker has the ability to control arbitrary
software reads to see how the system was attacked, far greater damage can
already be made.
If however we do decide that errors need to be hidden, we need to apply
this concept to the entire design, because right now blocks like kmac /
otp_ctrl / flash / keymgr expose in quite a fine granularity the actual
error.
On Fri, Oct 1, 2021 at 5:44 AM Pirmin Vogel @.***>
wrote:

Hey @cindychip https://github.com/cindychip ,
yes I can confirm that this is intended behavior. The idea is that
whenever a fatal alert is occurring (no matter whether it's a storage error
in the shadow register or some other fault inside the design), all status
bits are cleared to 0.
The motivation for this behavior is that we don't want to give away
potentially useful information to an adversary to refine an attack. For
example, if a fault is injected while the cipher core is idle and the
module locks up with the idle bit high, the adversary knows that the timing
was wrong and needs to change that.
As for the clearing: The update error is cleared whenever the shadow
register is written.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/lowRISC/opentitan/issues/8460#issuecomment-932194010,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAH2RSVWPEZJAF2SI6H7PDTUEWUJFANCNFSM5FDHBQGQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

hmmmm Pirmin since you have multiple fault error possibilities inside AES, does that mean there's no way to dis-ambiguate which fault caused the error?

Yes, that is correct. But in AES, all error sources except one indicate that an FI attack is ongoing. They are all fatal. The only one error/alert that can be triggered by software is the shadow register update error.

I think this is something we should discuss in upcoming silicon / security meetings. I'm still not convinced that we need to "hide" software error readbacks.
I still maintain that if an attacker has the ability to control arbitrary software reads to see how the system was attacked, far greater damage can already be made.

I tend to agree with you but to be honest, I don't have the system-level picture required to really judge this. In my view, even if the attacker has control over software it doesn't mean that he has access to sideload keys. But through FI attacks on AES,  he could try to make the module output intermediate state and then use that to guess the sideload key.

If however we do decide that errors need to be hidden, we need to apply this concept to the entire design, because right now blocks like kmac / otp_ctrl / flash / keymgr expose in quite a fine granularity the actual error.

I think every single one of these blocks are far more complicated in terms of software interaction than AES (doesn't even have interrupts). So, I would say having fine granularity is very useful for software development/bring up etc. But maybe some of these registers should be gated by LC?

Added this issue for further discussion in the Security/Software WG meetings.

As there was no time in the Security WG meeting:
KMAC behaves similar (does not clear the status, but also does not tell what happened). I favor this behavior, as I see no point to tell an attacker what was going on. However this is in contrast with the "Implementation Guidance"

If a particular alert sender has multiple error sources within the IP, the IP shall expose an associated error register to indicate the cause.

If this is crucial for debugging, we should disable it in the prod-phase.
Or at least for faults that cannot be triggered by software (e.g. invalid sparse encoding of an FSM).
Anyway it's worth a discussion.

yeah i think there are opposing views on the project on this point right now.
I tend to fall into the camp that if we are worried about an attacker being able to read a status, then it implies that software is already compromised to some extent (since an attacker is able to control arbitrary reads).  If we go with that perspective, we are already in a world of trouble.
So for this version, I tend to favor exposing debugging information (with some software ability to lock it out).  Since most of our debugging will actually be done in PROD state, and if PROD state exposes nothing, we effectively have no avenues of information.

Hey @cindychip ,
@rasmus-madsen and I noted that AES currently has some CSR tests failing. Investigating the failures it seems that the environment expects the recoverable alert bit (shadow register update error) to remain set until the corresponding status bit is read.
However, if another write to the shadow register happens, this actually resolves the condition for the recoverable alert. So in our view, it makes less sense to keep the status bit high. What is your view on this? Do you think it would be possible to adapt the tests accordingly?

@vogelpi @cindychip
we need to resolve this fairly quickly as it is holding back PRs due to CI failures
I am inclined to just exclude the registers

I agree with @tjaychen. It is possible for software bugs to trigger recoverable errors, inclusive when operations are interrupted due to unexpected context switching.  An attacker with access to the software interface will presumably know how to trigger this error by looking at the RTL, so I think hiding this information is has a lower security value in our implementation.
I prefer to prioritize debuggability in this case considering that the crypto library is part of the TCB, and other debug interfaces are disabled in PROD mode.

We have been discussing this issue in the Security meeting and concluded the following:

In the case of fatal alerts, we prefer debug-ability over hiding information. This holds not just for software that can cause fatal alert conditions (in many modules such as AES, KMAC this isn't even possible) but also for physical chip-level effects that may trigger some FI countermeasures, e.g., power distribution issues. Not clearing the status register in this case can provide useful information for debugging the chip. For this reason, I will change the AES implementation to not clear the status register upon a fatal fault. I can make the behavior parametrizable to change it back easily if it's considered useful at a later point.
For recoverable alerts, we have concluded that they should no longer be visible in the status register once the condition triggering the alert has been resolved. In the case of AES, this means if a new message is started by writing the control register, we should clear the recoverable alert bit in the status register. We will have to adjust DV accordingly.

Thank you Pirmin, that sounds great. I will align the DV common test accordingly for 2).
@eunchan, could you see if we want to align Kmac for 1). Currently in hmac storage fatal alert, it will clear the update recoverable alert bit to 0.

Thanks @cindychip that would be appreciated if you could take care of the DV adjustment. If you have any questions, don't hesitate.

My pleasure. @vogelpi could you tag me on the PR that reflects the change on item (1) once it is ready?

Thanks Pirmin for the PR, one last question - for item 2), do we plan to standardize it across the IPs or just AES only?

Hey @cindychip ,
sorry for keeping you waiting. I've added my responses inline below.

One last question - for item 2),
a). do we plan to standardize it across the IPs or just AES only?

Yes, I think eventually we should standardize on that behavior. However, I don't consider it super important for ES. People are probably very busy with other, more critical stuff at the moment. So, I recommend to just do it for AES right now.

b). the condition to clear update error status is register based? For example, if I have update error on ctrl_shadowed register, then I write twice with the same value to ctrl_aux_shadowed. This won't clear the shadow reg update error, am I understand it correctly?

This is not fully correct. To clear the update error status bit, the AES operation needs to be restarted by re-writing the main control register (ctrl_shadowed) twice. This is what the spec says. Writing ctrl_aux_shadowed twice doesn't clear the status bit even if the alert was triggered by that register.

Hi @cindychip is this still an issue? Who should it be assigned to?

When moving some shadow register bits around for #15641 I came again across this issue. I went through it again. The corresponding tests are passing, and the behavior of DV and RTL is documented (with links to this issue and lowRISC/OpenTitan#10617). I am now closing this issue.
Please feel free to re-open if you don't agree.

authors: cindychip m-temp moidx rasmus-madsen tjaychen vogelpi


text:
[veriblelint] Update to v0.0-3622-g07b310a3

The version we have been using was quite outdated and did not catch some legacy always @(*) statements that it should have warned about.
This updates to v0.0-3622-g07b310a3, and updates the installation instructions.
We may want to consider holding off merging, though, if this causes any infrastructure issues.

Need to update this too: https://github.com/lowRISC/opentitan/blob/e844018f2c29ee58e95340e264f519dbc8271c80/ci/install-package-dependencies.sh#L99

Need to update this too (no more Ubuntu-specific builds):
https://github.com/lowRISC/opentitan/blob/e844018f2c29ee58e95340e264f519dbc8271c80/ci/install-package-dependencies.sh#L99

Good catch - fixed!

It appears that Private CI will fail until we merge a corresponding fix over there: https://github.com/lowRISC/opentitan-private-ci/pull/35
This is because the URL has changed for newer releases, and Private CI relies on scripts from the main repo.

Reopening as I think this got closed accidentally. (If you disagree, please close again.)

Merging to keep public CI in sync with private CI. Bypassing failed checks as this addresses a CI problem on master.

authors: a-will andreaskurth msfschaffner


text:
Service creation failed error handling

It was observed that when a service is created with InactiveTimeout more than Allowed Timeout Value(24*3600) then loxilb returns error but kube-loxilb shows success.

kube-loxilb used to check http status code, which is usually 200 (OK) . The return code is passed as a "Success" or "Error" string in the body.  Parsing code as expected has been implemented and verified.

authors: TrekkieCoder nik-netlox

repo: loxilb-io/kube-loxilb


text:
Underscores in number representations are dropped

Before formatting:
defmodule NumberExample do
  def large_number do
    10_000
  end
end

After formatting
defmodule NumberExample do
  def large_number do
    10000
  end
end

Desired:

[ ] Do not drop underscores from numbers
[ ] Add underscores to large numbers where not present (?)

:+1:

authors: jfornoff lpil

repo: lpil/exfmt

license_source: bigquery


text:
🛑 Neo Viewer is down

In b84da2c, Neo Viewer (http://neoclinica.ddns.net) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Neo Viewer is back up in 6a69910 after 55 minutes.

repo: lprone/upptime


text:
Always N locked submissions in assessment overview

Describe the bug

After finishing the example submission. Immediately 5 locked students submissions are shown. Furthermore, when assessing the submissions by clicking on "continue assessment". The locked submssion is indeed asessed, but another submission got locked so that there are still 5 locked submissions (and 1 assessed submission now for example).
Expected behavior

Now submissions should be locked automatically.
Screenshots

Jun 1, 2020 11:28 AM here locked:

Jun 1, 2020 11:28 AM now assessed:

May 25, 2020 now locked, which was the submission for "start new assessment" earlier.
UPDATE:
After re-visiting the page, even more submission got locked:

Environment

 - OS: Windows
 - Browser Chrome (Version 83.0.4103.61)
 - Version [e.g. 22]

Additional context

Does this happen only for text exercises? (we just fixed a bug here that could explain your problem, which is not yet released) or also for other exercise types?

It seemed to appear on some Team exercises. I did my assessments and cleared all my locks from other submissions since I was aware of that issue. After one day I saw that some were locked by me which doesn't make any sense since I didn't open any submissions after this.
Hovever I'm afraid I don't have any further data but wanted to let you know that it was not only about this specific exercise.

please double check if this bug still occurs after the release 4.1.3 and report back. Thanks!

Today I had two locks which was weird, but maybe it was my fault. Therefore I'll get back in a week if it persists or not. Thanks to that new lock overview I can now see this way better.

Then I'll close this for now. Please reopen it, if the problem persists

I checked again and no new locked assessments appear after assessing the falsely locked ones. Also, the others can be assessed correctly. Seems to be fixed in the new release, good job.

authors: FuchsDominik julian-christl krusche


text:
Specific notification types

Checklist

[ ] I tested all changes and all related features with different users (student, tutor, instructor, admin) on the test server https://artemistest.ase.in.tum.de.
[x] Server: I added multiple integration tests (Spring) related to the features
[x] Server: I implemented the changes with a good performance and prevented too many database calls
[x] Server: I documented the Java code using JavaDoc style.
[ ] Client: I added multiple integration tests (Jest) related to the features
[x] Client: I documented the TypeScript code using JSDoc style.
[x] Client: I translated all the newly inserted strings into German and English

Motivation and Context
Tbd
Description
Tbd
Steps for Testing

Log in to Artemis
Navigate to Course Administration
...

I'll close this for now to improve the overview on the PR page. Feel free to open it at a later point when you work on it again


text:
Bugfixes: re-evaluate mc-quiz 2

Checklist

[x] I tested all changes and all related features with different users (student, tutor, instructor, admin) on the test server https://artemistest.ase.in.tum.de.
[ ] Client: I documented the TypeScript code using JSDoc style.

Motivation and Context

Follow-up PR to https://github.com/ls1intum/Artemis/pull/1515
After saving changes and changing thing again 'pending changes' isn't shown and therefore the later changes can't be saved.
Description

Use lodash to clone objects in order to prevent references being assigned.
(And also delete an empty file)
Steps for Testing

Log in to Artemis
Create a mc-quiz which is ready to re-evaluate.
3a. Change the text (or an answer option)
3b. 'pending changes' is shown
3c. save your changes
3d. 'pending changes' isn't shwon
4a. Change the text (or an answer option)
4b. 'pending changes' is shown
4c. save your changes
4d. 'pending changes' isn't shwon

Log in to Artemis
Create a mc-quiz which is ready to re-evaluate.
3a. Change the text (or an answer option)
3b. 'pending changes' is shown
3c. save your changes
3d. 'pending changes' isn't shown
Works fine upto here.
4a. Change the text (or an answer option)
4b. 'pending changes' is shown --> As of yet, I have to manually refresh the page and change the text again to see 'Unsaved Changes' in the quiz

Log in to Artemis Create a mc-quiz which is ready to re-evaluate. 3a. Change the text (or an answer option) 3b. 'pending changes' is shown 3c. save your changes 3d. 'pending changes' isn't shown
Works fine upto here.
4a. Change the text (or an answer option) 4b. 'pending changes' is shown --> As of yet, I have to manually refresh the page and change the text again to see 'Unsaved Changes' in the quiz

@Varnika-Tyagi where did you test it? I think the version containing the solution is not yet deployed to a test server

Ah, I'm using the test server 1 ( https://artemistest.ase.in.tum.de) for testing.

Ah, I'm using the test server 1 ( https://artemistest.ase.in.tum.de) for testing.

@Varnika-Tyagi it should be deployed there now

Works as expected. Well done!

One (probably unrelated) thing I noticed was, that changes which contain Case changes (e.g. changing quiz title from Test to TEST) are not recognized as changes. Do you know whether this is desired?

As it was implemented like that before, I assume that it is desired behaviour. @krusche should I change it to be case-sensitive?

@tessaRuckstuhl @krusche it is case-sensitive now, since I think it makes more sense

Works as described, new changes can always be saved.
But I ran into a strange issue how changes are detected if you make a small change (e.g. add a letter) and then undo that change again (delete or undo command). One would expect it to show "Saved!", but it depends on some other factors:

In the first answer text area of each MC quiz question, it works as expected.
In the question text area and other answer text areas, it does not work, it only shows "Unsaved Changes", even after that undo.
However, as soon as I save this no-change once, this text area becomes "immune" to the bug from then on, as long as I don't go back or reload the page. The other text areas don't change.

Note: It behaves exactly the same and in a reproducible way both in Chrome and Firefox.
My first guess would be that backupQuiz does not get loaded correctly when the re-evaluate is opened and the JSON-Strings differ in some way, but I am not experienced with TS.
I assume this bug is out of scope here and requires a new PR, since the main goal to allow saving changes works. Is that correct? I would then approve this PR.

@MaisiKoleni Thanks for testing.
Yes I've encountered the issue myself, too, but I will address it in a follow-up PR, since this one's goal is to make changes savable which is quite an important functionality.

authors: Varnika-Tyagi filip-gregurevic


text:
General: Improve student statistics layout, show learning goals to students again and align colors on course overview

Checklist
General

[ ] I tested all changes and their related features with all corresponding user types on a test server.
[x] Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
[ ] I chose a title conforming to the naming conventions for pull requests.

Client

[x] I followed the coding and design guidelines.
[x] I documented the TypeScript code using JSDoc style.
[x] I added multiple screenshots/screencasts of my UI changes.
[x] I translated all newly inserted strings into English and German.

Motivation and Context

The current layout of the student statistics view is improbable. @krusche pitched me some ideas about an enhanced design.
These ideas are realized in this PR.
Additionally, the learning goals are currently not displayed to the students. This is the case because the activated route is not able to fetch the course id from the parent. This information is needed for the REST request to the server.
Description

The following changes have been done:

General structure: The different charts are now aligned below each other. Additionally, every section has its own title and is separated by lines
Ordering: Below the doughnut chart, the Performance in Exercises line chart appears. Below that, the horizontal bar charts are placed
Color distribution: The color legend in the doughnut chart is now moved from the horizontal bar charts to the chart itself. The color legend also ensures translation sensitivity.
Chart appearance: The height of a horizontal bar has been halved
Font size: The font size of the statistical values on the right side of the view have been aligned with the color legend of the charts
Color alignment: The doughnut chart on course overview now provide colors aligned with the reworked color distribution of Artemis.
Learning goals: The learning goals appear in the statistics again.

Steps for Testing

Prerequisites:

1 Course with exercises and participation
1 Instructor
1 Student
1 Programming Exercise with Complaints enabled

Log in to Artemis as student
Inspect the course overview page. Make sure the color of the doughnut charts are muted
Navigate to the student statistics tab
Check the new layout. Make sure everything is still readable and nothing is misaligned. Make sure no learning goals are visible
Log in as instructor
Create a learning goal for a exercise attached to a lecture
Log in as student again
Navigate to the student statistics tab
Make sure the learning goal is visible now
Also make sure everything works with the new dark mode
If you have suggestions, feel free to share them. Then we can consider them for potential changes

Review Progress

Code Review

[ ] Review 1
[ ] Review 2

Manual Tests

[x] Test 1
[x] Test 2

Test Coverage

Class/File
Branch
Line

course-statistics.component.ts
87.09%
95.09%

course-card.component.ts
88.67%
96.15%

Screenshots

The new colors of the doughnut charts on course overview:

First part of the new student's statistics structure:

Second part:

The code looks good to me. You could also think about if .statistics-legend is really needed. Instead you could use fw-bold (Link) 🤔

Thanks for mentioning that. I was not aware of the fact that bootstrap provides this class (even though it should be quite obvious that bootstrap has your back there as always). I changed the implementation accordingly. 👍


text:
Cordova dev process: cordova launches ahead of asset bundler

Expected Behavior
Running cordova prepare or any command that includes it should run the cordova app after rollup has finished bundling the css and javascript.
Current Behavior
cordova prepare launches too early, missing the updated bundle files.
Failure Information (for bugs)
Due to cordova-run-x being launched by run-p, there is no check built in place to prevent this.
Steps to Reproduce

www/ is empty of build.x / build.map.x files
cordova run dev-electron
The electron viewscreen will produce an empty / unstyled page, as build.css / build.js have not been created and were not moved to the platforms/x/platform_www directory

A solution would be to speed up the bundling process via tailwindcss: mode: 'jit' https://tailwindcss.com/docs/just-in-time-mode
This does not suffice to have everything bundled ahead of cordova though.

repo: lsglab/ticket-management


text:
🛑 SoHard Cloud is down

In 9f61f60, SoHard Cloud (http://cloud.lsy223622.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: SoHard Cloud is back up in 3a13751 after 655 days, 13 hours, 29 minutes.


text:
🛑 Mirror-爬 is down

In 49f6084, Mirror-爬 (https://ncov.zhouym.tech/) was down:

HTTP code: 404
Response time: 1024 ms

Resolved: Mirror-鱼露 is back up in 901c076.


text:
🛑 Mirror-爬 is down

In ecae386, Mirror-爬 (https://ncov.zhouym.tech/) was down:

HTTP code: 404
Response time: 907 ms

Resolved: Mirror-鱼露 is back up in 507c1df.


text:
🛑 Mirror-爬 is down

In 891a4c3, Mirror-爬 (https://ncov.zhouym.tech/) was down:

HTTP code: 404
Response time: 1020 ms

Resolved: Mirror-鱼露 is back up in a49b7cd.


text:
🛑 Mirror-没有女朋友跨年的群傻逼 is down

In a5f959b, Mirror-没有女朋友跨年的群傻逼 (https://x.ksfu.top/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Mirror-木生 is back up in 9447db8.


text:
🛑 Mirror-没有女朋友跨年的群傻逼 is down

In ba7fef0, Mirror-没有女朋友跨年的群傻逼 (https://x.ksfu.top/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Mirror-木生 is back up in 75a515a.


text:
🛑 Mirror-爬 is down

In 9286268, Mirror-爬 (https://ncov.zhouym.tech/) was down:

HTTP code: 404
Response time: 1179 ms

Resolved: Mirror-鱼露 is back up in 3242a19.


text:
🛑 Mirror-爬 is down

In 8175b4d, Mirror-爬 (https://ncov.zhouym.tech/) was down:

HTTP code: 404
Response time: 2468 ms

Resolved: Mirror-鱼露 is back up in 1a5d63a.


text:
🛑 Mirror-爬 is down

In 18bdb51, Mirror-爬 (https://ncov.zhouym.tech/) was down:

HTTP code: 404
Response time: 857 ms

Resolved: Mirror-鱼露 is back up in 55bca2e.


text:
🛑 Instagram is down

In 17c9452, Instagram (https://www.instagram.com/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Instagram is back up in efc515e.


text:
🛑 Antidopingová agentúra SR is down

In d56e753, Antidopingová agentúra SR (https://www.antidoping.sk/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Antidopingová agentúra SR is back up in 0a681cb.

repo: lubosm/minedusk-uptime


text:
Limit users according to client scope

Sorry if here is not the place to ask this, but I'm trying to use OAuth on my API, I already installed and it's working. I can get an access token for my user and I can make requests with it. Now I'm trying to restrict some routes according to clients scopes, but I don't know how to do it. I'll appreciate if someone can help me with this.
for example, i have these routes:
/state (get)
/state(post)
but I want to give the access to /state (post) if the user is using a specific client to access is it possible or I have to create user scopes? In any case how can I do it?

I would go with scopes in the case. When you generate a new access token you fetch it with required scopes and the use that access token to post to /state.

Thank you @vinkla I read more about scopes, and tryied it.
It's working as I wanted.

:+1:

authors: geshner vinkla

license: mit


text:
Gamepads' attribution to windows

Hello Lucas,
Glad to see that this project is still going.
Seeing as Christmas is approaching, the time for an extended-family-wide LAN party, and this calls for Nucleus Coop.
Problem is, the gamepads are still hard to associate to specific windows. With the Alpha 6, Borderlands 2 (Steam version, up to date), a kb+m and 3 XBOX 360 gamepads... Well, the program pretty much still behaves the way it did last year : that is to say, the keyboard always gets its own window but as far as gamepads go, there's always one that fucks up and controls nothing. At the very least, one gamepad will control 2 windows, sometimes all three.
Also, by the way, nice touch with the multi screen ability. That'll come in handy.
Thanks for your time, and good luck with this.

On the nucleuscoop.exe you have to wait for all the screen to be on the main menu (when you see the logo of the game) then you have to use the first player controller first, then the second, then the third etc... Or it is the other way : the last player controller to the first player controller.
Personally I still play on the the splitool.exe so I can't remember well how it's working on the nucleuscoop tool.

Okay, so now that you've explained it, it works. But only backwards: the last controller must be the first one to be pressed. Why wouldn't it work the other way around as well?

Don't ask me, i'm just a guy who want to enjoy this tool like everyone else and if I know something that the other don't because of hours of testing with my friends... I try to help ;)

I know it is unlikely you will see this, but where is any download for splitplay? I cant find anything but the codeplex link, and all the downloads there are broken

authors: Kyphros Sphaxtoast TomSputz

repo: lucasassislar/nucleuscoop


text:
docs: Add link to website at noticeable part

I found this repository at https://github.com/DeepSourceCorp/good-first-issue/issues/646#issuecomment-1749379277.
So, I reached this repository before visiting website, and I confused a little to find out the website.
More noticeable link help people like me. How about this change?
Thanks.

@tooth-palm Approved, but there are conflicts.

authors: lucavallin tooth-palm

repo: lucavallin/verto


text:
0.1.18 issues training SoundStream

Since the latest version, I'm getting an error training sound stream: AttributeError: 'tuple' object has no attribute 'to'
A more complete stack trace:
Traceback (most recent call last):
File "train_soundstream.py", line 20, in
trainer.train()
File "/mnt/c/audio-ml-workspace/audiolm/audiolm_pytorch/trainer.py", line 376, in train
logs = self.train_step()
File "/mnt/c/audio-ml-workspace/audiolm/audiolm_pytorch/trainer.py", line 271, in train_step
wave = wave.to(device)
AttributeError: 'tuple' object has no attribute 'to'
I'll see if I can figure out what caused this...

@djqualia this is my fault too, for trying to introduce some magic to make text conditioned training easier
should be fixed!

Yes that worked thanks!

authors: djqualia lucidrains

repo: lucidrains/audiolm-pytorch


text:
2 dumb questions about tweaking

In order to change the learning rate, does that mean substituting for the value 0.07 on line 32 in the latent coordinates section?
Changing the manual seed is trivial, but how does one "tweak the losses"?
Thank you

Not sure if this is what you're asking but you can change the learning rate with the command line flag --lr 0.07.

authors: illtellyoulater tangopapa

repo: lucidrains/big-sleep


text:
[Question] difference between num_mem_kv and num_memory_tokens

TransformerWrapper can take num_memory_tokens
AttentionLayers can take num_mem_kv
It seems like these are the same thing no ?
Otherwise, what's the difference?

Also, I think i've asked this somewhere before, but does having memory tokens achieve the same as having a null key/value? Essentially, attending to the memory tokens only, is the same as not attending to input at all, and therefore the same as attending to a null key/value?
Or am I wrong?
I can't see a use-case for having both memory tokens AND a null kv

@pfeatherstone it is slightly different
the memory tokens attends (as well as get attended to) and provides a dedicated information lane, as was explored here
the num_mem_kv are just extra key / values that get attended to, so you can view them as feedforward weights within the attention layer

whether they achieve the same thing will need more follow up papers. i would be curious to see if the vit pathological issues can be alleviated with null / memory key / values alone

I can't see a use-case for having both memory tokens AND a null kv

yes, going forward, i'm choosing one or the other for newer attention model depending on what fits better. if you go with memory tokens, you will not need the null key / values, but that's not true the other way around

In this paper https://arxiv.org/pdf/2309.16588.pdf they propose adding learnable registers. It looks like those are the same as memory tokens. Maybe. Looks like these are becoming more and more important.

@lucidrains The reason you need a null key-value is because of the softmax layer. Do you know if anyone has every tried replacing softmax with sigmoid. Then, even without a null key-value or memory tokens, the attention map can be zero.

@pfeatherstone yea, people have tried relu, although it underperforms softmax
sigmoid in particular was tried under a different context

@pfeatherstone are you a ML consultant? evidently you work in the audio domain and follow my repositories closely

authors: lucidrains pfeatherstone

repo: lucidrains/x-transformers


text:
🛑 Nextcloud is down

In 3756084, Nextcloud (https://nextcloud.kerl.io) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Nextcloud is back up in 92a220f.

repo: lucienkerl/status


text:
🛑 Nextcloud is down

In c3e462e, Nextcloud (https://nextcloud.kerl.io) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Nextcloud is back up in b588f24.


text:
java.lang.OutOfMemoryError: unable to create new native thread

Hey, for the past few weeks my servers have been timing out and not working due to 'java.lang.OutOfMemoryError: unable to create new native thread'.
There is a log below of the error on my bungee server. If anyone could help that would be great!
log on pastebin - https://pastebin.com/ic4X2kt2
16:12:50 [SEVERE] May 29, 2017 4:12:50 PM net.md_5.bungee.event.EventBus post
WARNING: Error dispatching event PlayerDisconnectEvent(player=ADAMbees) to listener me.lucko.luckperms.bungee.BungeeListener@71466383
java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:717)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
    at net.md_5.bungee.scheduler.BungeeScheduler.schedule(BungeeScheduler.java:101)
    at net.md_5.bungee.scheduler.BungeeScheduler.schedule(BungeeScheduler.java:85)
    at me.lucko.luckperms.bungee.LPBungeeScheduler.asyncLater(LPBungeeScheduler.java:83)
    at me.lucko.luckperms.common.managers.impl.GenericUserManager.scheduleUnload(GenericUserManager.java:144)
    at me.lucko.luckperms.bungee.BungeeListener.onPlayerQuit(BungeeListener.java:150)
    at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at net.md_5.bungee.event.EventHandlerMethod.invoke(EventHandlerMethod.java:19)
    at net.md_5.bungee.event.EventBus.post(EventBus.java:46)
    at net.md_5.bungee.api.plugin.PluginManager.callEvent(PluginManager.java:376)
    at net.md_5.bungee.connection.UpstreamBridge.disconnected(UpstreamBridge.java:58)
    at net.md_5.bungee.netty.HandlerBoss.channelInactive(HandlerBoss.java:56)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)
    at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
    at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:278)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1329)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
    at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:908)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:744)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at java.lang.Thread.run(Thread.java:748)

Not an LP issue.
See:
https://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-thread

repo: lucko/LuckPerms


text:
set permission command, fallback to true

Hi,
it would be great if the command to set a permission would automatically set it to true if the value is not defined. The need to set a permission to false is extremely low compared to true and most users probably never use it at all.

Thank you :)


text:
Package android files as .apk

Right now it seems that the android builds are only .so files and not .apk files. it would be great if this could be achieved to make life easier for people building for android. I for example have no idea on how to install a .so file on an android phone and run the app..

To generate the apk we need fyne package command supports go modules https://github.com/fyne-io/fyne/issues/655

Available on develop for testing

I sadly don't really own any Android devices. Will see if I can borrow a friends phone and test this sometime in the coming week :)

Sorry for the late testing, but I am getting this on building:
Status: Downloaded newer image for lucor/fyne-cross:develop-android
go [-e -json -compiled=true -test=false -export=false -deps=false -find=true -- .]: exit status 2: go: downloading golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028
go: extracting golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028
go: finding golang.org/x/net latest
go: downloading golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
go: extracting golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
# runtime/cgo
gcc: error: unrecognized command line option '-marm'; did you mean '-mabm'?

Could not package the Fyne app: exit status 1

@Jacalz good catch thanks. Indeed the docker android image need to be updated in order to support the upcoming version of fyne (v1.2.4). Current version should work against v1.2.3.
The image should be updated in next days, looking if we can update the base image to go v1.14 too :)
In the meantime, as temporary fix, installing the latest fyne cli command into the android docker image should fix:
cat <<EOF > Dockerfile
FROM lucor/fyne-cross:develop-android

# fyne release/v1.2.4 branch
# install fyne cli from the release/v1.2.4 branch
RUN GO111MODULE=on go get -v "fyne.io/fyne/cmd/fyne@79fe7824d3ae53b16019b1a7ffcf17fb6cfe217d"
RUN mv /go/bin/fyne /usr/local/bin
EOF

docker build -t  lucor/fyne-cross:develop-android .

authors: Jacalz lucor

repo: lucor/fyne-cross


text:
VERSION_NAME and VERSION_CODE not set in iOS

Though not super clear in the docs, this package pre-populates some useful variables.
These work as expected on Android, but on iOS both VERSION_NAME and VERSION_CODE were undefined.
I believe these should come from the Bundle versions string, short and Bundle version, which are set in the project I am using.

Hi @jehartzog . Same problem here. Were you able to get this values for iOS?

@iadcg No, I ended up using a specific IOS_VERSION_STRING and IOS_BUILD_NUMBER values and used a wrapper to grab the correct one for iOS vs android.

@jehartzog Sorry mate. I lost you there. Didnt get what you mean by using a wrapper. Im new to react native + iOS kinda thing. Can explain a bit more?

authors: iadcg jehartzog

repo: luggit/react-native-config


text:
Use Implicit Grant Type

Hi,
I am working on a project where the backend uses Implicit Grant type (I know this is a HUUUUUGE risk factor but I do not have hands on access to the backend).
Is there any way this plugin can work with Grant type explicit?

Hi, currently this plugin uses oauth2 which only allows the following grant types:

Authorization Code Grant
Client Credentials Grant
Resource Owner Password Grant


text:
performance opening gzip with from_path() and serde_json::from_reader()

Describe the bug
I was surprised that serde_json::from_reader(niffler from_path) was slower than Python's json.load(gzip.open("repodata.json.gz"))
$ time python3 baseline.py

real    0m2.863s
user    0m2.648s
sys     0m0.209s

$ time pypy3 baseline.py

real    0m1.680s
user    0m1.500s
sys     0m0.180s

$ cargo build -r ; time ./target/release/nifftime
   Compiling nifftime v0.1.0 (/home/dholth/prog/nifftime)
    Finished release [optimized] target(s) in 0.71s

real    0m10.187s
user    0m9.917s
sys     0m0.270s

It helps if you wrap the niffler reader in a let mut buffered = BufReader::new(reader);, for a time of real    0m3.448s
On the other hand, decompressing to std::io::sink is very fast.
nifftime.zip.zip

@dholth first of all thank for your interest on niffler project.
I reproduce your result. I work on a benchmark to compare data reading between a flate file and all compression available in niffler (flate file seems to be too fast compare to compressed file).
I think that these bad performances are not due to niffler which in itself does not do much. But maybe because of the library used in backend. A modification of build system (#54) is planned to allow the user to choose which backend features are activated.  I think this could help performance.

We already have benchmark to compare parsing with niffler and directly with backend crate, niffler is little bit slower but not by a factor of 10 as you have shown in your benchmark.

I compared pypy (Python) deserializing a gzipped 170Mb https://conda.anaconda.org/conda-forge/linux-64/repodata.json against niffler not-wrapped-by-a-BufReader.
I understand that the pure-Rust zlib is a bit slow. I would like to try the cloudflare zlib.
pypy  1.6s value = json.load(gzip.open("repodata.json.gz"))
 py3  2.9s value = json.load(gzip.open("repodata.json.gz"))
rust  3.4s (buffered)
rust 10.2s (unbuffered)

Diff between buffered and unbuffered is logic, I agree default zlib isn't the best one. If you use branch transitive_feature, you have access to feature to control which zlib is use. I will try to finish pull request #54 quickly.

I think the serde access pattern is harder on the unbuffered logic than simply copying to e.g. https://doc.rust-lang.org/std/io/fn.sink.html

Ok I write a small benchmark with a "usage" of json value in branch json_parse.
wget https://conda.anaconda.org/conda-forge/linux-64/repodata.json
gzip -k repodata.json
bzip2 -k repodata.json
xz -k repodata.json
cargo build --release --bin json
hyperfine "./target/release/json 1 repodata.json" "./target/release/json 0 repodata.json" "./target/release/json 1 repodata.json.bz2" "./target/release/json 0 repodata.json.bz2" "./target/release/json 1 repodata.json.gz" "./target/release/json 0 repodata.json.gz" "./target/release/json 1 repodata.json.xz" "./target/release/json 0 repodata.json.xz"

I get this result:
Benchmark 1: ./target/release/json 1 repodata.json
  Time (mean ± σ):      2.779 s ±  0.015 s    [User: 2.573 s, System: 0.194 s]
  Range (min … max):    2.762 s …  2.800 s    10 runs

Benchmark 2: ./target/release/json 0 repodata.json
  Time (mean ± σ):     39.408 s ±  0.116 s    [User: 6.217 s, System: 33.056 s]
  Range (min … max):   39.250 s … 39.656 s    10 runs

Benchmark 3: ./target/release/json 1 repodata.json.bz2
  Time (mean ± σ):      6.112 s ±  0.040 s    [User: 5.881 s, System: 0.206 s]
  Range (min … max):    6.055 s …  6.171 s    10 runs

Benchmark 4: ./target/release/json 0 repodata.json.bz2
  Time (mean ± σ):      6.094 s ±  0.072 s    [User: 5.860 s, System: 0.208 s]
  Range (min … max):    6.021 s …  6.241 s    10 runs

Benchmark 5: ./target/release/json 1 repodata.json.gz
  Time (mean ± σ):      6.489 s ±  0.033 s    [User: 6.276 s, System: 0.197 s]
  Range (min … max):    6.450 s …  6.560 s    10 runs

Benchmark 6: ./target/release/json 0 repodata.json.gz
  Time (mean ± σ):      6.533 s ±  0.085 s    [User: 6.305 s, System: 0.204 s]
  Range (min … max):    6.470 s …  6.763 s    10 runs

Benchmark 7: ./target/release/json 1 repodata.json.xz
  Time (mean ± σ):     11.254 s ±  0.054 s    [User: 11.003 s, System: 0.214 s]
  Range (min … max):   11.165 s … 11.341 s    10 runs

Benchmark 8: ./target/release/json 0 repodata.json.xz
  Time (mean ± σ):     11.243 s ±  0.037 s    [User: 10.985 s, System: 0.228 s]
  Range (min … max):   11.198 s … 11.335 s    10 runs

Summary
  './target/release/json 1 repodata.json' ran
    2.19 ± 0.03 times faster than './target/release/json 0 repodata.json.bz2'
    2.20 ± 0.02 times faster than './target/release/json 1 repodata.json.bz2'
    2.33 ± 0.02 times faster than './target/release/json 1 repodata.json.gz'
    2.35 ± 0.03 times faster than './target/release/json 0 repodata.json.gz'
    4.05 ± 0.03 times faster than './target/release/json 0 repodata.json.xz'
    4.05 ± 0.03 times faster than './target/release/json 1 repodata.json.xz'
   14.18 ± 0.09 times faster than './target/release/json 0 repodata.json'

Except for not compressed file, buffer version (1) and not buffer version (0), have almost same time, for me it's logic because stream decompressing algorithm use buffer.
It's not solve our performance trouble compare to python but it's a base of work

Comparison against python script:
Benchmark 1: ./target/release/json 1 repodata.json.gz
  Time (mean ± σ):      6.581 s ±  0.114 s    [User: 6.356 s, System: 0.200 s]
  Range (min … max):    6.488 s …  6.830 s    10 runs

Benchmark 2: ./target/release/json 0 repodata.json.gz
  Time (mean ± σ):      6.536 s ±  0.054 s    [User: 6.323 s, System: 0.194 s]
  Range (min … max):    6.483 s …  6.674 s    10 runs

Benchmark 3: ./script.py repodata.json.gz
  Time (mean ± σ):      1.990 s ±  0.019 s    [User: 1.711 s, System: 0.272 s]
  Range (min … max):    1.970 s …  2.030 s    10 runs

Benchmark 4: pypy script.py repodata.json.gz
  Time (mean ± σ):      1.219 s ±  0.017 s    [User: 0.951 s, System: 0.251 s]
  Range (min … max):    1.200 s …  1.244 s    10 runs

Summary
  'pypy script.py repodata.json.gz' ran
    1.63 ± 0.03 times faster than './script.py repodata.json.gz'
    5.36 ± 0.09 times faster than './target/release/json 0 repodata.json.gz'
    5.40 ± 0.12 times faster than './target/release/json 1 repodata.json.gz'

This is what I meant about buffering
} else if buffer == "2" {
        let value: serde_json::Value =
        serde_json::from_reader(std::io::BufReader::new(niffler::from_path(json_path).unwrap().0)).unwrap();
    count += value.as_object().unwrap().len();
    }

On my slow laptop,
$ time target/release/json 2 ../repodata-000.json.gz
2 ../repodata-000.json.gz
5

real    0m4.732s
user    0m4.201s
sys     0m0.530s

$ time target/release/json 1 ../repodata-000.json.gz
1 ../repodata-000.json.gz
5

real    0m13.151s
user    0m12.368s
sys     0m0.782s

niffler::from_pathalready use BufReader. For me add a BufReader on another BufReader haven't any impact but your example show an impact, maybe because the first buffer contains compress data and second buffer uncompress data. So uncompress buffer is more consume than compress buffer.
Did you use default gzip compression level ?

Yes I used the default level

Ok the double buffer seems to be a very good idea:
Benchmark 1: ./target/release/json 2 repodata.json
  Time (mean ± σ):      2.352 s ±  0.020 s    [User: 2.135 s, System: 0.204 s]
  Range (min … max):    2.321 s …  2.385 s    10 runs

Benchmark 2: ./target/release/json 1 repodata.json
  Time (mean ± σ):      2.843 s ±  0.028 s    [User: 2.607 s, System: 0.219 s]
  Range (min … max):    2.805 s …  2.886 s    10 runs

Benchmark 3: ./target/release/json 0 repodata.json
  Time (mean ± σ):     32.659 s ±  0.132 s    [User: 6.204 s, System: 26.310 s]
  Range (min … max):   32.418 s … 32.874 s    10 runs

Benchmark 4: ./target/release/json 2 repodata.json.bz2
  Time (mean ± σ):      5.072 s ±  0.077 s    [User: 4.813 s, System: 0.224 s]
  Range (min … max):    4.973 s …  5.191 s    10 runs

Benchmark 5: ./target/release/json 1 repodata.json.bz2
  Time (mean ± σ):      6.306 s ±  0.103 s    [User: 6.038 s, System: 0.226 s]
  Range (min … max):    6.150 s …  6.479 s    10 runs

Benchmark 6: ./target/release/json 0 repodata.json.bz2
  Time (mean ± σ):      6.318 s ±  0.084 s    [User: 6.041 s, System: 0.227 s]
  Range (min … max):    6.207 s …  6.485 s    10 runs

Benchmark 7: ./target/release/json 2 repodata.json.gz
  Time (mean ± σ):      2.596 s ±  0.029 s    [User: 2.362 s, System: 0.214 s]
  Range (min … max):    2.555 s …  2.666 s    10 runs

Benchmark 8: ./target/release/json 1 repodata.json.gz
  Time (mean ± σ):      6.691 s ±  0.080 s    [User: 6.414 s, System: 0.226 s]
  Range (min … max):    6.598 s …  6.885 s    10 runs

Benchmark 9: ./target/release/json 0 repodata.json.gz
  Time (mean ± σ):      6.663 s ±  0.053 s    [User: 6.409 s, System: 0.216 s]
  Range (min … max):    6.597 s …  6.742 s    10 runs

Benchmark 10: ./target/release/json 2 repodata.json.xz
  Time (mean ± σ):      3.592 s ±  0.150 s    [User: 3.352 s, System: 0.217 s]
  Range (min … max):    3.466 s …  3.779 s    10 runs

Benchmark 11: ./target/release/json 1 repodata.json.xz
  Time (mean ± σ):     11.597 s ±  0.055 s    [User: 11.294 s, System: 0.230 s]
  Range (min … max):   11.505 s … 11.652 s    10 runs

Benchmark 12: ./target/release/json 0 repodata.json.xz
  Time (mean ± σ):     11.673 s ±  0.088 s    [User: 11.364 s, System: 0.229 s]
  Range (min … max):   11.544 s … 11.781 s    10 runs

Thanks again. ./target/release/json 2 is pretty good. pypy's json parser is probably not comparable to serde (very different, and doing optimizations that Rust would not consider)

authors: dholth natir

repo: luizirber/niffler


text:
how to show an indicator (bullet) when not using thumbnails

Im my use case I don't want to show thumbnails but I would like to have some kind of indicator which shows how many images are in the gallery (eg. bullets ->  also with navigation capabilities)
Is this possible with ngx-gallery?
TIA

Hi @maku, unfortunately no.

I've added imageBullets and previewBullets in 5.8.0

I've added imageBullets and previewBullets in 5.8.0

Everytime i add imageBullets into my code i get this error
core.mjs:6485 ERROR TypeError: Cannot read properties of undefined (reading 'length')
at NgxGalleryImageComponent_ngx_gallery_bullets_2_Template (kolkov-ngx-gallery.mjs:875:3285)
at executeTemplate (core.mjs:9618:1)
at refreshView (core.mjs:9484:1)
at refreshEmbeddedViews (core.mjs:10609:1)
at refreshView (core.mjs:9508:1)
at refreshComponent (core.mjs:10655:1)
at refreshChildComponents (core.mjs:9280:1)
at refreshView (core.mjs:9534:1)
at refreshEmbeddedViews (core.mjs:10609:1)
at refreshView (core.mjs:9508:1)

authors: dv0c lukasz-galka maku

repo: lukasz-galka/ngx-gallery