text:
Fixed new lines issue in messages.

EOF keys in windows: ctrl+z.
I used sys.stdin.read instead of input.
tested on two channels works fine.

If i change code like u commited it i get this:
Messages will be sent to https://discord.com/channels/
Traceback (most recent call last):
File "C:\Users\User\Desktop\discord_bot\bot.py", line 110, in
message = sys.stdin.read("Enter your message, when finished press EOF:")
TypeError: argument should be integer or None, not 'str'
Thank you!

If i change code like u commited it i get this:
Messages will be sent to https://discord.com/channels/ Traceback (most recent call last): File "C:\Users\User\Desktop\discord_bot\bot.py", line 110, in message = sys.stdin.read("Enter your message, when finished press EOF:") TypeError: argument should be integer or None, not 'str'
Thank you!

Oops sorry added the wrong print function. Fixed the issue.

If you are using windows:
1)Press Enter for a newline after entering your message.
2) In Windows, press Ctrl + Z to enter an EOF. (this depends on your operating system for example for macos: "software recognizes EOF when Control+D is pressed at the beginning of a line.")
3) Press Enter for a newline again.
4) program jump to messages amount input.

authors: Heggeg direrFlood

repo: nonrice/discord-auto-message


text:
Remove long queries from read_system

Environment info

Version: 1.11
Deployment: NA
Customer: NA

Actual behavior

read_system performs queries to the mongo, this can take quite a while in systems with many entities (buckets, accounts,objects etc.)
For example -
obj_count_per_bucket: MDStore.instance().count_objects_per_bucket(system._id)

Expected behavior

Should avoid querying and use cached / pre-calculated data
Add a tooltip for aggregated calculations (number of objects - bucket list and bucket summary) that data is aggregated over time

Relates to #5593

source: gharchive/issue

repo: noobaa/noobaa-core

license_type: permissive

license_source: github-api


text:
Start markdown

Testing

Testing my learning

source: gharchive/pull-request

license: MIT


text:
Mempool mirror tests

Adding test cases to MempoolMirror.cs

Why close this?

Didn't your branch changed? In the new Mempool Mirror PR the branch goes like revert-revert...
But I will re-open it

I think you're right. I didn't realize this was for the old branch. Anyhow, waiting for the real PR.


text:
RA-01-SH: WIFI Conflict

Hi,
RA-01-SH, utilizing SX1262, operates properly until the invocation of esp_wifi_start(); && esp_wifi_connect();. Below are the pin configurations, utilizing SPI2 host:
Clk = 18 Mosi = 23 Miso = 19 CS = 16 DIO1 = 15 RST = 22 BUSY = 13
Thanks

These samples use it at the same time as WiFi.
https://github.com/nopnop2002/esp-idf-sx126x/tree/main/http
https://github.com/nopnop2002/esp-idf-sx126x/tree/main/mqtt

Thanks, It's a great library.


text:
Is it possible to use express-formidable only to parse the bodies of certain POST requests?

I am currently using body parser to parse the bodies of the requests, etc, but I need to upload big files, songs and videos, to the server (so far my requests consisted of text or json), so I thought this module could be useful. I was wondering if it can be used along with body parser and specifically to parse only certain requests, letting body parser continue to work as before.
If it is possible to do it, could you please show a concrete example?

Hi, sorry for the (very) late response as I've not worked on this project personally. express-formidable is just a connect middleware, so how to use is completely up to you.
You can use it for some of routes:
app.post('/no-upload', (req, res) => {
  // ...
});

app.post('/upload', formidable(), (req, res) => {
  req.files // ...
});

You can even decide to use it in a single route(may not look so good though):
app.post('/upload', (req, res) => {
  if (someSetting.version === 'old') {
    res.end('no');
    return;
  }
  formidable()(req, res, function (err, req, res) {
    // ...
  });

About using a middleware, please refer to the middleware doc of express.

http://expressjs.com/en/guide/using-middleware.html

Thanks!

authors: nelson-brochado noraesae

repo: noraesae/express-formidable

license: mit

license_source: bigquery


text:
Added all jupyter notebooks to Travis

Last time I only added the BASIC and Cheryl notebook to the notebooks that are evaluated in Travis, now added all notebooks in the ipynb folder...
While testing on my local computer some notebooks did not fully run. If this is appreciated I might have some time to look into the possible issues next week. Let me know if this is something you want!

Related to this: some notebooks can only run in Python 2, others only in Python 3. Any preference for having them all in Python 3, and/or runnable in Python 2 by importing things from the future package?

Ah, that's the problem with having some notebooks from many years ago. My preference would be to do what is best for the user, and since we don't know what version a usert will be running, having them all run in both versions is best, if that's not too complicated. I think from __future__ import print_function should be used everywhere. The other major complication would be things like map being a list versus an iterator -- that might be more complicated to fix up.

authors: norvig rmeertens

repo: norvig/pytudes


text:
Use makefile in subdirectory instead of project root

Hello, I love this tool. I work with multiple components in one project, and I use multiple Makefiles in subdirectories of project. When I open the folder of the project, atom-build uses the makefile in project directory. Can I make it use the closest makefile to the ACTIVE_FILE?

With something like that:
{
  "cwd": "cd {FILE_ACTIVE_PATH} && make"",
  ...

no?

Almost, if you change cwd to cmd as key.
But I would do it like this:
{
  "cmd": "make",
  "cwd": "{FILE_ACTIVE_PATH}"
}

authors: alshakh mthcrts noseglid

repo: noseglid/atom-build


text:
可以设置每一句话出现的时间吗？

现在每一句话出现的时间是一样的吗？可以动态设置每一句话出现的时间吗

你好。可以的。
关于出现的时间没有在typeMonkey.js中定义。
先拿simple案例来举例。
关于时间控制是直接简单粗暴的setInterval
tm.init()
let going = setInterval(()=>{
  if ( tm.isEnd ){
    clearInterval(going)
    return;
  }
  tm.start()
},1000)

到了music中。每句话出现的时间是读lrc歌词文件中判断的。
lrc歌词文件中时间格式如下。
[00:00.00]先帝他妈才干到一半
[00:01.74]就撒手不管了
[00:03.19]孤儿寡母的就这么扔给我了
[00:05.36]有木有
[00:06.28]有木有啊！！！

通过正则将时间标签取出来->lrcFormat方法
然后requestAnimationFrame轮询去执行musicGoing方法。
根据当前audio的currentTime。来判断应该显示哪一句。
var currentTime = $mp3.currentTime,
    currentMs = currentTime * 1000;
for (let i = 0, lrc = music.lrc.list, length = lrc.length; i < length; i++) {
  if (currentMs >= lrc[i].time && (i === length - 1 || currentMs < lrc[i + 1].time)) {
    if (music.currentID === i) {
      break;
    }
    music.currentID = i;
    tm.next();
    break;
  }
}

typeMonkey.js没有设置列表中每句话出现时间的功能。需要自己去模拟。。。
要说为什么呢。我觉得时间这个功能。应该是结合音乐的。
这就需要lrc歌词文件来定义了。
毕竟我们自己手写出现时间的话。还是不好判断的。

authors: nostarsnow znbflyingant


text:
Add Items from NIP-24

There was a discussion on NIP-24 items here:
https://github.com/nostr-protocol/nips/issues/1253
The following 4 fields could be added to the schema, and are in wide usage:

display_name: an alternative, bigger name with richer characters than name. name should always be set regardless of the presence of display_name in the metadata.
website: a web URL related in any way to the event author.
banner: an URL to a wide (~1024x768) picture to be optionally displayed in the background of a profile screen.
bot: a boolean to clarify that the content is entirely or partially the result of automation, such as with chatbots or newsfeeds.

The first being a string, the next 2 URIs, and the final term a boolean

Should probably include the fields from NIP01 too:
https://github.com/nostr-protocol/nips/blob/master/01.md

0: metadata: the content is set to a stringified JSON object {name: , about: , picture: <url, string>} describing the user who created the event. Extra metadata fields may be set. A relay may delete older events once it gets a new one for the same pubkey.

Note: forgot to add bot.  But that may be just as well, because parent class is Person.  And bot may be Agent.  Tricky!

repo: nostrcg/schema


text:
Segmentation fault qemu-system

Describe the bug
Cannot get past the setup step for Monterey install.
To Reproduce

Tried both the "manual install" and the "windows install" procedures separately.
Run HEADLESS=1 ./basic.sh and connect via vnc
Create and format new disk
Let the setup procedure run
When finishing the setup it crashes with ./basic.sh: line 41: 173 Segmentation fault qemu-system-x86_64 "${args[@]}"

Any attempts to boot the mac-vm or doing a re-install on the same disk crashes immediately with the same error

Expected behavior
For install OS to succeed and create a bootable partition.
Desktop (please complete the following information):

Windows 11, WSL2 running:

Ubuntu 20.04 running:

$: uname -r
4.19.104-microsoft-standard

Additional context
Getting a bunch of ALSA related errors as well - I have no ALSA support currently so this is not weird.
Log output

$: HEADLESS=1 ./basic.sh
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open default': alsa: Reason: No such file or directory ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default alsa: Could not initialize DAC alsa: Failed to open default':
alsa: Reason: No such file or directory
audio: Failed to create voice `dac'
BdsDxe: loading Boot0001 "UEFI QEMU HARDDISK QM00017 " from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x2,0xFFFF,0x0)
BdsDxe: starting Boot0001 "UEFI QEMU HARDDISK QM00017 " from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x2,0xFFFF,0x0)
#[EB|LOG:EXITBS:END] _
#[EB.BST.FBS|-]
#[EB|B:BOOT]
#[EB|LOG:HANDOFF TO XNU] _
======== End of efiboot serial output. ========
qemu-system-x86_64: warning: guest updated active QH
qemu-system-x86_64: warning: guest updated active QH
BdsDxe: loading Boot0001 "UEFI QEMU HARDDISK QM00017 " from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x2,0xFFFF,0x0)
BdsDxe: starting Boot0001 "UEFI QEMU HARDDISK QM00017 " from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x2,0xFFFF,0x0)
#[EB|LOG:EXITBS:END] _
#[EB.BST.FBS|-]
#[EB|B:BOOT]
#[EB|LOG:HANDOFF TO XNU] _
======== End of efiboot serial output. ========
*./basic.sh: line 41:   173 Segmentation fault      qemu-system-x86_64 "${args[@]}" *

@RasmusWesterlundh Try replacing basic.sh with this:
#!/usr/bin/env bash

OSK="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
VMDIR=$(realpath $(dirname $0))
OVMF=$VMDIR/firmware
#export QEMU_AUDIO_DRV=pa
#QEMU_AUDIO_DRV=pa

MOREARGS=()

[[ "$HEADLESS" = "1" ]] && {
    MOREARGS+=(-nographic -vnc :0 -k en-us)
}

qemu-system-x86_64 \
    -enable-kvm \
    -m 4G \
    -machine q35,accel=kvm \
    -smp 4,cores=2 \
    -cpu Penryn,vendor=GenuineIntel,kvm=on,+sse3,+sse4.2,+aes,+xsave,+avx,+xsaveopt,+xsavec,+xgetbv1,+avx2,+bmi2,+smep,+bmi1,+fma,+movbe,+invtsc \
    -device isa-applesmc,osk="$OSK" \
    -smbios type=2 \
    -drive if=pflash,format=raw,readonly=on,file="$OVMF/OVMF_CODE.fd" \
    -drive if=pflash,format=raw,file="$OVMF/OVMF_VARS-1024x768.fd" \
    -vga qxl \
    -device ich9-intel-hda -device hda-output \
    -usb -device usb-kbd -device usb-tablet \
    -netdev user,id=net0 \
    -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
    -device ich9-ahci,id=sata \
    -drive id=ESP,if=none,format=qcow2,file="$VMDIR/ESP.qcow2" \
    -device ide-hd,bus=sata.2,drive=ESP \
    -drive id=InstallMedia,format=raw,if=none,file="$VMDIR/BaseSystem.img" \
    -device ide-hd,bus=sata.3,drive=InstallMedia \
    -drive id=SystemDisk,if=none,file="$VMDIR/macOS.qcow2" \
    -device ide-hd,bus=sata.4,drive=SystemDisk \
    "${MOREARGS[@]}"

Thanks @notAperson535 - the main changes seems to be replacing OpenCore.qcow2 with one called ESP.qcow2?
-    -drive id=OpenCore,if=none,format=qcow2,file="$VMDIR/OpenCore.qcow2" \
-    -device ide-hd,bus=sata.2,drive=OpenCore \
+   -drive id=ESP,if=none,format=qcow2,file="$VMDIR/ESP.qcow2" \
+   -device ide-hd,bus=sata.2,drive=ESP \

This fails with a No such file or directory when running HEADLESS=1 ./basic.sh.
How do I generate the ESP-drive instead of the OpenCore-drive to try this out?

@RasmusWesterlundh oh sorry, change esp.qcow2 to OpenCore.qcow2

Still the same issue with the suggested changes. The apple-logo shows up and it begins finishing setup, but after a while it crashes with:
#[EB|LOG:EXITBS:END] _
#[EB.BST.FBS|-]
#[EB|B:BOOT]
#[EB|LOG:HANDOFF TO XNU] _
======== End of efiboot serial output. ========
./basic.sh: line 38:   321 Segmentation fault

After changing from ESP back to OpenCore my basic.sh ended up looking like:
#!/usr/bin/env bash

OSK="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
VMDIR=$(realpath $(dirname $0))
OVMF=$VMDIR/firmware
#export QEMU_AUDIO_DRV=pa
#QEMU_AUDIO_DRV=pa

MOREARGS=()

[[ "$HEADLESS" = "1" ]] && {
    MOREARGS+=(-nographic -vnc :0 -k en-us)
}

qemu-system-x86_64 \
    -enable-kvm \
    -m 4G \
    -machine q35,accel=kvm \
    -smp 4,cores=2 \
    -cpu Penryn,vendor=GenuineIntel,kvm=on,+sse3,+sse4.2,+aes,+xsave,+avx,+xsaveopt,+xsavec,+xgetbv1,+avx2,+bmi2,+smep,+bmi1,+fma,+movbe,+invtsc \
    -device isa-applesmc,osk="$OSK" \
    -smbios type=2 \
    -drive if=pflash,format=raw,readonly=on,file="$OVMF/OVMF_CODE.fd" \
    -drive if=pflash,format=raw,file="$OVMF/OVMF_VARS-1024x768.fd" \
    -vga qxl \
    -device ich9-intel-hda -device hda-output \
    -usb -device usb-kbd -device usb-tablet \
    -netdev user,id=net0 \
    -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
    -device ich9-ahci,id=sata \
    -drive id=OpenCore,if=none,format=qcow2,file="$VMDIR/OpenCore.qcow2" \
    -device ide-hd,bus=sata.2,drive=OpenCore \
    -drive id=InstallMedia,format=raw,if=none,file="$VMDIR/BaseSystem.img" \
    -device ide-hd,bus=sata.3,drive=InstallMedia \
    -drive id=SystemDisk,if=none,file="$VMDIR/macOS.qcow2" \
    -device ide-hd,bus=sata.4,drive=SystemDisk \
    "${MOREARGS[@]}"

@RasmusWesterlundh how many cpu cores and ram are you using?

@notAperson535 8 cpus , 8gb ram

@RasmusWesterlundh try the default settings

Geez I'm so sorry for wasting your time with this @notAperson535  and honestly more sorry that it took me 12 hours to realize that the disk I'm creating the image on is full (becomes 0 bytes free on the physical disk during the macOS install).
I will change to a different disk and reopen the issue if any problems persist after that.

Same mistake, lol

authors: RasmusWesterlundh diskree notAperson535


text:
chore: fixed error messages of trust policy

A quick PR to fix error messages of trust policy.

Codecov Report

Merging #326 (ab6cc3f) into main (d65bba3) will not change coverage.
The diff coverage is 100.00%.

:exclamation: Current head ab6cc3f differs from pull request most recent head 1986706. Consider uploading reports for the commit 1986706 to get more accurate results

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@           Coverage Diff           @@
##             main     #326   +/-   ##
=======================================
  Coverage   74.80%   74.80%
=======================================
  Files          23       23
  Lines        2203     2203
=======================================
  Hits         1648     1648
  Misses        437      437
  Partials      118      118

Impacted Files
Coverage Δ

verifier/trustpolicy/trustpolicy.go
91.48% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: Two-Hearts codecov-commenter

repo: notaryproject/notation-go


text:
Should not be able to restore an anonymous (pure) stub

For anonymous (pure) stubs, that is to say a stub created via stub = sinon.stub() that is not overriding another function, the Sinon.JS library does not allow you to call the restore() method.
e.g.
stub = sinon.stub()
stub.restore() // throws "TypeError: stub.restore is not a function"

While the exact error message doesn't matter too much, Sinon.PY should probably also throw an error. It doesn't make sense to try to restore a pure stub.

The solution is simple but unsafe.
If user try to write:
stub = sinon.stub()
try:
    stub.restore() // throws "TypeError: stub.restore is not a function"
except:
    ...
# later, the _queue still contains anonymous stub, it should be fixed
# clean up `pure` might be harmless in this case, but it will make code inconsistent against to other types


text:
Contador para el siguiente feriado

Seria interesante tener un contador para el siguiente feriado nacional

Gracias por la sugerencia. Lo haremos.

Agregado en la nueva version

repo: noteregalesuwu/frontend-when-cobramos


text:
Add popularity column to notebooks

Add a column called popularity or similar to the table notebooks. The default value should be zero.
Then, when someone clicks on “Run in Livebook” or on the link to GitHub on a notebook page such as this one, increase popularity by one.
This was proposed by Rick_Morin in the elixirforum to track the interest in notebooks. Thanks!

i can try to do this one, this is my first contribution so please understand

authors: hectorperez tanavdev

repo: notesclub/notesclub


text:
bottom part of the fragment in the viewpager is clipped or cut of how to fix this

hello
i have a view pager with fragment in the scrollable , when i scroll down you can notics that the bottom part of the fragment in the viewpager is clipped or cut of how to fix this

Hello @xbahaa !
Well, let's see it. Can you please attach your layout xml and a screenshot of this behaviour?

https://youtu.be/tgz2fMPRBcc
in the bottom of the view some part of the item is cut of this a tabs and view pager in the viewpager
there fragment contains a recycle view how to fix this

Hey @xbahaa !
I didn't notice any cutout on the video... can you share the XML that is used to build the layout?

@xbahaa try changing the ViewPager like this:
<android.support.v4.view.ViewPager
    android:id="@+id/productDetailsViewPager"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:layout_marginTop="48dp"/>

It changes the layout_marginTop attribute to match the height of the AppBarLayout

thanks for you it;s work's like a charm, but way it's works that's the question?

@xbahaa it works because:

ScrollableLayout positions its children like vertical LinearLayout, but measures them like a FrameLayout

This is very important part to know when creating layouts with ScrollableLayout. So, if you put 2 views inside a ScrollableLayout with layout_height="match_parent" they both will have the same height (equal to the height of ScrollableLayout) but one will be placed under another.

authors: noties xbahaa

repo: noties/Scrollable


text:
How to append blocks

I'm trying to append paragraph blocks to a page and it gives me an exception. Exception claims text is not defined even though it's in the block.
Sample code and exception report attached.

After checking out unit tests for adding blocks, I discovered I was not creating the Paragraph blocks correctly.

repo: notion-dotnet/notion-sdk-net


text:
Rename project to pylogctx 1.0

@lorehov here is mine.

Green :-)

Yeah. This is much better :)

authors: bersace lorehov

repo: novafloss/pylogctx


text:
Polkadot Facade Feedback

We are working towards building better APIs to simplify the interacting with "Polkadot Network".
What do we mean here by Polkadot Network? Foremost Polkadot Relay Chain and System Chains (everything DOT related), and then other Parachains.
In a few months (and after gathering feedback via such issues), more details of this project will be shared. To give it a name for now, we are calling it "Polkadot Facade".
The aim of this issue is to learn about the instances where you had to interact with Polkadot Network (as defined above), and you found it difficult. This interaction can be in any form, but to help give you a mental model, I can categorize it as such:

🤓 read: You had/wish to read something from the chain (most likely a pallet's storage / runtime-api), and you found it difficult.
✍ write: You had/wish to compose a transaction to do something, and you found it difficult.

A few notes:

I used the "had/wish" intentionally: I would like to hear both about scenarios that you had to do something, and they were hard, and those that you couldn't even figure out how to do because it was too hard.
You can focus both on single-network interactions, and multi-network (involving XCM) interactions. That being said, we will likely first consider single-network cases.
When it comes to reading, don't limit yourself to what can be read from the state of a certain block, historical queries also count.

Please feel free to reach out to me on Matrix (@kianenigma:parity.io) if need be.

Hey Kian! Thank you for opening the issue. We are gathering the feedback from our and once ready will publish it here.

authors: ERussel kianenigma

repo: novasamatech/nova-wallet-ios


text:
🛑 NpComplete is down

In 0c6b2f1, NpComplete (https://npcomplete.dev/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: NpComplete is back up in 2fd1944.

repo: npcomplete-dev/podcast-site-upptime


text:
ブック・トピックの編集ボタンはホバー時のみ表示する

どのような問題を解決する機能追加でしょうか?
編集権限のあるユーザーがブック閲覧等を表示した際に編集ボタンが表示されるが、
画面上に常時編集ボタンが表示されているため、操作用のUIが煩雑に配置されている印象をうける
あなたが望む解決方法について書いてください
マウスホバー時のみ編集ボタンが表示されるようにスタイルを変更する

編集ボタンを表示することで編集可能かどうかのインジケーターにもなっているので、ホバー時表示という変更にする必要はない

repo: npocccties/ChibiCHiLO


text:
feat:ユーザ情報の同期機能

0b10f3f730bc9478f7e66cdbb5b91917cd6b278e
ここまでで、Learning Tools Interoperability Names and Role Provisioning Services(以下：NRPS)でLMSのmembershipの値を取得するメソッドを作成するところまで作成した。
TODO一覧
受講者を同期する実装

[x] initでupsertLtiMemberで初期化している箇所を削除（ 0b10f3f730bc9478f7e66cdbb5b91917cd6b278e ）

[x] LSMユーザーをNRPSで取得するAPIを作成する。（ e2fddbbc0e7805513ee4e0b53fbddd19d8dc008f ）

[x] クライアントで上記のAPIを呼び出す。

[x] 「受講者を同期（登録）」ボタンを設置する。( 67f26b830c685b91a1ebe57207b542b75d1233a5 )

[x] ボタンをクリックした際に、呼び出したユーザーの値を表示する（ひとまずidのみを表示した 67f26b830c685b91a1ebe57207b542b75d1233a5 624f64093d4b5fbc6d750e432d0031362652e977 ）。

[x] upsertMemberships（ltiMemberテーブルをLMSユーザーIDのものに更新する関数）を作成する( 71c55b61207ebc3c681cdb4553afa427b2e3c21d )。

[x] 確認アラートをクリックした際に、上記のAPIを呼び出してltiMemberテーブルを更新する( 8eecb665d7c5c469b4d8cc00702ba42a6e839c92 )。

[x] 既存の学習履歴に存在しない学習者のみを表示するようにする( 0fe310197d4c0687a02406cafeb12b5c8c4b44d5 1fbda6942371024fe8abf28c55978a6098531f30)

[x] NRPSでname, idを返すようにできるか調査（無理そうであればuserのidを表示するようにする）
nameはNRPSで取得することはできないので、ログイン前ユーザーは「名前非公開」などのラベル、ログイン済みのユーザーはユーザー名などを表示するようにする

学習者のブック視聴履歴の更新
(この処理は不要。LtiMemberテーブルにLtiMemberが紐づいているため)

[x] ブックなど学習状況の更新時に渡すIDもuser.idになっているので、上記のMembershipのIDで更新をするように変更する。

初回同期（初めて更新する際には、確認アラートを出さない）
↑の仕様変更により、既存のテーブルが消えるわけではないので、この初回同期のみの変更は不要と考える。
（新学期の場合など、）学習活動が０の場合という条件で以下のようなロジックの出し分けをする。

[ ] 「受講者を登録」に文言を変更する。
[ ] ボタンをクリックしたときにupsertMembershipsを呼び出してテーブルを更新する

その情報をもとにupsertLtiMember(s)を行う（LtiMemberテーブルの更新）←既存のltiMemberのidとmembership.idが一致した場合に、更新するようにできる？（一括で差し替える方法がなさそう？）

Prismaには、「存在しなければcreate、もしすでに存在すればupdate」という振る舞いの upsert() API は存在しますが、それを複数レコードに対して実現する upsertMany() のような API は提供されていないです。
これはRDBテーブルの有無とは関係無く、Prismaを使う以上はやむを得ない部分かと思います (今後のバージョンアップで変わる可能性はあるかもですが)。
そういった機能を提供するAPIは存在しないですが、もちろん実現すること自体は可能で、例えば

upsert() を複数呼び出す
先に delete() を呼び出し削除して → createMany() を呼び出す

など考えられます。
参考: CRUD (Reference) https://www.prisma.io/docs/concepts/components/prisma-client/crud
参考: Prisma で Upsert Many はないのか - Qiita https://qiita.com/dosukoi_man/items/ca3d7202a8762169edd0

なるほど、参考資料までありがとうございます！
仕組み的には実装可能な感じですね。
LMSのmember.idとchibichiloのuser.idが一致するものがあるか調べつつ、パターン１の方向で進めてみようかと思いました。
仕様を１回めの同期で、差分を表示するアラートを出すとしても良いかと思いました(既存のデータの更新となるので)。1回目の時は「LMSとの同期を行います」みたいなメッセージがあると良いかも。
以下の仕様の部分です。

コース毎の「受講者の同期 (初回は「受講者の登録」ボタンの方が良いかも)」ボタンを押した初回は、自動的に反映するが、2 回目以降の更新作業で「受講者の同期」ボタンを押したときは、増減する受講者の人の情報 (増減人数 must, 名前もできれば) を表示した上で、実行して良いか Y/N 確認ダイアログを出した上で実行。

TODOを更新しました。上記のような方針で実装を進めていきます

LTI Names and Role Provisioning Service ParameteでLMSメンバーを取得するAPIを作成しました。やっぱり名前などは含まれないですね.....。

あとOpenAPIの型が Promise<[key: string]: object> のような変な型になってしまうのですが、これはこういうものなのでしょうか...？お手数ですがここまでレビューお願いします。

今のところのUIです。

メンバーのname,emailを取得するにはユーザーからの設定が必要？
ツールはこの値に依存すべきではないので、別の値を使って表示してあげる必要があるかもです。
https://www.imsglobal.org/spec/lti-nrps/v2p0#sharing-of-personal-data

Any other member attributes will need an explicit consent from the Platform to be shared with the Tool. The Platform may delegate that consent to the actual member, therefore a Tool should never rely on additional member attributes to be present.

ログイン前ユーザーの扱いが悩ましい。

Learning Tools Interoperability Names and Role Provisioning Servicesではユーザー情報が取れなさそうなので、表示させる要素がない
upsert.LtiMemberもログイン前ユーザーの場合、参照するユーザーのデータがないのでエラーが出てしまう。

model LtiMember {
  context    LtiContext @relation(fields: [consumerId, contextId], references: [consumerId, id])
  user       User       @relation(fields: [consumerId, userId], references: [ltiConsumerId, ltiUserId])
  consumerId String     @map("consumer_id")
  contextId  String     @map("context_id")
  userId     String     @map("user_id")

  @@id([consumerId, contextId, userId])
  @@map("lti_members")
}

upsert.LtiMemberもログイン前ユーザーの場合、参照するユーザーのデータがないのでエラーが出てしまう。

userについてもupsertするのが妥当かもですね。
ただ、ltiUserId以外に関してはLTI NRPSだと原理的に存在が約束されないので何かしら仮の値(例えば ""など)を設定しておく必要があるのかなと考えます。

表示させる要素

そうですね。もし仮にLTIにおけるUser IDのみしか存在しないのであれば、例えば「ID: 123」「名前非公開」などと表示するとよいのかも。重要なのはどのくらいの人数かということ、そしてその人がこのコースにおける受講者であるということ、この2点、本PRで伝わるような実装にできていればOKと考えます。

あとOpenAPIの型が Promise<[key: string]: object> のような変な型になってしまうのですが、これはこういうものなのでしょうか...？

LtiNrpsContextMembershipSchemaとLtiMembersSchemaに含まれるadditionalPropertiesを削除したところanyでなくなりました。おそらくSwagger Generatorの処理での不具合と思うので本件のスコープに含めなくてよいです。

@kou029w
念の為仕様の共有です。

コース毎の「受講者の同期 (初回は「受講者の登録」ボタンの方が良いかも)」ボタンを押した初回は、自動的に反映するが、2 回目以降の更新作業で「受講者の同期」ボタンを押したときは、増減する受講者の人の情報 (増減人数 must, 名前もできれば) を表示した上で、実行して良いか Y/N 確認ダイアログを出した上で実行。
https://github.com/orgs/npocccties/projects/3/views/1?pane=issue&itemId=15226205

こちら初回の時はダイアログを出さない&「受講者の登録」に文言を変更するという仕様がありますが、
「コース活動履歴に一人のユーザーも存在しない」という判定で上記の機能を作成しようかと思います。

残りタスクのメモ：

フロントエンドに仮の値を表示する
もし仮にLTIにおけるUser IDのみしか存在しないのであれば、例えば「ID: 123」「名前非公開」などと表示するとよいのかも。重要なのはどのくらいの人数かということ、そしてその人がこのコースにおける受講者であるということ、この2点、本PRで伝わるような実装にできていればOKと考えます。
初回同期時
こちら初回の時はダイアログを出さない&「受講者の登録」に文言を変更するという仕様がありますが、
「コース活動履歴に一人のユーザーも存在しない」という判定で上記の機能を作成しようかと思います。

ユーザー情報一覧のUIは以下のような感じで作成しました。
ログイン済みユーザーはUserデータが取れるはずなので、できれば表示したいと思っています。

未ログインユーザー（学習履歴がないユーザー）の場合、学習履歴の表示は以下のようになります。

ログイン済みユーザーはUserデータが取れるはずなので、できれば表示したいと思っています。

Moodleの場合、Site administration > Plugins > (CHiBi-CHiLO) > Privacy > [Share launcher's name (かつ email) with tool] で name プロパティ並びに email プロパティが得られることを確認しました。まずはこれらのプロパティの使用してもらえるのが良さそうです。(任意、本PRのスコープに含めずともよいです)

repo: npocccties/chibichilo


text:
Merge updates for support AdonisJS v4

Hello.
I'm add support for AdonisJS v4 and send PR
It would be great if you merge this changes.
Thanks
https://github.com/nrempel/adonis-scheduler/pull/4
https://github.com/nrempel/adonis-scheduler/pull/5

+1 @nrempel

Thank you @ntvsx193! I've just merged your changes and published to npm as version 3.0.0.
Your test suite passes but I haven't yet had time to test the provider in a real scenario.
If you have time, please test and let me know if you run into any issues.
Thanks all!

@ntvsx193 I've made you a collaborator for this repository.
If you would like to push changes, please make sure you are diligent with test coverage and code quality. I hope that you can help me to keep this library up to date!

@nrempel I would be update test, currently test states in progress.
Thx for access to contribute :)

authors: danilopolani nrempel ntvsx193

repo: nrempel/adonis-scheduler


text:
[nrf noup] include: net: socket_ncs: remove deprecated RAI

This commit removes deprecated RAI socket options.
Deprecated back in version 2.6.0.
manifest-pr-skip

I think this should have a !squash [nrf noup] prefix and mention the commit it should be squashed into: https://github.com/nrfconnect/sdk-zephyr/commit/d9216e3efe913cf0350ed3c24358d9e3c86576e4

True, although it's nrf-squash! now:
https://nordicsemi.atlassian.net/wiki/spaces/NCS/pages/108201225/Pull+Requests#PullRequests-ModifyingNCS-specificcommitsfromanOSSrepositoryfork

Updated.

@MirkoCovizzi I think the nrf squash! line should indicate which commit we should squash this one into:
nrf-squash! [nrf noup] include: net: add NCS extensions

authors: MirkoCovizzi lemrey rlubos

repo: nrfconnect/sdk-zephyr


text:
RFC: Ingest API over HTTP

This issue has been opened by SuperFly.tv on behalf of EVS Broadcast Equipment (henceforth EVS).
We propose to extend the Sofie API to present an ingest API to allow ingest data to be sent to Sofie from external sources via HTTP rather than DDP.
Motivation
There is a desire for an external system to be able to "push" data to Sofie via an API. As the HTTP API has been used in other areas of EVS work, it is desirable to also have this external system use the HTTP API.
Proposal
A draft of the shape of the API has been published and a PR has been opened.
There are a couple of patterns to point out in this draft. The first is the use of ETags and If-None-Match, the second is that the proposed API allows for external Ids to be used interchangeably with Sofie-internal Ids. Both patterns are explored below.
ETags and If-None-Match
The ETag and If-None-Match HTTP headers have been included as part of this proposal as a way of identifying the version of each ingest item (e.g. Rundown, Segment) when it is pushed to Sofie. Using these headers means that an external system can provide us a version string for some ingest data via the ETag and tell us to only update an existing item if it has a different ETag to the one provided. This means that we have a chance of defending against too many updates being sent by an external system, so long as the external system provides us with a sensible ETag to use as the version of the data (e.g. timestamp that the data was last modified).
Using this method also allows the entire API to be expressed as PUTs while still allowing for "forced" updates - e.g. in the case of re-syncing data. We achieve this by specifying that any update that is not accompanied by an If-None-Match header is always treated as a "create" event rather than an update, meaning that we will overwrite any existing data in this case.
This approach also allows for some performance improvements to be made in some cases, as we will be able to discard an update without performing any costly work within blueprints if we have received data that has the same version string as the object already in the database.
However, to use this version information we will need to store it somewhere which means that this proposed change does mean a change to the data present in the ingest data collections in the form of an optional version: string property. Also of note is the fact that bulk updates (e.g. a PUT for a list of Rundowns) will use If-None-Match to optimize bulk updates, but will not accept an ETag for these updates, so the next update to each individual element within a collection will be treated as a create. This is in-line with the recommended usage of ETag. The effect of this is likely to be that we will be able to optimize the initial ingest of data at the expense of unnecessary updates in the future when script edits are carried out on individual elements of the show.
Interchangeable Ids
In the current proposal, external systems can make calls to the Ingest API using either the Id internal to Sofie's database, or using an arbitrary external Id (so long as this Id is sufficiently unique). This allows the external system to interact with Sofie purely through its own Ids, without having to perform lookups to Sofie's API to maintain its own map between Ids. This also means that we can express the entire API as a series of PUTs rather than needing data to be created via a POST initially, which reduces the need for the external system to have an awareness of the data that is currently held within Sofie and by extension reduces the amount of knowledge the external system needs to have about the lifecycle of Sofie.
The cost of this approach should be minimal as it can be implemented using the $or operator in Mongo queries. Also, Ingest operations are generally not so time-sensitive that this lookup will cause excessive delay.
It may be useful to extend this approach to other areas of the API but that is outside the scope of this RFC.
Other things of note
The design of this API is such that the ingest objects contain a generic payload object, the exact shape of which is left to be agreed between the blueprints authors and the authors of the system that is pushing data to Sofie - in much the same way that the existing ingest gateways work. This also follows the pattern established by the Studio and ShowStyle config APIs, so anything that is developed in terms of validation and schema discovery for those methods can be shared with this API.
As part of the Playlist ingest objects, a resyncURL can optionally be provided that will be used by the "Reload data" action within the Rundown UI. This URL will receive a POST with an empty body to prompt an external system to resync data with Sofie, which is described as a request for all data for a Playlist to be re-sent to Sofie without the If-None-Match header present.
Request For Comment

Does this make sense to expose over the HTTP API?
Does the proposed shape make sense?
Are we happy to employ this pattern of mixing external Ids with Sofie-internal Ids?

So here are my very hot takes:

I think there is a general sentiment that we need to orient ourselves towards removing DDP as an integration protocol (at the very least), so I think having an HTTP ingest protocol is a good idea
That being said, I think having an way of doing ingest via HTTP requires us to map the PeripheralDevices API onto HTTP somehow. There's a couple of things to think about there:

Authentication
Configuration
Status
Callbacks (such as "Reload from NRCS" action in case of Ingest devices)

I don't think we should create an API only to then rip it out once we settle on - what we already know - is needed, so I personally would prefer that going this route would include settling on an API for HTTP-based PeripheralDevices, which I think is a worthy goal.

Now, for authentication, I can easily see Sofie going in the way of OAuth2 client credential workflow for PeripheralDevices, or going with a Authorization Code Flow for situations in which the Peripheral Device can act on behalf of a user. For callbacks, I think we could investigate something like GitHub's webhook system, which GitHub apps can automatically configure: decide what events they want to subscribe to, expose what callbacks they support, etc. However, what I can't immediately picture is what the Status reporting and centralized Configuration would look like in this API.
NRK's Sofie team has a "Developer's meeting" every Friday, I'll bring this RFC up, so that we can produce a fully-baked position.

So, here are our notes after the developers meeting, on the general sentiment of the NRK team:

We would like to have a roadmap for this feature, before we bring it into the fold. The REST API so far has been "remote buttons" for the system, executing actions that were otherwise available in the GUI anyway. This change would expose what used to be tied into the system through the server-integration package and make a "public" Ingest API. As such, since we've made a pinky-promise to keep the Stable API "Stable", unless otherwise neccessary, we would like to make sure that this API is reasonably complete and will be in fact "Stable" to the point where non-Sofie related systems can integrate it into their code.
Having a HTTP-only protocol for Peripheral Devices would be nice, and we've brainstormed a couple of solutions that could potentially enable feature-parity with DDP but still offer a graceful degradation to a "simple curl client". This would probably include some sort of a token for authentication and maybe an "upgrade" of a WebSocket connection for status reporting, pings and interactive configuration, allowing us to mirror DDP-based PeripheralDevices in functionality.
Ideally, the new HTTP ingest protocol should enable one to create various ingest gateways and combine the "generic" input gateway methods and the MOS-specific input methods into a single, flexible system
We think that this API, considering that it's supposed to be the "ingest" interface, should still maintain little knowledge about Sofie's internals. As such, we're unsure about including the knowledge of "Studios" and "Playlists", or Sofie internal IDs, since these are Sofie internal inventions.

(The opinions expressed below are my own - operating as an employee of SuperFly.tv - and do not necessarily reflect the opinions of EVS Broadcast Systems nor those of SuperFly.tv).
My responses to the above comments:

I think having an way of doing ingest via HTTP requires us to map the PeripheralDevices API onto HTTP somehow.

I disagree, as I see it the Peripheral Device API is an "internal" API - for devices within the ecosystem which Sofie has some degree of control over. From my perspective, the job of the ingest gateways is to translate data that is owned by an external service into data that becomes owned by Sofie (in the form of ingest data). These gateways have a very simple lifecycle as they do not own the their input nor their output, they're only a conduit for information; they also tend to not have an independent lifecycle as they require a connection to Sofie and attachment to a Studio in order to function.
The HTTP API, on the other hand, has been designed for systems with their own independent lifecycle in order for them to interact with Sofie as part of a wider, interconnected, ecosystem. Essentially, the gateways can be thought of as async tasks running within Sofie, as I see it. The fact that the gateways run as their own processes and communicate over a network protocol (DDP) instead of via in-process message channels/memory buffers/function calls is purely a convenience for allowing them to be run on-site while Sofie is hosted off-site. In fact, the design of DDP is akin to the use of memory buffers and function calls, which is why there is a very tight coupling between Core and the gateways.
If there is a desire to remove DDP, I would suggest that perhaps the nature of the gateways needs to be reconsidered. If we were to think of gateways as truly standalone services with their own, independent, lifecycles and ownership of ingest data, then what might change? To address some points under the above quote:

Authentication

I'd suggest that this is out of scope as it is a topic that affects the whole HTTP API and should be solved at the top-level of the API. Further, standard HTTP authentication methods are, when running in an open state, entirely backwards compatible as they use headers added to existing request without modifying the request body.

Configuration

If ingest gateways were to be independent services (who knows, perhaps a gateway is serving multiple Sofie instances and multiple Studios within each instance!) then I would argue that these are services that are to be configured independently. Perhaps Sofie then would need to be able to embed config pages? Or approaches such as Adaptive Cards could be adopted to allow a gateway to publish a configuration interface to Sofie? Or maybe it would be enough for the gateway to present its own configuration interface, independently of Sofie.

Status

Perhaps Sofie should be able to display device status pulled from external monitoring tools (e.g. Xymon, Graphana, etc.) in order to display status from services. This helps to decouple the service in question from Sofie and would allow Sofie to pull status from more devices and services.

Callbacks

This is where a WebSocket upgrade makes sense, or Webhooks - although Webhooks are not necessarily designed for real-time communication. Perhaps the /ingest/{studioId} route could be upgradeable in order to provide a Studio with place to send callbacks. However, I don't think this needs to be part of this work, nor does it need to be specified at this time as I am of the opinion that these things could be added in a backwards-compatible way in the future. Moreover, much like authentication, I think this is a topic which affects the whole API and is something that should be engineered with the whole HTTP API in mind.
With all that said, the point here is that there is a big difference between something that is "internal" to the Sofie ecosystem (tightly coupled) and something "external" that operates independently with its own lifecycle and release train. As far as I'm concerned, the HTTP API is aimed at "external" services, where DDP - or a DDP replacement - would be aimed at "internal" services. I think it's fine to consider a WebSocket upgrade path for the future, but this cannot be a mirror of the existing DDP interface and needs to be approached problem-first: why is there a desire to move gateways away from DDP? I agree that moving gateways to HTTP would be nice but it cannot be a direct port, as this would not solve the underlying issues of tight coupling and poor API discoverability, in my opinion.

Interchangeable Ids

Do we need to do it like this? As far as I'm aware, the current generic ingest api over ddp solely uses the externalId types and we don't even provide back the actual document ids.
We did this for precisely the same reason, so that the gateway can use the ids as defined in the system it gets the rundowns from.

I agree that this API should only care about external Ids. Something I would like to discuss in the future is how an external system can send Playlist ingest data to Sofie and then activate the Playlist and Take via the Sofie API. This is part of the reason that I included internal Ids in this proposal, but I now realize that those Ids would be for the ingest documents not the Rundowns etc. themselves. Perhaps it would be acceptable to allow other methods of the API to use external Ids? This is a topic for a separate RFC I think, I will remove the internal Ids from this proposal for now.

One major difference I have noticed is that you are wanting to be able to define the playlists over this api. I am not sure if this will work, as this goes against the current model of how playlists are managed in sofie. We currently expect it to be the blueprints who define the playlistExternalId as part of ingesting a rundown.
[...]
Have you thought about whether it will be possible to have these two systems in tandem?
[...]
Do you need to have playlists in the api, or was this added for completeness?

I proposed this structure because Rundowns are owned by Playlists, so it makes sense from a RESTful point of view to place the Rundowns under the Playlist resource. Also, personally I do not like the way that Playlists work as they allow the Rundowns to implicitly declare ownership - the owned resource declares its owner - which feels wrong.
As for whether this could be done in tandem with the existing system, I believe so. We have an existing IngestPlaylist structure that we can use (in fact the Playlist structure in this proposal was designed with that interface in mind) and the existing Playlist logic already looks for whether a Rundown has a Playlist Id when calling the Blueprints to determine the Playlist that a Rundown belongs to, so I think there is a path forward here in terms of having the API pre-define the Playlist Id. We will, however, need to decide if the Blueprints are allowed to override the Playlist that a Rundown belongs to if the Rundown is provided via the API. Doing that in a clean way that does not treat the API in a special way is what needs careful consideration in my eyes.

The REST API so far has been "remote buttons" for the system

I disagree, the effort undertaken in developing the API so far has been done with the goal of making Sofie a component within a broadcast ecosystem, defining intrinsic properties of the system in a way that makes integration with Sofie easier via a well-documented API. Certainly, I would not describe the methods for configuring Studios and ShowStyles as "remote buttons". There may not be much logic contained within the existing API implementation, but the code that does exist is there to ensure that the API exhibits defined behaviors to translate the complex interactions within Sofie into a simpler and more predictable interface.

Ideally, the new HTTP ingest protocol should enable one to create various ingest gateways and combine the "generic" input gateway methods and the MOS-specific input methods into a single, flexible system

As noted above, this proposal doesn't necessarily allow the creation of gateways as we know them today, but certainly has been designed to allow a variety of independent services to publish data to Sofie. I believe there is a path forward - with some of the above issues addressed - that would allow, for example, a MOS ingest service to be created (or the MOS gateway to be reengineered into a standalone service).

We think that this API, considering that it's supposed to be the "ingest" interface, should still maintain little knowledge about Sofie's internals. As such, we're unsure about including the knowledge of "Studios" and "Playlists", or Sofie internal IDs, since these are Sofie internal inventions.

Internal Ids will be removed as noted above.
In my initial draft of this API, the route for ingest was /studios/{studioId}/ingest because Ingest data, at least as far as Sofie is designed today, belongs to a Studio. We are unable to process ingest data without a Studio blueprint and Playlists must be attached to a Studio. I changed it because /ingest is a separate concern to Studios, however, I am not satisfied with the use of Studio Ids under the Ingest resource because they are owned by the Studio resource.
Perhaps we need something like /ingest/{ingestFlowId} (terrible name but we can bikeshed that later) where an ingest flow can be attached to a Studio within Sofie? I'm not sure that I like that, but it does solve problems like Playlist ownership (a flow owns a Playlist, multiple services could be given their own flows and therefore denied access to each other's Playlists). I don't like that it means that the ingest API has to be explicitly enabled, in essence, but if the API was expanded to allow services to register their own flow and attach it to a Studio of their choosing then I think I'd be satisfied with that sort of solution - but now I'm just giving myself more work!
I've addressed Playlists above in response to Julian's comment but to revisit, in my mind Playlists are an intrinsic property of Sofie as a system and have explicit ownership of their Rundowns. It's true that Rundowns can be moved between Playlists but perhaps moving is not the right approach, perhaps a Rundown belongs to a Playlist, the one it was ingested into, but can be mutably borrowed by other Playlists. This does start to get into fundamental architectural decisions of Sofie very quickly if you follow this line of thinking. I would be interested to hear how often Rundowns change which Playlist they belong to in existing deployments, I personally have only experienced configurations where a Rundown decides it belongs to, say, Playlist A and always comes up with the same answer every time it's asked so I have limited knowledge of the requirements here.
To be clear, I don't think that the proposed API in its current form is perfect, but I worry that this conversation so far has not been particularly focused; that the comments on this RFC so far have proposed a lot of changes but have not actually set out problems that need to be solved. I also wish to defend this API against being blindly adapted to work for the existing gateways without consideration for why it is desirable to change the protocol used by the gateways. Perhaps these discussions have in fact been had, but they are not discussions I have been a part of so for me a lot of these comments came out of the blue without context.

authors: jstarpl sbaudlr

repo: nrkno/sofie-core


text:
Check user agent header of our oracle

See neo-project/neo-modules#581. I doubt we have the same problem (we're using standard net/http client), but this needs to be checked and fixed if there is in fact a problem.

Default net/http client doesn't set anything, so yes, we need to add this header too.

repo: nspcc-dev/neo-go


text:
How to pass variables between code blocks with different kernels

Is it possible to pass variables between cells with different interpreters/kernels as in ipython/jupyter?

I might not be familiar with what you are asking. Do you mean connecting another "frontend" (e.g. notebook, qtconsole) to an existing kernel so that you have access to the same variables across more than one frontend?
If so, check out this section of README.md. The example focuses on connecting to a remote kernel, but you can connect to local kernels as well.

Hi @BenRussert, what I mean is the equivalent of this in jupyter noteboks.

Yes, Hydrogen can run the same kernels the Jupyter notebook does:

Sorry I'm coming off as  dumb. Yes, I see that it runs different kernels in different cells. But being able to pass variables between code blocks with different kernels goes beyond that. Is there some documentation on this?
In the example above, how does hydrogen know that in the space between echo "a + b = $(($1 + $2))" and print(c) we've transitioned from the bash kernel to python? If just an empty line, that seems pretty inconvenient and dangerous.
What about when using fenced code blocks as in R markdown? What would be the syntax in that case?

So, I tried running something like what you have above. But when I step through it with shift-enter, it always balks when I land on a line starting a code block with a magic command like %%R. I get
"UsageError: %%R is a cell magic, but the cell body is empty. Did you mean the line magic %R (single %)?"

However, if I select the entire block, it will run properly.
So this confirms exactly what I suspected: The current hydrogen syntax does not currently include a way to implicitly identify code cells. I think this is a serious shortcoming.
I think this syntax is very important.
While I love Python, I think the R-markdown system is really fabulous. I think it would be great if hydrogen would expand on this so that executable code blocks were in fences with metadata to show which language, and which variables were passing in and out.
```(language="python", push=['a','b'])
a=10
b=20
```
```(language="R", pull=['a','b'], push=['a','b','m'])
m = 1/2*(a + b)
```
```(language="Python", pull=['m'])
print("m = " + str(m))
```
It would be nice if there were shortcuts for the code block metadata.
But at the very least, need some syntax to show the start of a code block:
#%%Python -o a b
a=10
b=20

#%%R -i a b -o m
m = 1/2*(a + b)

#%%Python -i m
print("m = " + str(m))

Did you this this in the read me?

Using that method of separating code blocks, how do you specify

Which kernel to use for the code block and?
How to pass variables in and out?

I love what hydrogen is trying to do. But syntax-wise, it is to an extent reinventing the wheel. R-markdown already specifies a way to perform literate programming in a file that can be either run in notebook format or as a script. It is widely used, widely understood and matches the markdown formatting that is virtually ubiquitously use by coders.
However, R-markdown only handles R. What the world needs is a version that can pipeline multiple kernels. I'd love to see hydrogen become that. But right now, there are too many ways to do the same thing, and none of them do this thing completely.

Since I have a bit of free time, I thought I'd write a bit about hydrogen's support of cells and how it has developed over time.
To begin there are two views of how to do literate programming:

Programming language as outermost grammar, annotations inside comments
Text as outermost grammar, code inside fenced blocks

Both of these have their strengths. The programming-language-outermost approach allows running the code as a script with a stock interpreter (this is great for collaboration with people who have a different workflow that doesn't involve hydrogen). On the other hand, the text-outermost approach (e.g. as in R markdown) is good for documentation and exploratory coding.
Hydrogen supports both of these. The run cells is aimed at the first, while the rich-document approach targets the second. The syntax between them is necessarily different. The different syntaxes %%, <codecell>, and In [] are for compatibility with existing tools, such as the "Export" functionality of Jupyter Notebook.
The rich-document functionality was only implemented recently, and it doesn't have any provisions for variable sharing. It's also not very clear how to implement that, since Hydrogen is entirely language-agnostic, and the Jupyter protocols aren't designed for variable sharing across kernels.
So the way to do variable-sharing today is to use the code-cells functionality inside a Python file, and rely on the existing magics that you used in your notebook example. For your example the equivalent file is:
#%%
%load_ext rpy2.ipython

#%%
a = !echo hello
print(a)

#%%
import numpy as np
x = np.array([2.3,-4.5,10.01])
x.mean()

#%%
%%R -i x -o m
m = mean(x)

#%%
print(m)

#%%
a=10
b=20

#%%
%%bash -s $a $b --out c
echo "this is displayed"
echo "the last thing printed to stdout will be passed back in the variable c"
echo "a + b = $(($1+$2))"

#%%
print(c)

The image by @n-riesco didn't use cell separator comments, which I agree is not a fit for everyone in terms of workflow (I believe the lack of separators forces you to select the different-language sections with your mouse and then run only the selected code).

On 19/04/17 01:15, Ariel Balter wrote:

Sorry I'm coming off as dumb. Yes, I see that it runs different kernels in different cells. [...]

Sorry, it wasn't my intention. In fact, the example you link is key to understand your question.
That example only runs the python kernel. All those magic % are run by the kernel python and under the hood import the rpy2 module to run R requests.
As fas as I know, the Jupyter messaging protocol doesn't provide any means for inter-kernel communication. And considering that the current notebook forces by design one kernel per notebook, this is unlikely to change any time soon.
Also note that sharing results between kernels isn't straight forward (different languages have different types; e.g. python has int and float numeric types, but javascript only has a number type).

On 19/04/17 04:20, Ariel Balter wrote:

So, I tried running something like what you have above. But when I step through it with shift-enter, it always balks when I land on a line starting a code block with a magic command like |%%R|. I get
|"UsageError: %%R is a cell magic, but the cell body is empty. Did you mean the line magic %R (single %)?" |
rmagic_empty https://cloud.githubusercontent.com/assets/5349876/25161836/034edc56-2473-11e7-80e5-0e21d130b5bc.png
However, if I select the entire block, it will run properly.
So this confirms exactly what I suspected: The current hydrogen syntax does not currently include a way to implicitly identify code cells. I think this is a serious shortcoming.

Although I didn't define any cells in the screencast I posted, Hydrogen does provide syntax for marking code cells.
See the documentation linked by @BenRussert https://github.com/nteract/hydrogen/issues/723#issuecomment-295059507
And see @nikitakit 's discussion for an example with code cells: https://github.com/nteract/hydrogen/issues/723#issuecomment-295088962

I think this syntax is very important.
While I love Python, I think the R-markdown system is really fabulous. I think it would be great if hydrogen would expand on this so that executable code blocks were in fences with metadata to show which language, and which variables were passing in and out.

I agree with your opinion on markdown, but as I said in my previous email, different languages have different types, and we would need to define an exchange format for sharing variables (json?).

On 19/04/17 05:00, Ariel Balter wrote:

I love what hydrogen is trying to do. But syntax-wise, it is to an extent reinventing the wheel. R-markdown already specifies a way to perform literate programming in a file that can be either run in notebook format or as a script. It is widely used, widely understood and matches the markdown formatting that is virtually ubiquitously use by coders.

I think after you read @nikitakit 's comment https://github.com/nteract/hydrogen/issues/723#issuecomment-295088962
you'll see how unfair the comment above is.
I want add to what @nikitakit said that in my opinion one of the main contributions Hydrogen has made to the Jupyter framework is the convention to mark code cells in cross-language manner (i.e. without any magics %).

authors: BenRussert abalter n-riesco nikitakit

repo: nteract/hydrogen


text:
KernelManager: Fix language detection

Use lowercase in language name comparisons

Ensured that grammarToLanguage(grammar) returns the mapped language.

Fixes #696

Oh! I thought the plan was to use grammar language names instead of kernel language names.

@lgeiger I've noticed that I need to close and open Atom for change in kernelSpec to take effect.
Shall I open an issue?

Sorry, I wasn't able to help earlier, glad you figured it out 👍

I'm chasing another bug; even with this PR { "babel": "javascript"} doesn't work.

You're right the language mappings are confusing, but we can't change the behavior since it would break all existing configs and cause even more confusion 😄
I tried to improve the documentation in my last PR to make it clearer.

I've noticed that I need to close and open Atom for changes in kernelSpec to take effect.

You should be able to run "Hydrogen: Update Kernels" to trigger a update. Is it broken?
If you like, you can chime in on #35 for automatically detecting kernel spec changes.

authors: lgeiger n-riesco


text:
🛑 TCP ping is down

In dda9d75, TCP ping (1.1.1.1) was down:

HTTP code: 403
Response time: 30 ms

Resolved: TCP ping is back up in f7abc0b.


text:
🛑 Difesa is down

In 8544213, Difesa (http://www.difesa.it/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Difesa is back up in f49a803.


text:
🛑 Difesa is down

In d33ec49, Difesa (http://www.difesa.it/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Difesa is back up in 9d80990.


text:
自己创建的广告位id 显示不出来

日志打印 ：D/BannerAdManager( 5831): 媒体在平台上申请的代码位“是否原生”属性与代码中使用的接口不匹配。例如：1. 媒体在平台上选择的是个性化模板banner广告的话，Android代码中请参考BannerExpressActivity进行调用；iOS代码中请参考BUDExpressBannerViewController进行调用。2. 媒体在平台上选择的是个性化模板插屏广告的话，代码中请参考InteractionExpressActivity进行调用；iOS代码位中请参考BUDExpressInterstitialViewController进行调用。
PS：模板广告会带有express字样标识。 40022

我在穿山甲后台，只能创建这种bannerId,没得选其他形式的BannerId

你好，这个广告位还没实现。你那边急吗？

现在穿山甲后台只能创建我这种FeedId，你用的这种怎么创建的，我这边用哪种广告位都行？能教我怎么创建么？是不是穿山甲官方废弃了你那种信息流广告位？------------------ 原始邮件 ------------------
发件人: "Nikola X"<notifications@github.com>
发送时间: 2020年8月13日(星期四) 下午5:52
收件人: "nullptrX/pangle_flutter"<pangle_flutter@noreply.github.com>;
抄送: "wumingman"<1006329824@qq.com>;"Author"<author@noreply.github.com>;
主题: Re: [nullptrX/pangle_flutter] 自己创建的广告位id 显示不出来 (#4)

你创建的不是banner广告位吗，怎么又是信息流了呢？穿山甲sdk加载banner的另一个接口就是你这种广告位，可能每个广告主可以申请的广告位不同吧

新版本已支持模版渲染，再试试。

dependencies:
    pangle_flutter: ^0.1.2

新版本已支持模版渲染，再试试。
dependencies:
    pangle_flutter: ^0.1.2

自己创建的BannerId可以显示出来了，但是FeedId 自己创建的还是显示不出来，发现个新问题，移除BannerView 还是 退出页面 都会报如下错误，但不影响使用

信息流广告也有模版渲染吗，这个还没实现。这个错误应该是个bug，我调试看看。

authors: nullptrX wumingman


text:
docs:updated quickstart

Signed-off-by: Edward Lee edward_lee@intuit.com
Updated quickstart

@edlee2121 - You might have missed following comments.

We need the file docs/README.md (Not ./README.md. It was deleted in your previous PR), that is the homepage of https://numaflow.numaproj.io/;
Also need to make change to https://github.com/numaproj/numaflow/blob/main/mkdocs.yml, which is used to manage the documents menu.

Thanks, restored docs/README and updated mkdocs

repo: numaproj/numaflow


text:
chore: refactor metric variable names

lot of redundancy

No value change right?

No value change right?

correct

authors: vigith whynowy


text:
segfault when allocating arrays in multithreaded program

I wrote a very complex project based on dask. The dask operations are numba kernels interleaved with basic dask functions. My numba kernels all look more or less like this:
@jit(nopython=True, nogil=True, cache=True)
def kernel(input):
    output = numpy.empty_like(input)
    # skip: fill output
    return output

The dask arrays are chunked in a way that the same numba kernel will typically be applied 8 times in parallel (once per thread) to different chunks of the same dask array.
All kernels pass all unit tests, and even the end-to-end dask tree resolves fine as long as you have tiny data involved (e.g. 1 chunk). However, as soon as I move to production-scale data (8+ chunks per dask array and 8 threads), the Python interpreter starts randomly crashing with segmentation faults. Once it also crashed with a double free() error.
Things that make the issue disappear:

Commenting out the @jit decorators
Limiting the number of threads to 1
Rewriting all kernels as follows:

def kernel(input):
    output = numpy.empty_like(input)
    kernel2(input, output)
    return output

@jit(nopython=True, nogil=True, cache=True)
def kernel2(input, output):
    # skip: fill output

Alas I cannot consistently reproduce the bug, and I am not 100% sure yet of fix n.3 above.

Running numba-0.29 and numpy-1.11.2 on Anaconda3 64 bit (latest) on Ubuntu.

Speculating based on similar sounding bugs in the past: This might be a race condition in LLVM compilation in multiple threads when unpickling? @sklam: Ideas how we could check that?

Or it is a threading bug in the memory management in the Numba runtime.

Can you run your workload with the PYTHONFAULTHANDLER environment variable set? This will dump the tracebacks of Python thread on a crash.
Also, please enable core dumps with ulimit -c unlimited. Then you'll be able to use gdb on the core dump file produced by the crash.
The core dump file will probably be saved in /tmp. You can check this by looking at /proc/sys/kernel/core_pattern.

Could this be related to caching?  Also, which dask scheduler is being used?

Hello, sorry for the long silence.
I'm afraid I rewrote all numba functions in cython now - a decision mostly driven by project deadlines.
I still have the old numba logic at hand, so eventually I'll spend some time trying to reproduce the issue again - just I'm afraid I don't have time right now.
@sklam I'm using dask.threaded.get.

I tried to reproduce using a heavily multi-threaded allocation workload, but failed. It may be that the original code is slightly incorrect and writes past array bounds, which could overwrite allocation-related structures in certain cases. Without a reproducer, it is difficult to investigate further.

Antoine,
My code being wrong was of course the first thing I checked. However if I
removed @jit, everything else being identical (including data) , I was not
getting any exceptions such as IndexError.
On 15 Feb 2017 10:55, "Antoine Pitrou" notifications@github.com wrote:

I tried to reproduce using a heavily multi-threaded allocation workload,
but failed. It may be that the original code is slightly incorrect and
writes past array bounds, which could overwrite allocation-related
structures in certain cases. Without a reproducer, it is difficult to
investigate further.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/numba/numba/issues/2213#issuecomment-279980099, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AF7OMDiNmwdj160ufOH2uOY1psH5vkr_ks5rctm8gaJpZM4K_l2I
.

@crusaderky, thanks for the answer. It would be really nice if you could devise some reproducer for us.

We received a similar report for this problem.  We are able to trace it to a problem with race condition during concurrent compilation with caching enabled.   The array allocation is likely just a red herring that increases the compilation time, making it more likely to cause a race condition.

authors: crusaderky pitrou seibert sklam

repo: numba/numba


text:
Cuda runtime errors when starting up numba

[X ] I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).

When this is run in a python script:
from numba.cuda.cudadrv.runtime import runtime
cudart_version_major = runtime.get_version()[0]

With the following installed with conda:

And the following from nvidia-smi:

I receive the following error:

Is this a compatibility issue or something else?
It is not easy for me to update the nvidia drivers (on a shared server). If it is a driver issue, why? What dependency does numba have on the nvidia drivers?

It looks like you have the correct cudatoolkit package installed to go with your NVIDIA driver.  Basically, CUDA requires that the system driver (shown by nvidia-smi) needs to be >= to the CUDA toolkit.  (So cudatoolkits older than the system driver are fine.)
That said, since this is an "unknown" error, it is hard to debug.  Does PyTorch work on the GPU in your environment?

One other question: What GPU do you have on that system?  I doubt it is too old to work with CUDA 10, but just wanted to check.

Understandable that it is difficult to debug. I was running into the same problem, so I wondered if it was a numba thing or my configuration.
Pytorch does work on the GPU. I get:
torch.cuda.is_available()
true
torch.cuda.get_device_name()
'Quadro RTX 6000'

This is nvidia-smi output

Not sure if this is helpful, but when I tried older versions of numba (0.49 and 0.42), I was getting this on a simple example script. Looks like numba.cuda.cudadrv.runtime didn't exist back then (as far as I can tell), so I couldn't test the same thing exactly.

@roshambo919 thanks for submitting this issue. To do some expectation management: our resident CUDA expert @gmarkall is current out-of-office so you may have to wait until January to receive more support on this one.

Actually, @testhound -- is there something you can perhaps do about this one?

Actually, @testhound -- is there something you can perhaps do about this one?

@esc I have no idea what to do on this one. I'll look into as best I can, but I am not optimistic without input from @gmarkall.

@testhound no worries at all sir!

@roshambo919 What's the output of numba -s? (Feel free to redact anything you don't want to share).

Thanks all for the quick replies. Here are some updates on my testing:
Nvidia drivers: 410.78
In the original environment where the problems above occurred.
Env-1:

cudatoolkit=10.0
cudnn=7.6.4 (not sure if this makes a difference)
cudatoolkit-dev=10.1 (not sure if this makes a difference)
numba=0.51.2
pytorch=1.3.1=py3.6_cuda10.0.130_cudnn7.6.3_0

I downgraded cudatoolkit to 9.2 in my environment. The above issue was "fixed" and gave expected outputs, but I ran into a later issue:
Env-2:

cudatoolkit=9.2
cudnn=7.6.4 (not sure if this makes a difference)
cudatoolkit-dev=10.1 (not sure if this makes a difference)
numba=0.51.2
pytorch=1.4.0py3.6_cuda9.2.148_cudnn7.6.3_0

I tried downgrading to 9.0 from there and saw the same problem as with 9.2
Env-3:

cudatoolkit=9.0
cudnn=7.6.4 (not sure if this makes a difference)
cudatoolkit-dev=10.1 (not sure if this makes a difference)
numba=0.51.2
pytorch=1.1.0=py3.6_cuda9.0.176_cudnn7.5.1_0
same error as in Env-2

Output of numba -s for env-2:
System info:
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2020-12-10 12:15:22.163720
UTC start time                                : 2020-12-10 17:15:22.163725
Running time (s)                              : 2.010989

__Hardware Information__
Machine                                       : x86_64
CPU Name                                      : cascadelake
CPU Count                                     : 32
Number of accessible CPUs                     : 32
List of accessible CPUs cores                 : 0-31
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : 64bit adx aes avx avx2 avx512bw
                                                avx512cd avx512dq avx512f avx512vl
                                                avx512vnni bmi bmi2 clflushopt
                                                clwb cmov cx16 cx8 f16c fma
                                                fsgsbase fxsr invpcid lzcnt mmx
                                                movbe pclmul pku popcnt prfchw
                                                rdrnd rdseed sahf sse sse2 sse3
                                                sse4.1 sse4.2 ssse3 xsave xsavec
                                                xsaveopt xsaves

Memory Total (MB)                             : 192059
Memory Available (MB)                         : 144900

__OS Information__
Platform Name                                 : Linux-4.15.0-1066-oem-x86_64-with-debian-buster-sid
Platform Release                              : 4.15.0-1066-oem
OS Name                                       : Linux
OS Version                                    : #76-Ubuntu SMP Fri Dec 6 12:04:50 UTC 2019
OS Specific Version                           : ?
Libc Version                                  : glibc 2.9

__Python Information__
Python Compiler                               : GCC 7.3.0
Python Implementation                         : CPython
Python Version                                : 3.6.12
Python Locale                                 : en_US.UTF-8

__LLVM Information__
LLVM Version                                  : 10.0.1

__CUDA Information__
CUDA Device Initialized                       : True
CUDA Driver Version                           : 10000
CUDA Detect Output:
Found 2 CUDA devices
id 0      b'Quadro RTX 6000'                              [SUPPORTED]
                      compute capability: 7.5
                           pci device id: 0
                              pci bus id: 115
id 1      b'Quadro RTX 6000'                              [SUPPORTED]
                      compute capability: 7.5
                           pci device id: 0
                              pci bus id: 166
Summary:
	2/2 devices are supported

CUDA Librairies Test Output:
Finding cublas from Conda environment
	named  libcublas.so.9.2.148
	trying to open library...	ok
Finding cusparse from Conda environment
	named  libcusparse.so.9.2.148
	trying to open library...	ok
Finding cufft from Conda environment
	named  libcufft.so.9.2.148
	trying to open library...	ok
Finding curand from Conda environment
	named  libcurand.so.9.2.148
	trying to open library...	ok
Finding nvvm from Conda environment
	named  libnvvm.so.3.3.0
	trying to open library...	ok
Finding cudart from Conda environment
	named  libcudart.so.9.2.148
	trying to open library...	ok
Finding libdevice from Conda environment
	searching for compute_20...	ok
	searching for compute_30...	ok
	searching for compute_35...	ok
	searching for compute_50...	ok

__ROC information__
ROC Available                                 : False
ROC Toolchains                                : None
HSA Agents Count                              : 0
HSA Agents:
None
HSA Discrete GPUs Count                       : 0
HSA Discrete GPUs                             : None

__SVML Information__
SVML State, config.USING_SVML                 : False
SVML Library Loaded                           : False
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : False

__Threading Layer Information__
TBB Threading Layer Available                 : True
+-->TBB imported successfully.
OpenMP Threading Layer Available              : True
+-->Vendor: GNU
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
None found.

__Conda Information__
Conda Build                                   : not installed
Conda Env                                     : 4.9.2
Conda Platform                                : linux-64
Conda Python Version                          : 3.6.12.final.0
Conda Root Writable                           : True

__Installed Packages__
_libgcc_mutex             0.1                        main
blas                      1.0                         mkl
blosc                     1.20.1               hd408876_0
brotli                    1.0.9                he6710b0_2
brotlipy                  0.7.0           py36h27cfd23_1003
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2020.12.8            h06a4308_0
certifi                   2020.12.5        py36h06a4308_0
cffi                      1.14.4           py36h261ae71_0
chardet                   3.0.4           py36h06a4308_1003
charls                    2.1.0                he6710b0_2
cloudpickle               1.6.0                      py_0
cmake                     3.18.2               ha30ef3c_0
conda                     4.9.2            py36h06a4308_0
conda-package-handling    1.7.2            py36h03888b9_0
cryptography              3.3              py36h3c74f83_0
cudatoolkit               9.2                           0
cudatoolkit-dev           10.1.243             h516909a_3    conda-forge
cudnn                     7.6.4                 cuda9.2_0
cycler                    0.10.0                   py36_0
cytoolz                   0.11.0           py36h7b6447c_0
dask-core                 2.30.0                     py_0
dbus                      1.13.18              hb2f20db_0
decorator                 4.4.2                      py_0
easydict                  1.9                        py_0    conda-forge
expat                     2.2.10               he6710b0_2
fire                      0.3.1              pyh9f0ad1d_0    conda-forge
fontconfig                2.13.0               h9420a91_0
freetype                  2.10.4               h5ab3b9f_0
giflib                    5.1.4                h14c3975_1
glib                      2.66.1               h92f7085_0
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb31296c_0
icu                       58.2                 he6710b0_3
idna                      2.10                       py_0
imagecodecs               2020.5.30        py36hfa7d478_2
imageio                   2.9.0                      py_0
intel-openmp              2020.2                      254
iou3d                     0.0.0                    pypi_0    pypi
jpeg                      9b                   h024ee3a_2
jxrlib                    1.1                  h7b6447c_2
kiwisolver                1.3.0            py36h2531618_0
krb5                      1.18.2               h173b8e3_0
lcms2                     2.11                 h396b838_0
ld_impl_linux-64          2.33.1               h53a641e_7
libaec                    1.0.4                he6710b0_1
libcurl                   7.71.1               h20c2e04_1
libedit                   3.1.20191231         h14c3975_1
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libllvm10                 10.0.1               hbcb73fb_5
libpng                    1.6.37               hbc83047_0
libssh2                   1.9.0                h1ba5d50_1
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.1.0                h2733197_1
libuuid                   1.0.3                h1bed415_2
libuv                     1.40.0               h7b6447c_0
libwebp                   1.0.1                h8e7db2f_0
libxcb                    1.14                 h7b6447c_0
libxml2                   2.9.10               hb55368b_3
libzopfli                 1.0.3                he6710b0_0
llvmlite                  0.34.0           py36h269e1b5_4
lz4-c                     1.9.2                heb0550a_3
matplotlib                3.3.2                         0
matplotlib-base           3.3.2            py36h817c723_0
mkl                       2020.2                      256
mkl-service               2.3.0            py36he8ac12f_0
mkl_fft                   1.2.0            py36h23d657b_0
mkl_random                1.1.1            py36h0573a6f_0
ncurses                   6.2                  he6710b0_1
networkx                  2.5                        py_0
ninja                     1.10.2           py36hff7bd54_0
numba                     0.51.2           py36h0573a6f_1
numpy                     1.19.2           py36h54aff64_0
numpy-base                1.19.2           py36hfa32c7d_0
olefile                   0.46                     py36_0
openjpeg                  2.3.0                h05c96fa_1
openssl                   1.1.1i               h27cfd23_0
pcre                      8.44                 he6710b0_0
pillow                    8.0.1            py36he98fc37_0
pip                       20.3.1           py36h06a4308_0
pointnet2                 0.0.0                    pypi_0    pypi
pycosat                   0.6.3            py36h27cfd23_0
pycparser                 2.20                       py_2
pyopenssl                 20.0.0             pyhd3eb1b0_1
pyparsing                 2.4.7                      py_0
pyqt                      5.9.2            py36h05f1152_2
pysocks                   1.7.1            py36h06a4308_0
python                    3.6.12               hcff3b4d_2
python-dateutil           2.8.1                      py_0
pytorch                   1.4.0           py3.6_cuda9.2.148_cudnn7.6.3_0    pytorch
pywavelets                1.1.1            py36h7b6447c_2
pyyaml                    5.3.1            py36h7b6447c_1
qt                        5.9.7                h5867ecd_1
readline                  8.0                  h7b6447c_0
requests                  2.25.0             pyhd3eb1b0_0
rhash                     1.4.0                h1ba5d50_0
roipool3d                 0.0.0                    pypi_0    pypi
ruamel_yaml               0.15.87          py36h7b6447c_1
scikit-image              0.17.2           py36hdf5156a_0
scipy                     1.5.2            py36h0b6359f_0
setuptools                51.0.0           py36h06a4308_2
sip                       4.19.8           py36hf484d3e_0
six                       1.15.0           py36h06a4308_0
snappy                    1.1.8                he6710b0_0
sqlite                    3.33.0               h62c20be_0
tbb                       2020.3               hfd86e86_0
termcolor                 1.1.0                    py36_1
tifffile                  2020.12.4          pyhd3eb1b0_0
tk                        8.6.10               hbc83047_0
toolz                     0.11.1                     py_0
torchvision               0.5.0                 py36_cu92    pytorch
tornado                   6.1              py36h27cfd23_0
tqdm                      4.54.1             pyhd3eb1b0_0
urllib3                   1.25.11                    py_0
wheel                     0.36.1             pyhd3eb1b0_0
xz                        5.2.5                h7b6447c_0
yaml                      0.2.5                h7b6447c_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.5                h9ceee32_0

No errors reported.

__Warning log__
Warning (roc): Error initialising ROC: No ROC toolchains found.
Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init:
NUMBA_HSA_DRIVER /opt/rocm/lib/libhsa-runtime64.so is not a valid file path.  Note it must be a filepath of the .so/.dll/.dylib or the driver:
Warning (psutil): psutil cannot be imported. For more accuracy, consider installing it.

PTX ISA version 6.3 is for CUDA toolkit 10.0. To get rid of that error you'll have to go back to 10.0. Here's a table I put together a while ago with the driver/toolkit versions. I haven't updated it with 11.0 yet.
As for the runtime error, if you directly import the runtime like that, has Numba initialized CUDA/the context yet?
PTX ISA 5.0
	Toolkit: 8.0
	Driver: 361.XX
PTX ISA 6.0
	Toolkit: 9.0
	Driver: 384.XX
	PTX ISA 6.1
	Toolkit: 9.1
	Driver: 387.XX
PTX ISA 6.2
	Toolkit: 9.2
	Driver: 396.XX
PTX ISA 6.3
	Toolkit: 10.0
	Driver: 400.XX
PTX ISA 6.4
	Toolkit: 10.1
	Driver: 418.XX
PTX ISA 6.5
	Toolkit: 10.2
	Driver: 440.XX

Hi all - wanted to return to this. Anything else I should try to get this working?

Hi all - wanted to return to this. Anything else I should try to get this working?

Hi @roshambo919 - My suspicion is that having both cudatoolkit and cudatoolkit-dev is causing a problem - I'm wondering if something from cudatoolkit-dev 10.1 is getting picked up and causing the issues you're seeing.
If you try with environments that don't have cudatoolkit-dev installed, do you still observe issues?

Hi @roshambo919 - My suspicion is that having both cudatoolkit and cudatoolkit-dev is causing a problem - I'm wondering if something from cudatoolkit-dev 10.1 is getting picked up and causing the issues you're seeing.
If you try with environments that don't have cudatoolkit-dev installed, do you still observe issues?

Thanks all for the replies. That would be a good debug step and it'd be interesting to see what happens, but we just had the drivers updated on the machine, and now the same environment is working like a charm.

Glad to hear it's now working! If there are no outstanding problems, please do close the issue :-)

authors: esc gmarkall roshambo919 seibert stuartarchibald testhound uchytilc


text:
Literal dispatch

Add literally(x) to force the dispatcher to pass in the value as a Literal type.
Works even when nested inside function calls.  The type inference will propagate the information up.

Add literally(x) to force the dispatcher to pass in the value as a Literal type.

This seems very cool. Although the tests don't demonstrate it, does the above imply that the following will be possible?
@njit
def foo(x):
    return x + 1

foo(literally(1))

@sjperkins, your example doesn't work currently, but we decided to add it into this PR.

Many thanks!

Getting literally(123) to work in pure-python code to mark value for literal typing seems to require making custom subclasses of int.  Right now with https://github.com/numba/numba/pull/4467/commits/e69460a5b50e226c77f390241e95f59d94ed3c21, literally(123) would return a IntegerLiteral[int](123) numba type object, but it does not support any arithmetic like an normal int would.

Getting literally(123) to work in pure-python code to mark value for literal typing seems to require making custom subclasses of int. Right now with e69460a, literally(123) would return a IntegerLiteral[int](123) numba type object, but it does not support any arithmetic like an normal int would.

I see the primary usefulness of literally in python mode as enabling dispatch on value from python in generated_jit. For example:
@generated_jit
def fn(a, mode):

  if mode.literal_value == "bob":
     def impl(a, mode):
         return a + 1
  elif mode.literal_value == "fred":
     def impl(a, mode):
         return a + 2
  else:
     def impl(a, mode):
         return a * 10

  return impl

assert fn(10, literally("bob")) == 11
assert fn(12, literally("fred")) == 12

This could be made more complex with tuples for example:
fn(10, literally(("option-1", 1, "option-2", 2)))

However, I would guess that it would be difficult to support the above and conversion of (somewhat arbitrary) literals into numba type within the nopython layer.
Would it not be simply useful to support the first case for python mode? If the literal is not directly convertible to a numba type I think it'll just end up as a (safely ignorable) pythonobj?

The trouble I am seeing is when code like:
y = literally("hello")
z = y + " world!"

is used in pure python code.  That can occur when @jit is disabled for debugging.
I need to somehow mark the return value from literally() to be typed "literally".  The only way I can imagine is to subclass builtin python types.  But, that limits what can use with literally().

Self note
TODOs:

[ ] explore usecase with @overload.

I am stuck trying to get literally(obj) to work in pure-python mode.  I cannot make it work in this PR.  For now, literally(x) is an identity function in pure-python code (i.e. literally = lambda x: x).
Details about the problem:
I can't make literally(obj) behave the same in numba and in pure-python code.  In numba, it is an identity function that signals the dispatcher about literal types.  In pure-python code, we wanted to create an illusion that it is an identity function but the dispatcher receiving its output will know to use type the value as a literal.  The trouble is in the latter.  I can't find a way to emulate it as an identity function while providing the literal annotation.
Approaches I tried:

literal(x) returns Literal(x) type. Problem: can't operate of it like a normal x.
literal(x) returns an instance of a subclass of type(x) such that the subclass has extra attribute for annotation.  Problem: some types (i.e. slice) cannot be subclassed.

@sklam FYI, none of the use cases I have in mind need such capability.

@sklam I just tested this and it works great!
Following up on our conversation at the core meeting: I was somehow under the impression that this will help with caching functions that have function (Dispatcher) arguments by making them literals, but Dispatcher type is sort of a literal already. The problem is pickling Dispatchers for caching...

@ehsantn, can you remind me of the pickling Dispatcher problem?

I think it tries to pickle the Dispatcher type as part of the cache index and complains about "weak reference".

@numba.njit(cache=True)
def f(g):
    # g = numba.literally(g)
    g()

@numba.njit
def bar():
    print(4)

f(bar)

Traceback (most recent call last):
  File "/Users/ehsan/dev/numba/numba/caching.py", line 482, in save
    data_name = overloads[key]
KeyError: ((type(CPUDispatcher(<function bar at 0x10baba050>)),), ('x86_64-apple-darwin18.7.0', 'skylake', '+64bit,+adx,+aes,+avx,+avx2,-avx512bitalg,-avx512bw,-avx512cd,-avx512dq,-avx512er,-avx512f,-avx512ifma,-avx512pf,-avx512vbmi,-avx512vbmi2,-avx512vl,-avx512vnni,-avx512vpopcntdq,+bmi,+bmi2,-cldemote,+clflushopt,-clwb,-clzero,+cmov,+cx16,+f16c,+fma,-fma4,+fsgsbase,-gfni,+invpcid,-lwp,+lzcnt,+mmx,+movbe,-movdir64b,-movdiri,-mwaitx,+pclmul,-pconfig,-pku,+popcnt,-prefetchwt1,+prfchw,-ptwrite,-rdpid,+rdrnd,+rdseed,+rtm,+sahf,+sgx,-sha,-shstk,+sse,+sse2,+sse3,+sse4.1,+sse4.2,-sse4a,+ssse3,-tbm,-vaes,-vpclmulqdq,-waitpkg,-wbnoinvd,-xop,+xsave,+xsavec,+xsaveopt,+xsaves'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../test_dispatch.py", line 14, in <module>
    f(bar)
  File "/Users/ehsan/dev/numba/numba/dispatcher.py", line 395, in _compile_for_args
    raise e
  File "/Users/ehsan/dev/numba/numba/dispatcher.py", line 352, in _compile_for_args
    return self.compile(tuple(argtypes))
  File "/Users/ehsan/dev/numba/numba/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/Users/ehsan/dev/numba/numba/dispatcher.py", line 744, in compile
    self._cache.save_overload(sig, cres)
  File "/Users/ehsan/dev/numba/numba/caching.py", line 665, in save_overload
    self._save_overload(sig, data)
  File "/Users/ehsan/dev/numba/numba/caching.py", line 675, in _save_overload
    self._cache_file.save(key, data)
  File "/Users/ehsan/dev/numba/numba/caching.py", line 491, in save
    self._save_index(overloads)
  File "/Users/ehsan/dev/numba/numba/caching.py", line 537, in _save_index
    data = self._dump(data)
  File "/Users/ehsan/dev/numba/numba/caching.py", line 565, in _dump
    return pickle.dumps(obj, protocol=-1)
TypeError: can't pickle weakref objects

i see. it is the same error as in https://github.com/numba/numba/issues/4534

I think i have addressed all the review comments

That should be it

authors: ehsantn sjperkins sklam


text:
Add codemirror addon to enable search and replace

Fixes #129.

:star2:

authors: christianp extigy

repo: numbas/editor


text:
How to configure client endpoint_url?

Hi,
I'm looking to use DynamoDB local with bloop and was wondering what would be the best way to have bloop configure the boto3 client with a new endpoint_url?
Thanks for the help!

Thanks for the question!  I want to add support for DynamoDB local (really, any custom boto3 client) but hadn't prioritized it yet.
From this SO post I think it would be easiest if I made bloop.Client take a boto3 client, and then it's the usual pattern to construct an engine.
I can get a revision up in a few hours.  How do you feel about this configuration?
import boto3
import bloop

# Talk to DynamoDB Local
boto_client = ddb = boto3.client("dynamodb", endpoint_url="http://localhost:8000")
bloop_client = bloop.Client(boto_client=boto_client)
engine = bloop.Engine(client=bloop_client)

[ ] bloop.Client takes an optional boto_client
[ ] bloop.Engine takes an optional bloop_client
[ ] bloop.Engine.bind constructs the default boto_client and bloop_client if it doesn't already have one

For an immediate workaround, you can use the following (it's not pretty, and it will try to make a default boto3 client first, so you'll need valid credentials there)
import boto3
import bloop
from bloop.client import Client

# Talk to DynamoDB Local
boto_client = boto3.client("dynamodb", endpoint_url="http://localhost:8000")

# bloop.client.Client creates a boto3 client on __init__, unfortunately.
bloop_client = Client()
# Replace the default client with the local client above
bloop_client.client = boto_client

engine = bloop.Engine()
engine.client = bloop_client

Alright, I think that should do it.  I want to mull it over a bit, and get some examples in the docs first.
For now feel free to check out the client-refactor branch and you should be able to use the first sample:
import boto3
import bloop

# Talk to DynamoDB Local
boto_client = ddb = boto3.client("dynamodb", endpoint_url="http://localhost:8000")
bloop_client = bloop.Client(boto_client=boto_client)
engine = bloop.Engine(client=bloop_client)

Awesome, thank you! Will take this for a test drive.

Sorry, didn't mean to close this without your feedback.  I'm uploading 0.9.11 to pypi now, can you update (pip install -U bloop) and give it a try?
Thanks!

The DynamoDB Local sample at docs/patterns is effectively identical to the proposed code at the top of this thread.

Fantastic. I'll update and give it try now.

Thank you for quick turn around on this! For what is worth, I like the implementation. Had some thoughts while testing out 0.9.11. Some might be best suited for another issue, but just wanted to share:

What do you think about updating the patterns doc to include  the region_name kwarg when creating a new boto3 client? Without it, botocore raises a NoRegionError exception.

The version for DynamoDB local i'm using, includes the LastIncreaseDateTime and LastDecreaseDateTime keys in the ProvisionedThroughput dict, causing a TableMismatch exception to be raised when the table is validated here. Maybe those keys can be added to _table_for_model or added to the junk_index_fields black list in describe_table? Heres the full exception message:

"Existing table for model <class 'frontend.views.user_patient.create_patient.<locals>.UserPatient'> does not match expected",
{
  'TableName': 'PatientUser',
  'KeySchema': [{'KeyType': 'HASH', 'AttributeName': 'id'}],
  'ProvisionedThroughput': {'ReadCapacityUnits': 5, 'WriteCapacityUnits': 10},
  'AttributeDefinitions': [{'AttributeName': 'id', 'AttributeType': 'S'}]
},
{
  'TableName': 'PatientUser',
  'KeySchema': [{'KeyType': 'HASH', 'AttributeName': 'id'}],
  'ProvisionedThroughput': {
    'ReadCapacityUnits': 1,
    'WriteCapacityUnits': 1,
    'LastIncreaseDateTime': datetime.datetime(1970, 1, 1, 0, 0, tzinfo=tzlocal()),
    'LastDecreaseDateTime': datetime.datetime(1970, 1, 1, 0, 0, tzinfo=tzlocal())},
    'AttributeDefinitions': [{'AttributeName': 'id', 'AttributeType': 'S'}]
}

It's most likely i'm doing something completely wrong, so here is a gist of the code i'm currently testing. See any glaring issues?
Happy to help with PRs. Thanks again :)

The gist looks fine; I'd recommend setting the region in your aws config so boto3 picks it up automatically.

I'll add a region to the docs; you can also fix that by specifying a default region in your aws config.  Boto3 refuses to pick a default region for you, so if there's not one in your config's profile or provided in kwargs, it raises.
[profile gaas-integ]
region = us-west-2
aws_access_key_id = ...
aws_secret_access_key = ...

Entirely agreed, I'm opening another issue.  I was pretty sure I got rid of those in a previous pass, but I guess they're not returned by DynamoDB, since tables are verifying correctly for me.

Thanks for the feedback; your descriptions are thorough enough to make fixes easy :guitar:

Just released 0.9.12, which handles extra fields in the response from DynamoDB.  Here's the test that ensures extra fields like "LastDecreaseDateTime" and "LastIncreaseDateTime" are dropped before comparing against the expected table.
If you don't mind, please give 0.9.12 a try and re-open #39 if you're still getting TableMismatch.

0.9.12 is working like a charm! I really appreciate you taking the time for this. Thanks again.

authors: jasonlockporcelane numberoverzero

repo: numberoverzero/bloop