text:
[0.37.x] Webview Crashes with IPC message

Hello,
First of all Thanks for the awesome work on electron, while trying to upgrade our app, we found webview is crashing, upon digging our logging mechanism which logs all object properties going into prototype chain was causing webview to crash.  we have given minimal code to reproduce the effect.
0.37.x appends a  key with each ipc message. when we encounter that key webview crashes.
Sample app can be found in the below git url, working fine in < 0.37.x
https://github.com/kamesh-a/electron-samples/tree/master/0.37.2_webview_crash

Electron version: 0.37.x
Operating system: Mac and Windows

oops missed onething, After the google page loads fully, hit  "Reproduce crash" button.

Here is the stack trace I got with that example you linked:
Process:               Electron Helper [82007]
Path:                  /Users/USER/*/Electron.app/Contents/Frameworks/Electron Helper.app/Contents/MacOS/Electron Helper
Identifier:            com.github.electron.helper
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        Electron [82002]
Responsible:           Terminal [721]
User ID:               501

Date/Time:             2016-03-21 15:39:26.865 -0700
OS Version:            Mac OS X 10.10.5 (14F1605)
Report Version:        11
Anonymous UUID:        9E9CB434-A4C0-CA4D-716E-2DA7D958B49C

Sleep/Wake UUID:       EF28841F-E6F2-4AA1-A185-22E3A7CE2CE8

Time Awake Since Boot: 180000 seconds
Time Since Wake:       3100 seconds

Crashed Thread:        0  CrRendererMain  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes:       0x0000000000000001, 0x0000000000000000

Thread 0 Crashed:: CrRendererMain  Dispatch queue: com.apple.main-thread
0   libv8.dylib                   	0x000000010daeed72 v8::base::OS::Abort() + 18
1   libv8.dylib                   	0x000000010daed2c1 V8_Fatal + 225
2   libv8.dylib                   	0x000000010d576ce3 v8::internal::ToBooleanStub::Types::UpdateStatus(v8::internal::Handle<v8::internal::Object>) + 419
3   libv8.dylib                   	0x000000010d576afa v8::internal::ToBooleanStub::UpdateStatus(v8::internal::Handle<v8::internal::Object>) + 42
4   libv8.dylib                   	0x000000010d83e1f6 v8::internal::Runtime_ToBooleanIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) + 246
5   ???                           	0x000006062100c1fb 0 + 6623393268219
6   ???                           	0x000006062101b9f2 0 + 6623393331698
7   ???                           	0x00000606214683c1 0 + 6623397839809
8   ???                           	0x0000060621468078 0 + 6623397838968
9   ???                           	0x0000060621036e64 0 + 6623393443428
10  ???                           	0x0000060621019a62 0 + 6623393323618
11  libv8.dylib                   	0x000000010d79a2b0 v8::internal::Execution::ToObject(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>) + 592
12  libv8.dylib                   	0x000000010d799fa8 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 456
13  libv8.dylib                   	0x000000010d50613c v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 300
14  libwebcore_shared.dylib       	0x00000001131abc40 blink::V8ScriptRunner::callFunction(v8::Local<v8::Function>, blink::ExecutionContext*, v8::Local<v8::Value>, int, v8::Local<v8::Value>*, v8::Isolate*) + 864
15  libwebcore_shared.dylib       	0x00000001131763f0 blink::ScriptController::callFunction(v8::Local<v8::Function>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 96
16  libwebcore_shared.dylib       	0x000000011319b499 0x113150000 + 308377
17  libwebcore_shared.dylib       	0x00000001131932d8 blink::V8AbstractEventListener::invokeEventHandler(blink::ScriptState*, blink::Event*, v8::Local<v8::Value>) + 248
18  libwebcore_shared.dylib       	0x0000000113193197 blink::V8AbstractEventListener::handleEvent(blink::ScriptState*, blink::Event*) + 199
19  libwebcore_shared.dylib       	0x00000001131930ab blink::V8AbstractEventListener::handleEvent(blink::ExecutionContext*, blink::Event*) + 171
20  libwebcore_shared.dylib       	0x0000000113a8ab37 blink::EventTarget::fireEventListeners(blink::Event*, blink::EventTargetData*, WTF::Vector<blink::RegisteredEventListener, 1ul, WTF::PartitionAllocator>&) + 1111
21  libwebcore_shared.dylib       	0x0000000113a8a49a blink::EventTarget::fireEventListeners(blink::Event*) + 282
22  libwebcore_shared.dylib       	0x0000000113a8a364 blink::EventTarget::dispatchEventInternal(WTF::PassRefPtr<blink::Event>) + 132
23  libwebcore_shared.dylib       	0x0000000113a8a250 blink::EventTarget::dispatchEventForBindings(WTF::PassRefPtr<blink::Event>, blink::ExceptionState&) + 256
24  libwebcore_shared.dylib       	0x00000001133642d7 0x113150000 + 2179799
25  libv8.dylib                   	0x000000010d51fca3 v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) + 163
26  libv8.dylib                   	0x000000010d556c3e v8::internal::Builtins::InvokeApiFunction(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 1534
27  libv8.dylib                   	0x000000010d5655eb v8::internal::Builtins::~Builtins() + 59243
28  ???                           	0x000006062100c1fb 0 + 6623393268219
29  ???                           	0x0000060621467c2c 0 + 6623397837868
30  ???                           	0x0000060621467890 0 + 6623397836944
31  ???                           	0x000006062117327e 0 + 6623394738814
32  ???                           	0x000006062100f6b7 0 + 6623393281719
33  ???                           	0x0000060621467694 0 + 6623397836436
34  ???                           	0x0000060621036e64 0 + 6623393443428
35  ???                           	0x0000060621019a62 0 + 6623393323618
36  libv8.dylib                   	0x000000010d79a2b0 v8::internal::Execution::ToObject(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>) + 592
37  libv8.dylib                   	0x000000010d799fa8 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) + 456
38  libv8.dylib                   	0x000000010d50613c v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 300
39  libnode.dylib                 	0x000000010cb122f7 node::MakeCallback(node::Environment*, v8::Local<v8::Value>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*) + 1239 (node.cc:1151)
40  libnode.dylib                 	0x000000010cb1274e node::MakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::String>, int, v8::Local<v8::Value>*) + 174 (node.cc:1201)
41  libnode.dylib                 	0x000000010cb127c1 node::MakeCallback(node::Environment*, v8::Local<v8::Object>, char const*, int, v8::Local<v8::Value>*) + 97 (node.cc:1211)
42  libnode.dylib                 	0x000000010cb12871 node::MakeCallback(v8::Isolate*, v8::Local<v8::Object>, char const*, int, v8::Local<v8::Value>*) + 161 (node.cc:1225)
43  com.github.electron.framework 	0x000000010bca5ada mate::internal::CallEmitWithArgs(v8::Isolate*, v8::Local<v8::Object>, std::__1::vector<v8::Local<v8::Value>, std::__1::allocator<v8::Local<v8::Value> > >*) + 170 (event_emitter_caller.cc:25)
44  com.github.electron.framework 	0x000000010bd0a5b1 v8::Local<v8::Value> mate::EmitEvent<std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > >(v8::Isolate*, v8::Local<v8::Object>, std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, std::__1::vector<v8::Local<v8::Value>, std::__1::allocator<v8::Local<v8::Value> > > const&) + 1153 (event_emitter_caller.h:34)
45  com.github.electron.framework 	0x000000010bd094ed atom::AtomRenderViewObserver::OnBrowserMessage(std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, base::ListValue const&) + 781 (atom_render_view_observer.cc:147)
46  com.github.electron.framework 	0x000000010bd0c089 void base::DispatchToMethodImpl<atom::AtomRenderViewObserver, void (atom::AtomRenderViewObserver::*)(std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, base::ListValue const&), std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> >, base::ListValue, 0ul, 1ul>(atom::AtomRenderViewObserver*, void (atom::AtomRenderViewObserver::*)(std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, base::ListValue const&), base::Tuple<std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> >, base::ListValue> const&, base::IndexSequence<0ul, 1ul>) + 185 (tuple.h:253)
47  com.github.electron.framework 	0x000000010bd0bfc5 void base::DispatchToMethod<atom::AtomRenderViewObserver, void (atom::AtomRenderViewObserver::*)(std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, base::ListValue const&), std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> >, base::ListValue>(atom::AtomRenderViewObserver*, void (atom::AtomRenderViewObserver::*)(std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, base::ListValue const&), base::Tuple<std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> >, base::ListValue> const&) + 69 (tuple.h:260)
48  com.github.electron.framework 	0x000000010bd09a0f bool AtomViewMsg_Message::Dispatch<atom::AtomRenderViewObserver, atom::AtomRenderViewObserver, void, void (atom::AtomRenderViewObserver::*)(std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, base::ListValue const&)>(IPC::Message const*, atom::AtomRenderViewObserver*, atom::AtomRenderViewObserver*, void*, void (atom::AtomRenderViewObserver::*)(std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, base::ListValue const&)) + 143 (api_messages.h:33)
49  com.github.electron.framework 	0x000000010bd091a6 atom::AtomRenderViewObserver::OnMessageReceived(IPC::Message const&) + 214 (atom_render_view_observer.cc:115)
50  libcontent.dylib              	0x000000010fe56165 content::RenderViewImpl::OnMessageReceived(IPC::Message const&) + 693
51  libcontent.dylib              	0x000000010fd8ab4c content::MessageRouter::RouteMessage(IPC::Message const&) + 140
52  libcontent.dylib              	0x000000010fd8aabc content::MessageRouter::OnMessageReceived(IPC::Message const&) + 28
53  libcontent.dylib              	0x000000010fc02a0f content::ChildThreadImpl::OnMessageReceived(IPC::Message const&) + 1055
54  libipc.dylib                  	0x00000001117750b8 IPC::ChannelProxy::Context::OnDispatchMessage(IPC::Message const&) + 120
55  libbase.dylib                 	0x000000010df4d93a base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) + 202
56  libscheduler.dylib            	0x0000000112abf330 scheduler::TaskQueueManager::ProcessTaskFromWorkQueue(scheduler::internal::WorkQueue*, scheduler::internal::TaskQueueImpl::Task*) + 688
57  libscheduler.dylib            	0x0000000112abe15e scheduler::TaskQueueManager::DoWork(base::TimeTicks, bool) + 286
58  libbase.dylib                 	0x000000010df4d93a base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) + 202
59  libbase.dylib                 	0x000000010df771c9 base::MessageLoop::RunTask(base::PendingTask const&) + 489
60  libbase.dylib                 	0x000000010df77818 base::MessageLoop::DoWork() + 632
61  libbase.dylib                 	0x000000010df3ecf1 0x10df3c000 + 11505
62  libbase.dylib                 	0x000000010df6aa5a base::mac::CallWithEHFrame(void () block_pointer) + 10
63  libbase.dylib                 	0x000000010df3e6f4 0x10df3c000 + 9972
64  com.apple.CoreFoundation      	0x00007fff95dc5a01 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
65  com.apple.CoreFoundation      	0x00007fff95db7b8d __CFRunLoopDoSources0 + 269
66  com.apple.CoreFoundation      	0x00007fff95db71bf __CFRunLoopRun + 927
67  com.apple.CoreFoundation      	0x00007fff95db6bd8 CFRunLoopRunSpecific + 296
68  com.apple.Foundation          	0x00007fff9043fb29 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 278
69  libbase.dylib                 	0x000000010df3f46e base::MessagePumpNSRunLoop::DoRun(base::MessagePump::Delegate*) + 126
70  libbase.dylib                 	0x000000010df3eb44 0x10df3c000 + 11076
71  libbase.dylib                 	0x000000010df8f1e3 base::RunLoop::Run() + 99
72  libbase.dylib                 	0x000000010df768fd base::MessageLoop::Run() + 29
73  libcontent.dylib              	0x000000010fe75397 0x10f824000 + 6624151
74  libcontent.dylib              	0x000000010f8291dc 0x10f824000 + 20956
75  libcontent.dylib              	0x000000010f8286d6 content::ContentMain(content::ContentMainParams const&) + 54
76  com.github.electron.framework 	0x000000010bac6af8 AtomMain + 104 (atom_library_main.mm:24)
77  com.github.electron.helper    	0x000000010babce08 main + 72
78  libdyld.dylib                 	0x00

Related to https://github.com/atom/electron/issues/4802#issuecomment-199588369

@deepak1556 and @zcbenz  thank you for fixing it. we have tested it works fine.

authors: deepak1556 kamesh-a kevinsawicki


text:
Update to Squirrel 0.9.0

Some notable points about this release:

Fixes orphaned processes hanging around on version obsolete (not sure if Atom hit this)
Infinite fixes around pinning shortcuts to the Taskbar
Work around infinite temp files being created which are currently breaking installs for some people
If the .NET Framework requires a reboot, Setup will now resume itself once the machine restarts
If users manually right-click on the app and "Run as Administrator", we'll now de-administer ourselves and run Setup, which should fix some permission problems.

Thanks @paulcbetts, published as 0.24.0

There's a race that Atom Shell apps probably won't hit, but I'm gonna publish a release for it anyways, will hit you up today

There's a race that Atom Shell apps probably won't hit

Is there an issue you could link me to about this?

https://github.com/Squirrel/Squirrel.Windows/issues/257

source: gharchive/pull-request

authors: kevinsawicki paulcbetts

repo: atom/grunt-atom-shell-installer

license: MIT

license_type: permissive

license_source: github-api


text:
download fails because repository changed atom/atom-shell to atom/electon

download fails because repository changed atom/atom-shell to atom/electon, maybe.
$ grunt download-atom-shell
Running "download-atom-shell" task
>> Cannot find atom-shell v0.23.0 from GitHub [Error: Not Found]
Warning: Task "download-atom-shell" failed. Use --force to continue.

I think L129 and L190 of tasks/download-atom-shell-task.coffee need to be updated to reflect the new name.

This is being handled in #23. Thanks for the helpful report :bow:

Another solution in #24, in a way that allows old builds of atom-shell to be used as well.

Fixed by #23

source: gharchive/issue

repo: atom/grunt-download-atom-shell


text:
Cross compile for ia32 target

This PR uses sysroot for building and cross compiles for ia32 target, making it easier to build arm and ia32 targets.

It seems extremely hard to make the sysroot way work, giving up now.

repo: atom/libchromiumcontent

license: mit

license_source: bigquery


text:
vim inserts are interrupted by cursor movement, vim-mode inserts aren't

In vim, if you move your cursor around (with mouse or with arrow keys), an insert gets confirmed and a new gets started - they can be undone separately, and only the last part can be redone. To demonstrate:

have 1234 in the text editor, with cursor on 2
type iab<right>cd<esc> – you will have 1ab2cd34 in the text editor
typing u should only undo cd, typing . should only insert cd in the current place

But in vim-mode, typing u undoes both ab and cd insertion, and typing . inserts abcd.
See vim docs for more info about this.
I'll now have a quick look if I can fix this easily.

I'd like to emphasize the importance of this one — sometimes I'm in insert mode for a long time, and find that undo undoes WAY too much. Sometimes the last several minutes of edits. I find that this leaves me no meaningful way to only undo a little of it.
It also overrides the behavior of cmd-z (which makes sense, but leaves no alternative when u undoes too much)

+1

Thanks, I should soon get some time to rebase this to the newest state of
the code. @maxbrunsfeld any suggestions in case you don't quite like how it
was done?
On 3 Jun 2015 22:20, "Markus Nilssen" notifications@github.com wrote:

+1
—
Reply to this email directly or view it on GitHub
https://github.com/atom/vim-mode/issues/565#issuecomment-108618928.

As stated in the README, this package is no longer maintained and is deprecated. We recommend that people use the vim-mode-plus package instead. Because of this, we are archiving this repository and closing all issues and pull requests. Thanks very much for your support and contributions!

authors: MarkusSN jacekkopecky lee-dohm peterjeliot

repo: atom/vim-mode


text:
Question: Can we potentially use x-ray on a thin-client while the x-ray core server could be installed remotely? (Hackable Cloud IDE?)

Firstly,  I think this project looks really promising, kudos to the team!
I was wondering if we could potentially use x-ray on a thin-client while the x-ray core server could be installed remotely (x-ray packages, extensions, language servers etc... all installed remotely)? Source files and dependencies would also reside in the server. However, the UX on the client feels as if everything is running locally.
This would obviously require an integrated terminal which is also fed from the one available on the machine x-ray core server is installed at. Sort of like a cloud IDE except completely hackable and configurable?
I got a sense from the architecture diagram that maybe you guys already have something like this in mind (beside realtime collaboration, I mean like a "Single player mode" would be nice).
I think this would eliminate so many issues teams face.  Like having to stick to certain hardware for their employees, configurations and setting each one up then fixing stability issues every now and then. It would be nice to simply give everyone a cloud instance running x-ray and any decent laptop where they could use the x-ray client. This would also make learning Vim or emacs for working over ssh unnecessary and we can finally have a nice remote development experience? It will also be significantly cheaper and much easier to keep upto date.
Thoughts?

I like this idea! It's possible to support RPC over ssh tunnel.

authors: Kasahs SunDoge

repo: atom/xray


text:
feat: Printing the url list to txt file with output-file

upd: The url list of all request is saved in the ./output/output.txt file.
feat: Print query results as array with output-json
upg: Dependencies upgraded.
upd: Readme new flags added.
I also checked it with 17k rows of data. It worked flawlessly.
I hope I made bad code. I recently started working to improve myself in Go. Thanks @atomicptr

The requirement here is met. https://github.com/atomicptr/crab/issues/4

When I get to the computer, I will ask about the necessary corrections and what some of the requirements mean in detail.  Thanks @atomicptr

I also updated the readme and help sections. There will be no such problem. Thanks for the information and collaboration. @atomicptr
Best regards from Turkey

There seems to be a problem with the records after scanning large xml files :(

Seems like it could solve this problem:
if data["status"] != nil || data["url"] != nil || data["time"] != nil || data["duration"] != nil {
	status = data["status"].(float64)
	url = data["url"].(string)
	time = data["time"].(float64)
	duration = data["duration"].(float64)

	_, err = file.WriteString(fmt.Sprintf("%d\t%s\t%d\t%d", int(status), url, int(time), int(duration)) + "\n")
} else {
	_, err = file.WriteString(message + "\n")
}

I may need some time to make sure the code works.

This time problem solved :)

I thought it would be good to relocate the error message so it can be parsed.

I'm sending the final version this way.
The command I used for testing: (It contains about 17000 pages.)
crab crawl:sitemap https://www.agroworlddergisi.com/sitemap_index.xml --num-workers=500 --http-timeout=60s --output-file ./output/output.txt --output-json ./output/output.json

Thanks for the changes, I found a few more things but otherwise it looks great :)

I have completed the final changes required. I am updating the code and sending it. @atomicptr

whoops something went wrong here :eyes:

I kinda fucked something up by using a gh cli tool not quite sure what yet although I merged the changes with #11
Thank you very much for the PR! :)

Thanks for the fixes and Merge. @atomicptr

authors: atomicptr ramazansancar

repo: atomicptr/crab


text:
Ensure client/server transports are closed on server leave()

clientTransport and serverTransport are not properly closed when a server is removed from the cluster. This PR simply adds the same block as is used in shutdown() to ensure Transports are properly shut down when the server is.

👍

authors: jhalterman kuujo

repo: atomix/copycat


text:
The single_value function throws an error when missing data in referenced store

atoti 0.7.0
When used on a column in a reference store, the single value function can throw an error if there are rows in the base store that do not map onto any rows in the reference store.
Code to reproduce:
import atoti as tt
import pandas as pd
from datetime import date

session = tt.Session()

base = session.read_pandas(
    pd.DataFrame(
        columns=["Date", "Id", "RefId", "Misc", "Qty"],
        data=[
            (date.fromisoformat("2022-10-03"), "1", "A", "X", 5),
            (date.fromisoformat("2022-10-03"), "2", "A", "X", 10),
            (date.fromisoformat("2022-10-03"), "3", "B", "X", 20),
            (date.fromisoformat("2022-10-03"), "4", "B", "Y", 30),
            (date.fromisoformat("2022-10-03"), "5", "B", "Y", 25),
        ],
    ),
    table_name="Base",
    keys=["Date", "Id"],
    types={"Date": tt.type.LOCAL_DATE}
)

ref = session.read_pandas(
    pd.DataFrame(
        columns=["Date", "RefId", "Rate"],
        data=[
            (date.fromisoformat("2022-10-03"), "A", 10)
        ],
    ),
    table_name="Ref",
    keys=["Date", "RefId"],
    types={"Date": tt.type.LOCAL_DATE}
)

base.join(ref)

cube = session.create_cube(base)

m = cube.measures

m["Rate"] = tt.agg.single_value(ref["Rate"])

cube.query(m["Rate"])

Note I attempted to recreate this with only a single key field in the reference store (no date column) and the issue did not occur.

Hi !
Thanks for reporting this issue, I can reproduce it. We'll work on a fix.

Hello @ollien1 ,
This will be fixed in the next release (0.7.1).

authors: bloussou mailys-ds ollien1

repo: atoti/atoti


text:
Setup environment for tests

read docs about testing django apps
choose framework for testing
write 5 simple tests

There is a traditional unittests library which is integrated in django by default. As pytest allows to write tests more densely (without large classes and suite aggregation), it was chosen to stick to this lib. To use it, additional dependency should be installed (https://pytest-django.readthedocs.io/en/latest/index.html).

repo: atsanda/Expenshare


text:
Sync progress callback

Is your feature request related to a problem? Please describe.
Currently apps don't have any callbacks from client SDK to know sync progress
Describe the solution you'd like

Expose an abstract listener which widgets can implement and listen to sync progress state

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

draft PR https://github.com/atsign-foundation/at_client_sdk/pull/399/files

Feature completed and published at client version 3.0.10


text:
fix: multibuild (+ reduced build duration)

- What I did

Converted to absolute path for multibuild upload artifacts, because actions/upload-artifact doesn't respect the working-directory.

See https://github.com/actions/upload-artifact/issues/294

Added a job to alert in google chat if ssh no ports builds fail
Doing other_build in parallel to reduce wait time of build

- How I did it
- How to verify it
See https://github.com/atsign-foundation/sshnoports/actions/runs/5606212030
- Description for the changelog
fix: multibuild (+ reduced build duration)

This needs to be merged so we can run multibuild against trunk for the next release

repo: atsign-foundation/sshnoports


text:
Oktameter research

Find a simple, low-power way to determine cloud cover - ideally in Oktas - as this will enable more costly cloud photogrammetry readings to be skipped in cases where there is no cloud or where the sky cannot be seen (eg. due to heavy fog, snow, etc).
Cloud presence can be determined at night using a Seebeck TEG sandwiched between two metal plates. If sky is clear, the top plate will be cooler than the bottom plate. This could double up as power generation during daytime?
A thermopile sensor filtered to wavelengths above 5µm (ideally 8–14µm). Other methods for measuring solar radiation might be valid.

Rainlight sensor could potentially also be used as an indicator?


text:
Issue when trying to run in Objective-C project

I got this error when tried to run my app with AudioKit.framework embedded (not used in code, just added)

dyld: Library not loaded: @rpath/libswiftAVFoundation.dylib
Referenced from: /Users/**********/Frameworks/AudioKit.framework/AudioKit
Reason: image not found

Changing Build Settings->Embedded Content Contains Swift Code to YES resolved the issue.
Can you add information about this to INSTALL.md?

@olexl If you're comfortable with adding it yourself and making a pull-request, feel free to. If you're not, I can add this note in the INSTALL.md file.

I think I'll do it myself, I'm gonna get the release ready for 3.1.1 anyway and might as well do it at the same time.

OK, added a blurb in INSTALL.md in develop, will be in 3.1.1

thanks

authors: megastep narner olexl

repo: audiokit/AudioKit


text:
readme.md edits for clarity

Made some edits to prep this document for ssbc.github.io, and decided it was smart to have them reviewed in case I accidentally changed the meaning of anything.

merged into 0.1.1

authors: dominictarr pfraze

repo: auditdrivencrypto/private-box


text:
Modals are not accessible

Modals are not accessible with focus stealing & recovery and using esc to close etc. A simple fix would be to use @headlessui/react which can manage all of the state for you, plus you get the advantage of having headlessui in the project which we can use for other things

I'll do some accessibility tweaks later this week!
If you have time, would you mind looking at #16?

authors: alii audn

repo: audn/frontend-boilerplate


text:
Remove xfail in test_cookiecutters

Thank you for merging https://github.com/audreyr/cookiecutter-jquery/pull/2, @audreyr :bow:

Current coverage is 99.27%

Merging #618 into master will not affect coverage as of 12668d7

@@            master    #618   diff @@
======================================
  Files           12      12
  Stmts          549     549
  Branches         0       0
  Methods          0       0
======================================
  Hit            545     545
  Partial          0       0
  Missed           4       4

Review entire Coverage Diff as of 12668d7

Uncovered Suggestions

+0.36% via ...ookiecutter/utils.py#27...28
+0.36% via ...okiecutter/config.py#19...20

Powered by Codecov. Updated on successful CI builds.

authors: codecov-io hackebrot

repo: audreyr/cookiecutter


text:
update to v0.99.0

https://github.com/Dogfalo/materialize/releases/tag/v0.99.0

Added support for jQuery 3
Fixed dynamic textarea resize bug
Added support for custom active elements in scrollspy
Added Time Picker
Updated styling for Date Picker
Added callbacks to side nav
Updated styling for switches

https://github.com/Dogfalo/materialize/issues/4871

Thanks, @tomscholz I'll change the AMD scripts accordingly. 👍

I didn't notice this thread so I opened a new one...sorry for that

No problem at all.

Datepicker layout glitch with an unmodified bridge and materialize v0.99.0:

Just added sidenav open/close events.
Needs a sample.

Custom active elements in Scrollspy:
Maybe I still got it wrong but this won't work at all: https://codepen.io/anon/pen/OgOmre?editors=1111
    $('.scrollspy').scrollSpy({
      getActiveElement: function(id) {
        return 'span[x-ref="#' + id + '"]';
      }
    });

html:

Introduction

Structure

Intialization

Closing since a new version is available.
Copied the one open point to: #459

authors: Thanood sabinrimni tomscholz


text:
di, nodejs

Can one use the di in a nodejs environment?
It seems that the normal usage scenario does not seem to work as the container instance in undefined.
Is there by any chance a example of using the di in nodejs?

have a look at this article http://aurelia.io/docs/testing/components#testing-custom-component-with-a-real-view-model
testing works in the context of nodejs
and there are examples of using DI

@pragmaproducts have you been able to check out my suggestion?

async triageQuestion(id) =>{
 respondSameDay()
 let response = await getResponseFromOP()
 if(!response) {
    closeQuestion(id)
 }
}

Closing. DI can definitely be used in node.js. We use it inside our CLI :)

repo: aurelia/dependency-injection


text:
Named slot inside ai-dialog not working.

I'm submitting a bug report

Library Version:
aurelia-dialog@1.0.0-beta.3.0.1
aurelia-templating@1.4.2

Please tell us about your environment:

Operating System:
Windows 10

Browser:
Chrome 58

Language:
TypeScript

Current behavior:
I've created a custom element for dialogs and tried to use named slots inside like this:
<template>
  <ai-dialog-body>
     <slot name="dialog-body">BODY_CONTENT</slot>
  </ai-dialog-body>
</template>

But the slot is never filled. Even the fallback content "BODY_CONTENT" is never filled.
Workaround
If I surround the slot with span like this, everything is working.
<slot name="dialog-body">BODY_CONTENT</slot>

Not dialog related.
Also I don't see it as an issue. When using a custom element and adding a slot, with fallback content, as a direct child of it, you should do it to target a slot in that custom element - more here.
<my-element>
  <slot slot="slot-name-inside-my-element">new fallback content<slot>
</my-element>

Wrapping it in another element, in your case , works since that element, with its content, gets projected into the default slot of <ai-dialog-body>, creating the dialog-body slot inside it.
@PWKad Do you see anything wrong with the explanation?

repo: aurelia/dialog


text:
Skeleton-navigation organization: prefer branches to subdirectories

I like what has been done with the skeleton-navigation project - consolidating all of the skeleton projects is a good idea.
I'd like to propose using long-lived, named git branches for the different skeletons, and maybe open discussion here about it and toss around some ideas, pros, and cons.
Pros:

Better separation of the individual skeletons.
git clone can pull only what's needed (via git clone url@branch instead of pulling in unneeded skeleton directories. (If I'm a Typescript user, I don't need the other three skeletons when I clone.)
Master branch could be used for active development, and merges could take place between master and skeleton-type branches.
Possibly more scalable in the future if/when the skeletons grow and multiply.

Cons:

Developers have to specify which branch they want to pull when cloning (or else they'll get master, which may be problematic).
Branches could get messy if they are also used for something other than the skeleton types.
Might be more difficult for beginners, people unfamiliar with git, and/or people unfamiliar with Github (viewing different branches, etc).

Of note, it's also possible to change the default branch of the repository, but I imagine there wouldn't be a good way to decide which skeleton should be the default HEAD.
Thoughts?

It's a good idea. However, long term, we want to get rid of this :) What we want is a "new project" wizard built directly into aurelia.io. You would click a button to start a new project, answer a few questions and then get a customized download with your skeleton. We'd like to base the output of that wizard on today's skeletons.
Thoughts? Interest in helping to build it? :smile:

That would definitely go a long way in getting developers started (as long
as others are still allowed to git clone an existing skeleton project). I
think it could work well with git branches, but compatibility with the
existing git system would depend upon the number of skeleton permutations
we'd have and the options we'd want to provide. Matthew James Davis and I
actually had a conversation about this a while back and the permutations
and different options could get quite large.
I'd be willing to help build this. I'm interested to see how this will turn
out.
On Sat, Jan 9, 2016 at 3:40 PM, Rob Eisenberg notifications@github.com
wrote:

It's a good idea. However, long term, we want to get rid of this :) What
we want is a "new project" wizard built directly into aurelia.io. You
would click a button to start a new project, answer a few questions and
then get a customized download with your skeleton. We'd like to base the
output of that wizard on today's skeletons.
Thoughts? Interest in helping to build it? [image: :smile:]
—
Reply to this email directly or view it on GitHub
https://github.com/aurelia/skeleton-navigation/issues/251#issuecomment-170279563
.

@EisenbergEffect a yeoman generator can act as a wizard from the cli.

Yes, but then you have to install yeoman and we are trying to avoid
installation requirements.
On Jan 9, 2016 7:10 PM, "Gheorghe Anghelescu" notifications@github.com
wrote:

@EisenbergEffect https://github.com/EisenbergEffect a yeoman generator
http://yeoman.io/authoring/ can act as a wizard from the cli.
—
Reply to this email directly or view it on GitHub
https://github.com/aurelia/skeleton-navigation/issues/251#issuecomment-170295726
.

I think the biggest problem with using Git branches is that we would no longer be able to use GitHub releases.

@AshleyGrant makes a good point here. Additionally, I didn't realize this until I looked at the .NET 5 skeletons, but the skeletons for the .NET projects are structured completely differently - they are geared specifically towards VS users and are structured as .NET project types. A lot of the advantages of branches may be lost if the structure is much different (as merging would be fairly difficult at best).
Matthew and I had briefly discussed the merits of creating Nuget packages for such skeletons (and creating separate Nuget packages for the build tooling for the 2013 skeleton that Ashley made) but it was all in passing.
While I like the branch idea, it seems as though there may be too many obstacles to make it viable. That said, an online generator (and a CLI yeoman generator for people familiar with the CLI, node, and yeoman) could go a long way towards mitigating these discrepancies.
@EisenbergEffect I'm not sure there's a way to avoid a CLI dependency to generate Aurelia projects unless you provide a CLI-specific tool for this (similar to SailsJS with their sails generate command). There is currently an Aurelia CLI tool, though, used for bundling - perhaps this could have a generate command associated with it? One could run aurelia-cli generate or aurelia-cli new and receive a prompt similar to the npm init command. Perhaps I'm getting off track here, though, as the main focus was supposed to be on the web generator and the branch feasibility.
TL;DR: Ashley raises a good point. Branches may not be viable for various reasons.   I suppose online generation and generation through Aurelia's CLI tooling.

The concept of GitHub releases is no longer valid when putting multiple folders with different skeletons in the same project anyway (at least to have a single .zip download that only contains the project you want) but lots of relevant discussion in here.
@jedd-ahyoung  One note is that the ASP.NET 5 application structure is based on Visual Studio's project structure but we could easily remove the solution file and the src file and bring it back down a notch but I could not get a clear answer from anyone on pro's or con's for either way.  In the end I left it in there because I figured it was easier for experienced users to know what to remove than for new beginners to figure out what to add, if that makes sense.

@EisenbergEffect

What we want is a "new project" wizard built directly into aurelia.io

That is good idea, and it doesn't conflict with @Gheoan's suggestion:

yeoman generator can act as a wizard from the cli.

aurelia.io backend could run yeoman generator and offer a zip for downloading (cached based on options selected from aurelia.io site). I think that Spring Initializr works the same way, and i love it. Also i like very much what i've seen about Yeoman for AnglularJS. I'd love to see smth even half as good for Aurelia

@EisenbergEffect, @jedd-ahyoung, is there any progress with the mentioned 'new project'? i'd love to participate in building it.

another way of tackling this would be to make each skeleton flavor a git-submodule, but then we're in a different kind of hell.
@jedd-ahyoung, regarding this:

git clone can pull only what's needed [...] instead of pulling in unneeded skeleton directories.

you can clone specific directories with git (>=1.7) using sparse-checkout.

Closing this for now. I don't think we're going to change the organization. In the future, we think most people will use the CLI, however skeletons will remain to show integration with other technology and variant setups.

authors: AshleyGrant EisenbergEffect Gheoan PWKad atsu85 eliranmal jedd-ahyoung

repo: aurelia/skeleton-navigation


text:
ValidationController.validate instance works differently depending on whether addObject has been called

I might be reporting a bug here, but am not sure.  The behavior is not what I would expect.
Suppose I want to validate an object instance.
I can call:
ValidationController.addObject(instance);
ValidationController.validate();

^^^ this works fine
Or I can call:
ValidationController.addObject(instance);
ValidationController.validate({ object: instance });

and this works too.
Or I can call:
ValidationController.validate({ object: instance })
without calling ValidationController.addObject(instance)
This ^^^ doesn't work.  The validation renderer gets all out of sync when revalidating the same instance repeatedly.  One ends up with validation errors not being unrendered when they should.
My expectation is that the latter case would work fine, that calling addObject is only meant to be a convenience so that ValidationController.validate can be used to validate all instances that have been added, and that addObject would not be required if one is willing to call ValidationController.validate({ object: instance }).

See this gist.  Just press the Submit button more than once and see then validation errors accumulate.

thanks for the gist- was able to track this down.  It's fixed in https://github.com/aurelia/validation/commit/d57f09ff35de4a2d95cef037ba008678e2d01636
I'm keeping this open and will re-test with the updated build when it's released.

repo: aurelia/validation


text:
Indicate relative magnitudes in common unit labels

A core part of Au's philosophy is "no preferred units".  So when we subtract, say, MPH from KPH, the result is labeled as essentially "the common unit of MPH and KPH": COM[km / h, mi / h].
I think this is a mixed bag.  It's good that we don't conjure up any unmentioned units.  COM[km / h, mi / h] does unambiguously identify what unit is being used.  But this isn't a commonly used unit, and it's clearly bad that the label doesn't give any clue as to its magnitude, relative to any other unit.
I've found a new candidate solution.  We can express the common unit in terms of each constituent unit --- and, for clarity, replace the , with ==.  For example: COM([1 / 15625] km / h == [1 / 25146] mi / h).

PRO: Doesn't put any unit on special footing.
PRO: Clearly indicates the common unit's magnitude in terms of each constituent unit.
CON: A little verbose.

I think this is a clear improvement.  Common-unit quantities generally only occur as the result of intermediate computations.  They're not the kind of thing that end users are going to want to print, unmodified.  But when they do end up being output, they'd better be unambiguous!
This solution depends on #85, because we need to be able to print magnitudes.

Related: mpusz/units#438.

Note: in discussions relating to the standard units project, we ended up converging on an EQUIV syntax:
Old:
COM([1 / 15625] km / h == [1 / 25146] mi / h)

New:
EQUIV{[1 / 15625] km / h, [1 / 25146] mi / h}

This was the brainchild of @JohelEGP.  I like it a lot!
Actually, we want to include the unscaled unit label inside of the square brackets, so it will be more like this:
EQUIV{[1 / 15625 km / h], [1 / 25146 mi / h]}

Actually, we want to include the unscaled unit label inside of the square brackets, so it will be more like this:
EQUIV{[1 / 15625 km / h], [1 / 25146 mi / h]}

For even better clarity, if the magnitude label has a /, we should enclose it in parentheses:
EQUIV{[(1 / 15625) km / h], [(1 / 25146) mi / h]}

repo: aurora-opensource/au


text:
aurutils fails when looking for orderfile in wrong place /temp${HOME}/.config/aurutils/sync/orderfile

Hi,
Running aurutils sync -u fails because aurutils is looking for orderfile in the wrong place.
I can reproduce this with aurutils and aurutils-git from the aur.
Seems like my home directory name is prepended by /temp for some reason (without / after /temp).
AUR_DEBUG=1 aur <name> <arguments> >aurutils.log 2>&1
[aurutils.log](https://github.com/user-attachments/files/15860048/aurutils.log)

This error is coming from git. Do you use any wrappers that change the behaviour of git? (E.g. type -P git in a bash shell)

No, I did not use any wrappers.
It seems that the error was caused by the PKGBUILD of alpine. I removed the cache of alpine, and I can not reproduce anymore.

I guess /temphome was set by git config for some reason. aur-sync would only do it based on the value of $HOME, when a package is cloned for the first time. So HOME=/temphome/... aur sync alpine would break future calls of aur sync alpine.

authors: AladW Earnestly WildPenquin

repo: aurutils/aurutils

license: ISC


text:
Token is being sent in header for https://www.example.com but not for https://example.com

I have whitelisted the domain.
whiteListedDomains: ['localhost', 'www.example.com']
When accessed the site with www in the url Ex: https://www.example.com, auth token gets added in request header successfully. But its wont work for the url https://example.com

what if you whitelist both? or just like this: whiteListedDomains: ['example.com']


text:
Handle cancel of authorization flow in MvcSample

Show an error page when we cancel the authorization flow (eg: cancel
Facebook consent)

excellent!

authors: sandrinodimattia siacomuzzi


text:
fix!: Prevent misconfiguration of the secret used with the HS*** algorithms

Description
Added checks to make sure that a valid secret is used with the HS*** algorithms when signing and verifying
BREAKING CHANGE: Requires node 12.x or later to allow use of KeyObject
Testing

All automated tests pass

Checked via script that this type of misconfiguration does not pass signing or verification

[x] This change adds test coverage for new/changed/fixed functionality

Checklist

[x] I have added documentation for new/changed functionality in this PR or in auth0.com/docs
[x] All active GitHub checks for tests, formatting, and security are passing
[x] The correct base branch is being used, if not master

Closed in favour of https://github.com/auth0/node-jsonwebtoken/pull/852


text:
fix TypeError when authenticate is sent with null

When 'authenticate' event is sent by client with null(by mistake or intentionally), TypeError occurs.

Thanks and sorry for the delay

Published as v4.4.1

authors: bartlomiej-korpus jfromaniello


text:
Upgrade from 4.38.16 to 4.38.17, SMTP authentication fails at startup.

Version
v4.38.17
Deployment Method
Kubernetes
Reverse Proxy
Traefik
Reverse Proxy Version
Latest
Description
Upgrading authelia from 4.38.16 to 4.38.17, the authentication to smtp fails at startup.
Rolling back to the prior version succeeds.
Upgrade/Downgrade is done via the authelia helm chart, by upgrading the image tag.
Reproduction
Upgrade with config as shown below
Expectations
Authelia should startup properly.
Configuration (Authelia)
---
# yaml-language-server: $schema=https://www.authelia.com/schemas/v4.38/json-schema/configuration.json
theme: 'light'
default_2fa_method: 'totp'
server:
  address: 'tcp://0.0.0.0:9091/'
  asset_path: ''
  headers:
    csp_template: ''
  buffers:
    read: 4096
    write: 4096
  timeouts:
    read: '6 seconds'
    write: '6 seconds'
    idle: '30 seconds'
  endpoints:
    enable_pprof: false
    enable_expvars: false
    authz:
      auth-request:
        implementation: 'AuthRequest'
      ext-authz:
        implementation: 'ExtAuthz'
      forward-auth:
        implementation: 'ForwardAuth'
log:
  level: 'info'
  format: 'text'
  file_path: ''
  keep_stdout: true
telemetry:
  metrics:
    enabled: false
identity_validation:
  elevated_session:
    code_lifespan: '5 minutes'
    elevation_lifespan: '10 minutes'
    characters: 8
    require_second_factor: false
    skip_second_factor: false
  reset_password:
    jwt_lifespan: '5 minutes'
    jwt_algorithm: 'HS256'
totp:
  disable: false
 ...
webauthn:
...
ntp:
  address: 'udp://time.cloudflare.com:123'
  version: 4
  max_desync: '3 seconds'
  disable_startup_check: false
  disable_failure: false
authentication_backend:
  password_reset:
    disable: false
    custom_url: ''
  ldap:
    implementation: 'custom'
...
password_policy:
  standard:
    enabled: false
    min_length: 8
    max_length: 0
    require_uppercase: false
    require_lowercase: false
    require_number: false
    require_special: false
  zxcvbn:
    enabled: false
    min_score: 0
session:
  name: 'authelia_session'
  same_site: 'lax'
  inactivity: '5 minutes'
  expiration: '1 hour'
  remember_me: '1 month'
  cookies:
    - domain: 'somplace.com'
      authelia_url: 'https://auth.somplace.com'
  redis:
...
regulation:
  max_retries: 3
  find_time: '2 minutes'
  ban_time: '7 weeks'
storage:
  postgres:
...
notifier:
  disable_startup_check: false
  smtp:
    address: 'submission://internal.somplace.com:587/'
    timeout: '5 seconds'
    username: 'authelia'
    sender: 'Authelia <authelia@internal.somplace.com>'
    identifier: 'authelia'
    subject: '[Authelia] {title}'
    startup_check_address: 'authelia@internal.somplace.com'
    disable_html_emails: false
    disable_require_tls: false
    disable_starttls: false
    tls:
      server_name: ''
      skip_verify: false
      minimum_version: 'TLS1.2'
      maximum_version: 'TLS1.3'
...```

authelia password secret
apiVersion: v1
kind: Secret
metadata:
  name: authelia
  namespace: web-auth
  uid: 7f3483ba-add6-4f0f-867f-324b1647d2ae
  resourceVersion: '41198365'
  creationTimestamp: '2024-06-26T04:44:05Z'
  labels:
    kustomize.toolkit.fluxcd.io/name: apps
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  selfLink: /api/v1/namespaces/web-auth/secrets/authelia
data:
  notifier.smtp.password.txt: YXV0aGVsaWE=
type: Opaque

Build Information
Last Tag: v4.38.17
State: tagged clean
Branch: v4.38.17
Commit: 592ab519b4167dfe4462ee0a613d38d3e4b8efa0
Build Number: 35061
Build OS: linux
Build Arch: amd64
Build Compiler: gc
Build Date: Thu, 31 Oct 2024 09:37:31 +1100
Extra:

Go:
    Version: go1.23.2
    Module Path: github.com/authelia/authelia/v4
    Executable Path: github.com/authelia/authelia/v4/cmd/authelia

Logs (Authelia)
Logs from 4.38.17
time="2024-11-07T15:19:50+13:00" level=info msg="Authelia v4.38.17 is starting"
time="2024-11-07T15:19:50+13:00" level=info msg="Log severity set to info"
time="2024-11-07T15:19:50+13:00" level=info msg="Storage schema is being checked for updates"
time="2024-11-07T15:19:50+13:00" level=info msg="Storage schema is already up to date"
time="2024-11-07T15:19:51+13:00" level=error msg="Error occurred running a startup check" error="failed to dial connection: SMTP AUTH failed: 535 5.7.8 Error: authentication failed: authentication failure" provider=notification
time="2024-11-07T15:19:51+13:00" level=fatal msg="One or more providers had fatal failures performing startup checks, for more detail check the error level logs" providers="[notification]" stack="github.com/authelia/authelia/v4/internal/commands/root.go:150 doStartupChecks\ngithub.com/authelia/authelia/v4/internal/commands/root.go:88 (*CmdCtx).RootRunE\ngithub.com/spf13/cobra@v1.8.1/command.go:985 (*Command).execute\ngithub.com/spf13/cobra@v1.8.1/command.go:1117 (*Command).ExecuteC\ngithub.com/spf13/cobra@v1.8.1/command.go:1041 (*Command).Execute\ngithub.com/authelia/authelia/v4/cmd/authelia/main.go:10 main\ninternal/runtime/atomic/types.go:194 (*Uint32).Load\nruntime/asm_amd64.s:1700 goexit"

Logs from prior version
time="2024-11-07T12:59:36+13:00" level=info msg="Authelia v4.38.16 is starting"
time="2024-11-07T12:59:36+13:00" level=info msg="Log severity set to info"
time="2024-11-07T12:59:36+13:00" level=info msg="Storage schema is being checked for updates"
time="2024-11-07T12:59:36+13:00" level=info msg="Storage schema is already up to date"
time="2024-11-07T12:59:37+13:00" level=info msg="Startup complete"
time="2024-11-07T12:59:37+13:00" level=info msg="Listening for non-TLS connections on '[::]:9091' path '/'" server=main service=server

Logs (Proxy / Application)
No response
Documentation
No response
Pre-Submission Checklist

[X] I agree to follow the Code of Conduct

[X] This is a bug report and not a support request

[X] I have read the security policy and this bug report is not a security issue or security related issue

[X] I have either included the complete configuration file or I am sure it's unrelated to the configuration

[X] I have either included the complete debug / trace logs or the output of the build-info command if the logs are not relevant

[X] I have provided all of the required information in full with the only alteration being reasonable sanitization in accordance with the Troubleshooting Sanitization reference guide

[X] I have checked for related proxy or application logs and included them if available

[X] I have checked for related issues and checked the documentation

Okay this is an issue with the helm chart so I've transferred it appropriately. It's not configured for some reason. Show your helm values?

Pretty sure this information is for the working version, can you double check this too? https://github.com/authelia/chartrepo/issues/275#issuecomment-2515946888

@mwinters-stuff think maybe you missed this?

authors: james-d-elliott mwinters-stuff

repo: authelia/chartrepo


text:
🛑 Andreas Iten is down

In ba9235f, Andreas Iten (https://www.andreasiten.ch) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Andreas Iten is back up in ef16418.

repo: authentiqdigital/upptime


text:
Support base32 encoded string as init string

Thank you for your work, It help me a lot.
When I uses this as a client, I find that I must use it like this:

auth = OtpAuth(base64.b32decode('ONSWG4TFOQ6T2',True))

so, I want the init function can do these things:

auth = OtpAuth('ONSWG4TFOQ6T2',encoded=True)
or
auth = OtpAuth('ONSWG4TFOQ6T2',True)

and it Compatible with old code：

auth = OtpAuth('secret')

Sorry for the late reply. This library has been redesigned, there is a from_b32encode method now:
https://otp.authlib.org/recipes/#use-base32-secret

authors: lepture tinyhare

repo: authlib/otpauth


text:
🛑 Automat eshop § obchod.auto-mat.cz is down

In 31608ed, Automat eshop § obchod.auto-mat.cz (https://obchod.auto-mat.cz/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Automat eshop § obchod.auto-mat.cz is back up in 9eb3309.

repo: auto-mat/automat-statuspage


text:
🛑 Zažit Město Jinak § zazitmestojinak.cz is down

In a93e411, Zažit Město Jinak § zazitmestojinak.cz (https://zazitmestojinak.cz) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Zažit Město Jinak § zazitmestojinak.cz is back up in 98674be after 671 days, 2 hours, 54 minutes.


text:
[BUG] inference time

It takes 4s for 100 pieces of data reasoning, and 10s for 10,000 pieces of data reasoning. Is this reasonable?

Can I load the trained basic models by yourself and reason by myself?

Can I load the trained basic models by yourself and reason by myself?

Yes, inference on multiple data points is faster due to batching.

authors: shchur xxyliuyang

repo: autogluon/autogluon


text:
[BUG]  NaN Loss Error in NNFastAiTabularModel When Continuous Feature Has Zero Variance and Missing Values

Hello,
I encountered an issue when using the NNFastAiTabularModel. Specifically, when dealing with a continuous feature that has missing values and zero variance, I receive the following error during training:
AssertionError: Exception occured in `EarlyStoppingCallbackWithTimeLimit` when calling event `after_epoch`:
	WARNING: NaN loss encountered in epoch 0!

Expected behavior
I believe the issue arises in line 180 of the following file:
autogluon/tabular/models/fastainn/tabular_nn_fastai.py#L182
A temporary fix that worked for me was adding this to before the line :
cont_std[cont_std==0] = 1e-10

To Reproduce
Here is a minimal example to reproduce the issue:
from autogluon.tabular.models import NNFastAiTabularModel
import numpy as np
import pandas as pd

def generate_xor_data(n_samples=100, feature_num=3):
    X = np.random.rand(n_samples, feature_num)
    X[:, 1] = 0.5
    X[0, 1] = np.nan
    df = pd.DataFrame(X, columns=["input_" + str(i) for i in range(feature_num)])
    df["class"] = np.logical_xor(X[:, 0] > 0.5, X[:, 2] > 0.5).astype(int)
    return df

df = generate_xor_data(n_samples=300)

model = NNFastAiTabularModel(
    path="./autogluon_output", problem_type="binary", eval_metric="roc_auc"
)
model.fit(
    X=df.drop(columns=["class"]),
    y=df["class"],
    num_cpus=1,
    num_gpus=0,
    verbosity=4,
)

Installed Versions

INSTALLED VERSIONS
------------------
date                   : 2024-10-02
time                   : 08:47:52.932468
python                 : 3.10.15.final.0
OS                     : Linux
OS-release             : 6.5.0-1027-oem
Version                : #28-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 25 13:32:46 UTC 2024
machine                : x86_64
processor              : x86_64
num_cores              : 20
cpu_ram_mb             : 63940.265625
cuda version           : None
num_gpus               : 0
gpu_ram_mb             : []
avail_disk_size_mb     : 350069
accelerate             : 0.21.0
autogluon              : 1.1.1
autogluon.common       : 1.1.1
autogluon.core         : 1.1.1
autogluon.features     : 1.1.1
autogluon.multimodal   : 1.1.1
autogluon.tabular      : 1.1.1
autogluon.timeseries   : 1.1.1
boto3                  : 1.35.30
catboost               : 1.2.7
defusedxml             : 0.7.1
evaluate               : 0.4.1
fastai                 : 2.7.17
gluonts                : 0.14.3
hyperopt               : 0.2.7
imodels                : None
jinja2                 : 3.1.4
joblib                 : 1.4.2
jsonschema             : 4.23.0
lightgbm               : 4.3.0
lightning              : 2.3.3
matplotlib             : 3.9.2
mlforecast             : 0.10.0
networkx               : 3.3
nlpaug                 : 1.1.11
nltk                   : 3.9.1
nptyping               : 2.4.1
numpy                  : 1.26.4
nvidia-ml-py3          : None
omegaconf              : 2.3.0
onnxruntime-gpu        : None
openmim                : 0.3.7
optimum                : None
optimum-intel          : None
orjson                 : 3.10.7
pandas                 : 2.2.3
pdf2image              : 1.17.0
Pillow                 : 10.4.0
psutil                 : 5.9.8
pytesseract            : 0.3.10
pytorch-lightning      : 2.3.3
pytorch-metric-learning: 2.3.0
ray                    : 2.31.0
requests               : 2.32.3
scikit-image           : 0.20.0
scikit-learn           : 1.5.2
scikit-learn-intelex   : None
scipy                  : 1.14.1
seqeval                : 1.2.2
setuptools             : 75.1.0
skl2onnx               : None
statsforecast          : 1.4.0
tabpfn                 : None
tensorboard            : 2.18.0
text-unidecode         : 1.3
timm                   : 0.9.16
torch                  : 2.4.0.post101
torchmetrics           : 1.2.1
torchvision            : 0.19.1a0+ee30128
tqdm                   : 4.66.5
transformers           : 4.38.2
utilsforecast          : 0.0.10
vowpalwabbit           : None
xgboost                : 2.1.1

I met similar issue in NeuralNetFastAI

/root/miniconda3/lib/python3.10/site-packages/autogluon/tabular/models/fastainn/tabular_nn_fastai.py:182: RuntimeWarning: invalid value encountered in divide
(X[self.cont_columns].values - cont_mean) / cont_std,
/root/miniconda3/lib/python3.10/site-packages/autogluon/tabular/models/fastainn/tabular_nn_fastai.py:182: RuntimeWarning: divide by zero encountered in divide
(X[self.cont_columns].values - cont_mean) / cont_std,
/root/miniconda3/lib/python3.10/site-packages/autogluon/tabular/models/fastainn/tabular_nn_fastai.py:182: RuntimeWarning: invalid value encountered in divide
(X[self.cont_columns].values - cont_mean) / cont_std,
Warning: Exception caused NeuralNetFastAI to fail during training... Skipping this model.
Exception occured in EarlyStoppingCallbackWithTimeLimit when calling event after_epoch:
WARNING: NaN loss encountered in epoch 0!
Detailed Traceback:
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/site-packages/autogluon/core/trainer/abstract_trainer.py", line 1904, in _train_and_save
model = self._train_single(X, y, model, X_val, y_val, total_resources=total_resources, **model_fit_kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/autogluon/core/trainer/abstract_trainer.py", line 1844, in _train_single
model = model.fit(X=X, y=y, X_val=X_val, y_val=y_val, total_resources=total_resources, **model_fit_kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/autogluon/core/models/abstract/abstract_model.py", line 856, in fit
out = self._fit(**kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/autogluon/tabular/models/fastainn/tabular_nn_fastai.py", line 359, in _fit
self.model.fit_one_cycle(epochs, params["lr"], cbs=callbacks)
File "/root/miniconda3/lib/python3.10/site-packages/fastai/callback/schedule.py", line 121, in fit_one_cycle
self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd, start_epoch=start_epoch)
File "/root/miniconda3/lib/python3.10/site-packages/fastai/learner.py", line 266, in fit
self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
File "/root/miniconda3/lib/python3.10/site-packages/fastai/learner.py", line 201, in with_events
try: self(f'before{event_type}');  f()
File "/root/miniconda3/lib/python3.10/site-packages/fastai/learner.py", line 255, in _do_fit
self._with_events(self._do_epoch, 'epoch', CancelEpochException)
File "/root/miniconda3/lib/python3.10/site-packages/fastai/learner.py", line 203, in with_events
self(f'after{event_type}');  final()
File "/root/miniconda3/lib/python3.10/site-packages/fastai/learner.py", line 174, in call
def call(self, event_name): L(event_name).map(self._call_one)
File "/root/miniconda3/lib/python3.10/site-packages/fastcore/foundation.py", line 159, in map
def map(self, f, *args, **kwargs): return self._new(map_ex(self, f, *args, gen=False, **kwargs))
File "/root/miniconda3/lib/python3.10/site-packages/fastcore/basics.py", line 910, in map_ex
return list(res)
File "/root/miniconda3/lib/python3.10/site-packages/fastcore/basics.py", line 895, in call
return self.func(*fargs, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/fastai/learner.py", line 178, in _call_one
for cb in self.cbs.sorted('order'): cb(event_name)
File "/root/miniconda3/lib/python3.10/site-packages/fastai/callback/core.py", line 64, in call
except Exception as e: raise modify_exception(e, f'Exception occured in {self.__class__.__name__} when calling event {event_name}:\n\t{e.args[0]}', replace=True)
File "/root/miniconda3/lib/python3.10/site-packages/fastai/callback/core.py", line 62, in call
try: res = getcallable(self, event_name)()
File "/root/miniconda3/lib/python3.10/site-packages/autogluon/tabular/models/fastainn/callbacks.py", line 63, in after_epoch
raise AssertionError(f"WARNING: NaN loss encountered in epoch {self.epoch}!")
AssertionError: Exception occured in EarlyStoppingCallbackWithTimeLimit when calling event after_epoch:
WARNING: NaN loss encountered in epoch 0!

Hi @amirbalef and @rsj123, thanks for highlighting this issue! I have fixed this bug in mainline and the fix will be included in the v1.2 release. Cheers!

authors: Innixma amirbalef rsj123


text:
Optimization of lab submission retrieval

To speed up loading of the page with lab results, multiple server requests (currently - one request for every student) will be batched into a single server call.

Coverage decreased (-0.2%) to 46.767% when pulling bbd6598ffa358c388fab666628679de510fff2dc on labresult_page_optimization into 8984ab3097679f09d9647ab4517fa4807da76d13 on master.

repo: autograde/aguis


text:
Fix build errors so other fixes can be applied.

While working on Bug 1643303, the tests in release-0.1 failed because of
all the formatting errors. I'm unsure how these ever built but they
definitely don't build now. This patch should clean things up then we
can proceed to fix Bug 1643303.

Before my fixes
registries/registry.go:112: Debug call has possible formatting directive %s
registries/registry.go:244: Debug call has possible formatting directive %s
registries/registry.go:245: Debug call has possible formatting directive %v
registries/registry.go:246: Debug call has possible formatting directive %v
registries/registry.go:255: Warning call has possible formatting directive %s
registries/registry.go:263: Warning call has possible formatting directive %s
# github.com/automationbroker/bundle-lib/apb
apb/executor.go:128: Debug call has possible formatting directive %v
apb/executor.go:159: Debug call has possible formatting directive %s
apb/executor.go:160: Debug call has possible formatting directive %s
apb/executor.go:161: Debug call has possible formatting directive %s
apb/executor.go:162: Debug call has possible formatting directive %s
apb/executor.go:163: Debug call has possible formatting directive %s
apb/provision_or_update.go:58: Info call has possible formatting directive %s
apb/secrets.go:154: Debugf call needs 1 arg but has 2 args
apb/types.go:236: Debug call has possible formatting directive %s
apb/types.go:238: Debug call has possible formatting directive %s
apb/types.go:239: Debug call has possible formatting directive %s
apb/types.go:240: Debug call has possible formatting directive %t
apb/types.go:241: Debug call has possible formatting directive %s
apb/types.go:242: Debug call has possible formatting directive %s
apb/types.go:247: Debug call has possible formatting directive %#v
apb/types.go:248: Debug call has possible formatting directive %s
apb/types.go:249: Debug call has possible formatting directive %s
apb/types.go:250: Debug call has possible formatting directive %s
apb/types.go:251: Debug call has possible formatting directive %#v
apb/types.go:252: Debug call has possible formatting directive %d
apb/types.go:253: Debug call has possible formatting directive %d
apb/types.go:254: Debug call has possible formatting directive %d
apb/types.go:255: Debug call has possible formatting directive %s
apb/types.go:256: Debug call has possible formatting directive %d
apb/types.go:257: Debug call has possible formatting directive %#v
apb/types.go:258: Debug call has possible formatting directive %#v
apb/types.go:259: Debug call has possible formatting directive %#v
apb/types.go:260: Debug call has possible formatting directive %#v
apb/types.go:261: Debug call has possible formatting directive %s
apb/types.go:262: Debug call has possible formatting directive %v
# github.com/automationbroker/bundle-lib/registries/adapters
registries/adapters/dockerhub_adapter.go:62: Debug call has possible formatting directive %s
registries/adapters/dockerhub_adapter.go:63: Debug call has possible formatting directive %s
registries/adapters/local_openshift_adapter.go:44: Debug call has possible formatting directive %s
registries/adapters/mock_adapter.go:45: Debug call has possible formatting directive %s
registries/adapters/mock_adapter.go:59: Debug call has possible formatting directive %v
registries/adapters/mock_adapter.go:61: Info call has possible formatting directive %d
registries/adapters/openshift_adapter.go:56: Debug call has possible formatting directive %s
registries/adapters/openshift_adapter.go:59: Debug call has possible formatting directive %v
registries/adapters/openshift_adapter.go:69: Debug call has possible formatting directive %v
registries/adapters/rhcc_adapter.go:78: Debug call has possible formatting directive %v
# github.com/automationbroker/bundle-lib/runtime
runtime/watch_pod_test.go:162:8: dashURL declared but not used
vet: typecheck failures
# github.com/automationbroker/bundle-lib/clients
clients/crd.go:39: Warning call has possible formatting directive %v
clients/kubernetes.go:212: Warning call has possible formatting directive %v
clients/openshift.go:277: Warning call has possible formatting directive %v
# github.com/automationbroker/bundle-lib/runtime
runtime/runtime.go:104: Info call has possible formatting directive %v
runtime/runtime.go:175: Info call has possible formatting directive %v
runtime/runtime.go:214: Debug call has possible formatting directive %s
runtime/runtime.go:283: Info call has possible formatting directive %s
runtime/runtime.go:339: Debug call has possible formatting directive %s
runtime/watch_pod.go:161: Warning call has possible formatting directive %v

make: *** [Makefile:24: test] Error 2

After
[jesusr@transam bundle-lib{fix-build-errors}]$ make test
ok      github.com/automationbroker/bundle-lib/apb      (cached)        coverage: 4.4% of statements
ok      github.com/automationbroker/bundle-lib/clients  (cached)        coverage: 20.8% of statements
ok      github.com/automationbroker/bundle-lib/crd      (cached)        coverage: 99.5% of statements
ok      github.com/automationbroker/bundle-lib/registries       (cached)        coverage: 74.7% of statements
ok      github.com/automationbroker/bundle-lib/registries/adapters      (cached)        coverage: 40.1% of statements
ok      github.com/automationbroker/bundle-lib/runtime  (cached)        coverage: 20.5% of statements
?       github.com/automationbroker/bundle-lib/runtime/mocks    [no test files]

Coverage remained the same at 39.022% when pulling 67694e08c9cf279d2eb50d2988540cbd5f5d36fb on jmrodri:fix-build-errors into 8d1b9c024b703db9168b4cfcb090decf0e8a41a6 on automationbroker:release-0.1.

authors: coveralls jmrodri

repo: automationbroker/bundle-lib


text:
TypeError when using Hono framework

Environment information
typescript, hono framework

in index.ts:

import { Hono } from 'hono'
import { prettyJSON } from 'hono/pretty-json'

import { getUsers } from './model'

const app = new Hono()

app.get('/', (c) => c.text('Hello Hono!'))
app.use('*', prettyJSON())
app.notFound((c) => c.json({ message: 'Not Found', ok: false }, 404))

app.get('/users', (c) => {
  const users = getUsers()
  return c.json({ users })
})

export default app

and model.ts:
import { autometrics } from 'autometrics'

export interface User {
  id: string,
  name: string,
  email: string,

}

export const getUsers = autometrics(async function getUsers(): Promise<User[]> {
  const users: User[] = []
  return users
})

### What happened?

$ bun run --hot src/index.ts
63 |   var _a, b;
64 |   const defaultPrepareStackTrace = Error.prepareStackTrace;
65 |   Error.prepareStackTrace = (, stack2) => stack2;
66 |   const { stack: stackConstructor } = new Error();
67 |   Error.prepareStackTrace = defaultPrepareStackTrace;
68 |   const stack = stackConstructor.map((callSite) => ({
^
TypeError: stackConstructor.map is not a function. (In 'stackConstructor.map((callSite) => ({
name: callSite.getFunctionName(),
file: callSite.getFileName()
}))', 'stackConstructor.map' is undefined)
at getModulePath (/Users/mies/p/am-samples/hono-autometrics/node_modules/@autometrics/autometrics/dist/index.js:68:16)
at autometrics (/Users/mies/p/am-samples/hono-autometrics/node_modules/@autometrics/autometrics/dist/index.js:309:17)
at /Users/mies/p/am-samples/hono-autometrics/src/model.ts:10:24
63 |   var _a, b;
64 |   const defaultPrepareStackTrace = Error.prepareStackTrace;
65 |   Error.prepareStackTrace = (, stack2) => stack2;
66 |   const { stack: stackConstructor } = new Error();
67 |   Error.prepareStackTrace = defaultPrepareStackTrace;
68 |   const stack = stackConstructor.map((callSite) => ({
^
TypeError: stackConstructor.map is not a function. (In 'stackConstructor.map((callSite) => ({
name: callSite.getFunctionName(),
file: callSite.getFileName()
}))', 'stackConstructor.map' is undefined)
at getModulePath (/Users/mies/p/am-samples/hono-autometrics/node_modules/@autometrics/autometrics/dist/index.js:68:16)
at autometrics (/Users/mies/p/am-samples/hono-autometrics/node_modules/@autometrics/autometrics/dist/index.js:309:17)
at /Users/mies/p/am-samples/hono-autometrics/src/model.ts:10:24

### Expected result

no error

Can you try this in Node.js and/or Deno? That stack trace is horrible, but I wouldn't be surprised if this is more of a Bun issue. It complains about the stackConstructor, which seemingly comes from the standard Error object. Unless the framework is doing some dirty prototype pollution, it's most likely Autometrics assumes some features of the runtime that are not there.

did a little further digging on this. The main problem is that the stackConstructor (an interface that we rely on) is V8-only (which is why it works on Node/Deno) meanwhile Bun uses JavaScriptCore. I probably need to revert to using good old manual "stacktrace as string" parsing for these runtimes.

authors: arendjr keturiosakys mies

repo: autometrics-dev/autometrics-ts


text:
Error when tuning number of hidden layers

Hi!
I faced an error trying when adding hidden_layers to a list of parameters. To replicate it, I followed Breast Cancer study, just had to change "shape" to "shapes" in the parameter list.
def breast_cancer_model(x_train, y_train, x_val, y_val, params):
    model = Sequential()
    model.add(Dense(10, input_dim=x_train.shape[1],
                    activation=params['activation'],
                    kernel_initializer='normal'))

    model.add(Dropout(params['dropout']))

    hidden_layers(model, params, 1)

    model.add(Dense(1, activation=params['last_activation'],
                    kernel_initializer='normal'))

    model.compile(loss=params['losses'],
                  optimizer=params['optimizer'](lr=lr_normalizer(params['lr'],params['optimizer'])),
                  metrics=['acc', fmeasure])

    history = model.fit(x_train, y_train,
                        validation_data=[x_val, y_val],
                        batch_size=params['batch_size'],
                        epochs=params['epochs'],
                        verbose=0)

    return history, model

p = {'lr': (0.5, 5, 10),
     'first_neuron':[4, 8, 16, 32, 64],
     'hidden_layers':[0, 1, 2],
     'batch_size': (2, 30, 10),
     'epochs': [150],
     'dropout': (0, 0.5, 5),
     'weight_regulizer':[None],
     'emb_output_dims': [None],
     'shapes':['brick','long_funnel'],
     'optimizer': [Adam, Nadam],
     'losses': [ binary_crossentropy],
     'activation':[relu, elu],
     'last_activation': [None]}

t = ta.Scan(x=x,
            y=y,
            model=breast_cancer_model,
            params=p,
            dataset_name='breast_cancer',
            experiment_no='1')

This returns an error:
IndexError: list index out of range
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-16-109ab999a0ed> in <module>
      4             params=p,
      5             dataset_name='breast_cancer',
----> 6             experiment_no='1')

~/.local/lib/python3.5/site-packages/talos/scan/Scan.py in __init__(self, x, y, params, model, dataset_name, experiment_no, x_val, y_val, val_split, shuffle, round_limit, grid_downsample, random_method, seed, search_method, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, reduce_loss, last_epoch_value, clear_tf_session, disable_progress_bar, print_params, debug)
    168         # input parameters section ends
    169
--> 170         self._null = self.runtime()
    171
    172     def runtime(self):

~/.local/lib/python3.5/site-packages/talos/scan/Scan.py in runtime(self)
    173
    174         self = scan_prepare(self)
--> 175         self = scan_run(self)

~/.local/lib/python3.5/site-packages/talos/scan/scan_run.py in scan_run(self)
     16                      disable=self.disable_progress_bar)
     17     while len(self.param_log) != 0:
---> 18         self = scan_round(self)
     19         self.pbar.update(1)
     20     self.pbar.close()

~/.local/lib/python3.5/site-packages/talos/scan/scan_round.py in scan_round(self)
     30     # fit the model
     31     try:
---> 32         _hr_out, self.keras_model = ingest_model(self)
     33     except TypeError as err:
     34         if err.args[0] == "unsupported operand type(s) for +: 'int' and 'numpy.str_'":

~/.local/lib/python3.5/site-packages/talos/model/ingest_model.py in ingest_model(self)
      8                       self.x_val,
      9                       self.y_val,
---> 10                       self.round_params)

<ipython-input-7-7bad7210c65c> in breast_cancer_model(x_train, y_train, x_val, y_val, params)
     10
     11     # if we want to also test for number of layers and shapes, that's possible
---> 12     hidden_layers(model, params, 1)
     13
     14     # then we finish again with completely standard Keras way

~/.local/lib/python3.5/site-packages/talos/model/layers.py in hidden_layers(model, params, last_neuron)
     68     for i in range(params['hidden_layers']):
     69
---> 70         model.add(Dense(layer_neurons[i],
     71                         activation=params['activation'],
     72                         use_bias=use_bias,
IndexError: list index out of range

Thank you!

Hi,
I have also noticed the same inconsistency, but looking into the network_shape.py clarified that there are only 4 options: 'brick', 'funnel', 'triangle' and a float number (smaller than 1).
Passing a string or value that is not defined in that list, will return an empty list, and in layers.py, then  layer_neurons = [] which as you can see causes problem:
[...]
    layer_neurons = network_shape(params, last_neuron)

    for i in range(params['hidden_layers']):

        model.add(Dense(layer_neurons[i],
[...]

So if you remove 'long_funnel' and add 'funnel' your code should work. But I still don't know what does shape argument in the dictionary represent, I am using talos.__version__ = '0.4.9'. However, in 0.6.0 there are changes in the arguments' names which would probably cause a lot of problems when one wants to use the newer version.

Thanks @Peon26, that's right. The documentation is not up-to-date, and will be updated to match v.0.6.0 once that makes it's way to pypi.

However, in 0.6.0 there are changes in the arguments' names which would probably cause a lot of problems when one wants to use the newer version.

While there are notable changes and improvements, the Scan argument changes are going to be very straightforward.
Closing here.

authors: Peon26 kateruksha mikkokotila

repo: autonomio/talos


text:
DevOps Dojo: CI - JSON Schema Validation of YAML Parameter File(s)

Checklist

[X] I've read the contribution guidelines.
[X] I've searched other issues and no duplicate issues were found.
[X] I've agreed with the maintainers that I can plan this task.

Description
JSON Schema validation to ensure that parameter file(s) are valid.
Purpose
Ensure that parameter files are valid to mitigate misconfiguration of ROS nodes.
Possible approaches
E.g., https://github.com/marketplace/actions/frontmatter-json-schema-validator
Definition of done
Pass/Fail for the JSON Schema validation using draft-07.
Pass/Fail for /<path_to_package>/schema/*.schema.json validation of parameter file(s) in /<path_to_package>/config/*.param.json.

@xmfcx @mitsudome-r
I spoke to @ambroise-arm earlier today. Arm is willing to take this issue on and with your input and feedback, add the validation into GitHub Actions.

https://github.com/marketplace/actions/json-yaml-validate This action might be a better option since https://github.com/marketplace/actions/frontmatter-json-schema-validator is for YAML front matter. Maybe we can also use the same action to validate the JSON files too.

https://github.com/autowarefoundation/autoware.universe/pull/4002 is a PR for validating the JSON Schema files against their meta-schema. This is the first half of the work. The other half is validating the yaml configuration files against the schema files, I still need to look at that.

autowarefoundation/autoware.universe#4002 is a PR for validating the JSON Schema files against their meta-schema. This is the first half of the work. The other half is validating the yaml configuration files against the schema files, I still need to look at that.
EDIT: I am not using the suggested json-yaml-validate for that because it is designed to take in a single schema to validate multiple files against. Whereas our usecase is to have multiple pairs of yaml/json where we want to validate each yaml config file against its associated json schema.

There might be cases where the are more than one *.param.yaml files in the config/ directory.

Find the GitHub action running here: https://github.com/autowarefoundation/autoware.universe/actions/workflows/json-schema-check.yaml

@kaspermeck-arm @ambroise-arm I found that current CI only allow single schema and configuration file. Therefore, I created the PR to fix this. Could you check that?

I don't think a package is meant to have several schema files. But it can have several different configuration files that comply to its schema, which I think your PR would not allow to check anymore.

Ah, make sense. So, each package can only have single schema file even if it has multiple configuration files, right?
Then I will close the PR.

So, each package can only have single schema file even if it has multiple configuration files, right?

It is rather "each package can have multiple configuration files, as long as they comply with a single schema file", at least currently.
But the situation in https://github.com/autowarefoundation/autoware.universe/pull/4902 looks like a valid use case for having several schemas. I don't know if you can make it work another way.

@ktro2828 @ambroise-arm
Could a solution be to have a nested schema and parameter file?

@ktro2828 @ambroise-arm
Could a solution be to have a nested schema and parameter file?

I'm in discussion with @ambroise-arm and we'll have an proposal soon which should resolve this issue.

We propose having:

one schema file per node, with a NODE_NAME.schema.json name (which is already the current design, except that the design expected one node per package)
one or more configuration files per node, with a NODE_NAME*.param.yaml name

So the delta to the current design would only be a a stricter naming convention for the yaml files. In the current main branch I think only autoware_auto_msgs_adapter would need a modification with the renaming of its yaml files. And I think it is also only a small deviation to what you currently have @ktro2828

@ambroise-arm  @kaspermeck-arm
The current validation mechanism poses issues in the following cases:

There are parameters that change in response to ros2 launch runtime arguments, and these need to be passed from the launch.xml using <param name=.
param.yaml is split into multiple files.
A single package has multiple node implementations, and each node has its own distinct param.yaml.

For example, pose_initializer.launch.xml corresponds to case 1, occupancy_grid_based_validator.launch.xml to case 2, and perception/image_projection_based_fusion to case 3.
In a discussion within this PR, the idea was proposed to write all parameters in param.yaml and override some of them in launch.xml using <param name=.
However, due to the complexity of the override rules, this approach may potentially introduce more bugs.
I have created tests to verify the parameter override rules; you can take a look at them here: GitHub - KYabuuchi/ros2_launch_learn at feat/minimum_case.
I believe we should consider the measures has pointed out before.
https://github.com/autowarefoundation/autoware.universe/pull/5773#issuecomment-1844664372

Define only the schema file and do not validate the parameter file
Split the schema file into expected division units of parameter file
Create a dummy parameter file that is not actually used but for validation (because it is unclear which parameter in the launch file or the parameter file will be used)
Create a mechanism to validate the temporary merged parameter file given to the node by the ROS launch system
Remove "required" field from schema file

@KYabuuchi thanks for your comment and helping us create a more rigid way with the parameters aspect of Autoware.
I'm not quite following your thought process and I don't understand where the complication is. I've tried to describe the process in a few bullet points:

A ROS package can have none or many nodes
Each node has one schema associated with it
A node has one or more parameter files associated with it
The schema validates the parameter file(s)

Am I addressing the issue with these bullet points?
We have not yet started the  DevOps Dojo: ROS Node Launch. I do believe that overwriting parameters can be useful but also has risks, as you also mention. As soon as a parameter is configured outside of the parameter file, it cannot be validated by the schema, or at least not with how we validate parameter files today.
@ambroise-arm

@kaspermeck-arm  Thank you for describing.
I noticed that a PR has been developed to address case 3 of my concerns. This will eventually be resolved. :+1:
However, the bullet points do not consider nodes that load multiple parameter files. Some nodes store their parameters in separate files and load multiple parameter files during launch.
For example, probabilistic_occupancy_grid_map loads two files.

Additionally, parameter overrides within launch files might lead to misunderstandings and bugs. Nevertheless, removing parameters determined at startup is challenging, so it would be helpful to have a method for handling this on the validation side.

@KYabuuchi
Just to understand. Why is it desirable to provide multiple parameter files to launch a single node?

@kaspermeck-arm

Why is it desirable to provide multiple parameter files to launch a single node?

In cases where developers want to prepare presets corresponding to several modes, they create multiple parameter files.
For example, the probabilistic_occupancy_grid_map has parameter files that vary based on observation methods and another set of parameter files for independent update rules.
If all parameters are stored in a single file, the number of files becomes large and difficult to maintain when there are many preset combinations.

Moreover, when some parameters are shared with other nodes, multiple parameter files may also be passed to a node.
For example, both the lane_departure_checker and vehicle_cmd_gate nodes require parameters such as the vehicle's wheel_base and max_steer_angle. These parameters are stored in the vehicle_info.param.yaml file.
These nodes, in order to read both the vehicle-related parameters and their specific parameters, load multiple parameter files.

When would you want to determine parameters at startup?

The paths for maps and DNN models and the localization mode are provided by the user during startup.
These parameters need to be provided not only to a single node but also to several nodes. Therefore, they are not managed in parameter files but are passed to the nodes from the launch file using <param name= instead of <param from=config.yaml.

@KYabuuchi

Thanks, that makes sense. Removing the parameters from being required is one way to resolve this. @ambroise-arm - what do you think?
Would it be possible to set this to an environment variable?
Schema:

        "test_env_var": {
          "type": "string",
          "description": "Environment variable",
          "default": "${ENV_VAR}",
          "enum": ["${ENV_VAR}"]
        }

and parameter file
    test_env_var: ${ENV_VAR}

Removing the parameters from being required is one way to resolve this. @ambroise-arm - what do you think?

Yes, It is one way to resolve this.
But I think the launch_ros rules in https://github.com/ros2/launch_ros/blob/humble/launch_ros/launch_ros/actions/node.py#L175-L182 are not that complex. And they would allow to keep all schema fields as required, even if some of the parameter from the config file are meant to always be overridden. In order to avoid bugs, I think the only thing we need to ensure is that the default parameter of package uses a "**/" namespace, which should already be enforced in the schema files (for example: https://github.com/autowarefoundation/autoware.universe/blob/main/perception/lidar_apollo_segmentation_tvm_nodes/schema/lidar_apollo_segmentation_tvm_nodes.schema.json#L87). It may indeed hurt clarity, as changing some of the values of the package's parameter file will have an effect, but not for others. But I think it is already not clear where parameters are set and how they are propagated in launch files, so it seems like a weak issue to me.
Also about validating multiple parameter files, I think the goal was never to validate the parameter files held by the launch packages. So I don't think multiple parameter file is an issue beyond validating those default parameters files. Please someone correct me if my assumption is incorrect.
And keeping all fields required seems especially important if we intend to rework the launch files in the future, as it keeps things clean and consistent.
(but also I am not in charge of maintaining the launch and parameter files working well together, and I understand the reluctance to change what currently works)

@ambroise-arm  @kaspermeck-arm

Would it be possible to set this to an environment variable?
test_env_var: ${ENV_VAR}

Sorry, is this method actually valid for ros2 node? I have never seen this approach, and when I tried it, the variables were passed as strings without being interpreted.
If we want to reference environment variables during the startup,  it is more common to use the $(env ) command within the launch file. And I do not think this would work in a parameter file.
Even if it is a valid method, I also do not think we want to introduce a new way to set parameters.

@ambroise-arm @KYabuuchi
I suggest the following:

Let's keep the the required fields in the schema and it will validate a default parameter file. For testing purposes, having a single complete parameter file is important. Then allow for parameters to be overwritten in the launch file. Does this sound OK?

Ok, let's pass on my ${ENV_VAR} idea. For variables which use environmental variables, let's rely on the $(env ) command. E.g.,

Schema:
        "data_path": {
          "type": "string",
          "description": "Absolute path to data and artifacts packages directory.",
          "default": "abs_path_to_data_and_artifacts"
        }

and parameter file:
/**:
  ros__parameters:
    data_path: abs_path_to_data_and_artifacts

Which will be overwritten using:
<arg name="data_path" default="$(env HOME)/autoware_data">

Does this sound OK?

is this method actually valid for ros2 node?

@KYabuuchi Yes, you just need to allow the launch file to interpret it correctly. See yolo_v2_tiny_example.param.yaml and allow_substs="true" in  yolo_v2_tiny_example.launch.xml

Ok, let's pass on my ${ENV_VAR} idea. For variables which use environmental variables, let's rely on the $(env ) command.

@kaspermeck-arm I think the point was that those $... were meant to be used in the launch file, as opposed to the parameter file. Not ${...} vs $(env ...).
For the data path of the artifacts, I think the yolo_v2_tiny links above are also a very good example of what can be done then.
But then I think there is a distinction between using $HOME or $(find-pkg-share ...) in a parameter file (which don't require the user to do anything and I think are quite useful to have in parameter files), and creating custom new environment variables (which is what I thought we were talking about above and I am against doing)

@KYabuuchi - I aligned with @ambroise-arm and this is our proposal.
All parameters which are expected to be used in a node must exist in the schema and parameter file according to the contribution guide. This is necessary for rendering the table view in the documentation and for testing purposes. There is nothing preventing  multiple parameter files in the launch file.
The yolov2 tiny example resolves the environmental parameter issue. In the schema

yolo_v2_tiny_example.schema.json

the env command is used:
        "data_path": {
          "type": "string",
          "default": "$(env HOME)/autoware_data",
          "description": "Packages data and artifacts directory path."
        }

And the corresponding parameter file

yolo_v2_tiny_example.param.yaml

/**:
  ros__parameters:
    ...
    data_path: $(env HOME)/autoware_data

Finally, in the launch file

yolo_v2_tiny_example.launch.xml

the following line needs to be added:
<param from="$(var tvm_utility_example_node_param_path)" allow_substs="true"/>

Note allow_substs="true", which enables the commands from the parameter file to be resolved.
Does this solve the problems which you're facing?

@kaspermeck-arm   It is difficult for me to judge whether that method is appropriate or not.
If you are going to proceed with using new environment variables, it would be better to summarize the modifications and ask for feedback not only in this thread but also in the discussion space. :pray:

@KYabuuchi - that's a good point that this could be a larger discussion. I'll bring this topic up in the next OADK meeting!
In my opinion there is no technical difference to setting the environmental parameters in the parameter file vs the launch file. It is however more transparent and follows the current contribution implementation details to do this in the parameter file.

When and who performs the variable substitution here? As late as node initialization or at an earlier step?

@doganulus - my understanding is that it's resolved when running the ros launch command, so during node initialization.

Launch configuration can also be used. I think this is easier to use because it has a narrower scope than environment variables.
test.param.yaml
/**:
  ros__parameters:
    foo: $(var foo)
    bar: data-bar
    baz: data-baz

test.launch.xml
<group>
  <let name="foo" value="test-foo"/>
  <node pkg="test_pkg" exec="node" name="node">
    <param from="$(find-pkg-share test_pkg)/config/test.param.yaml" allow_substs="true"/>
  </node>
</group>

output
foo: test-foo
bar: data-bar
baz: data-baz

I am leaning towards that keeping *.param.yml node parameter files free of any variable substitution as this means the implicit use of ros launch, which performs the substitution. If we want to use Eclipse Bluechi or Ankaios at some point, these tools probably would not support variable substitution as ros launch.
So the following solution would be more robust if my last assumption is true.
First, we have a pure yaml parameter file.
# test.param.yml
/**:
  ros__parameters:
    foo: /opt/autoware/<component>/<node>/config/test.param.yaml # Hardcoded conventional path
    bar: data-bar
    baz: data-baz

Then, I assume parameter overwrite will work as follows (needs to be verified):
# test.launch.xml
<group>
  <node pkg="test_pkg" exec="node" name="node">
    <param from="$(find-pkg-share test_pkg)/config/test.param.yaml" allow_substs="true"/>
    <param name="foo" value="test-foo"/>
  </node>
</group>

@doganulus - I'm not following your reasoning regarding Bluechi and Ankaios. These are orchestrators, like k3s. Are you suggesting that these will remove the need to use the ros launch command entirely?

@kaspermeck-arm Yes, in the long run though. Those projects are at early stages but I think they can start ROS/Zenoh executables or containers directly.

@doganulus - moving away from using ros launch to launch ROS nodes is something which I've never considered and I don't know how that would work in practice. I think this topic in and of itself is a way bigger topic of discussion.
The experience I have with BlueChi is that it is Kubernetes yaml compatible, translates these files to systemd service files and then runs containers as systemd services. When the container is launched, a command/file inside that environment is run (e.g., entry point file) and for ROS that would be the ros launch command. Can you share more information on how these orchestrators will address launching ROS nodes without using the ros launch command?

I think this topic in and of itself is a way bigger topic of discussion.

Definitely needs a bigger discussion. Below is a starting example.
For the example, let's work in a test container:
docker run --rm -it ros:humble

And we install some example ROS nodes inside the container, create an example param.yml file, and start the ROS node parameter_blackboard directly as follows:
apt update && apt install -y ros-humble-demo-nodes-cpp
printf '/**:\n  ros__parameters:\n    a_string: $(env HOME)/autoware_data\n' >> param.yml
/opt/ros/humble/lib/demo_nodes_cpp/parameter_blackboard --ros-args --params-file param.yml & # Launch the node directly
ros2 param get parameter_blackboard a_string                                                 # Check the parameter set

Normally, the command ros launch is a convenient way to start (many) nodes via launch files but we also can start nodes directly as in the example. And here we are able to set parameters using --params-file argument. First observation, starting nodes directly does not substitute variables as you can see in the output. I think it is a feature of ros launch. Therefore, I suggest not using variable or shell commands in the parameter files, which can be used outside ros launch.
Then, if we can start nodes directly in this way, systemd can start them via unit files. And this is the connection with other projects. I think BlueChi can orchestrate all ROS node unit files all at some point.

@doganulus
I understand your point of not being dependent on ros launch and I think this is fair. However, not using variable/shell commands in the parameter files complicate things. This is a very useful feature of ROS as hard-coded environmental variables would be very tedious to change and not flexible, i.e., going against our goal of making Autoware software-defined.
How about we find a way to evaluate these variable/shell commands and then write the output to a new interpreted parameter file which is free of variable/shell commands and with the correct, e.g., paths to local directories. What do you think?

I am not absolutely against parameter file templates and environment variables in parameter files. They can be very handy when they are used right.
The current implementation does not feel right, as parameter files reside at a lower level than launch descriptions. Hence, parameter files should not be aware of specific ros launch features. If we could bring the substitution functionality to the lower level, then my objection would be resolved.
But this is not an easy task to handle properly. So I do not suggest immediate action as we gonna use ros launch for some time anyway. Still, avoiding variables as much as possible would be better. Setting some path conventions in the OpenAD Kit may help reduce the need for configuration.

@doganulus
In the end, Autoware is a ROS project and I am in favor of using the features which ROS provides to make it more convenient to configure nodes. The situation with adding a schema was different as there was no solution to this built into ROS already. I suggest that if someone has different needs, i.e., no dynamic variables in the parameter file, then they can make these changes themselves with, e.g., a parameter resolving script. If you want to bring this topic further, I suggest you create a new discussion with all the background.

The clash between ROS practices and modern cloud-native practices is unavoidable. I side with the latter when they overlap.
As a compromise: We can at least restrict it to $VAR or ${VAR} syntax for environment variables. That can be substituted by external tools like envsubst. We don't have this opportunity for $(find-pkg-share) and friends.

authors: KYabuuchi ambroise-arm doganulus isamu-takagi kaspermeck-arm ktro2828 xmfcx