text:
Unicode error

I got this error when running setup:
Traceback (most recent call last):
  File "setup.py", line 5, in <module>
    long_description = fh.read()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 641: ordinal not in range(128)

to solve I did this
export LC_ALL=en_US.UTF-8
then re-ran and it worked.

I too received a unicode error, although at the stage of trying to run a subset of my actual samples (no errors arose during install, set-up, or testing as outlined).  Below is the output from my terminal:
eukdetect --mode runall --configfile Metagenomics_I-Ching_Nachbac_configfile_TESTING_v1.yml  01/24/2023 18:06:24:  Parsing config file ... Traceback (most recent call last):   File "/share/jwaters/anaconda3/envs/eukdetect/bin/eukdetect", line 33, in <module>     sys.exit(load_entry_point('EukDetect==1.0.1', 'console_scripts', 'eukdetect')())   File "/share/jwaters/anaconda3/envs/eukdetect/lib/python3.6/site-packages/EukDetect-1.0.1-py3.6.egg/eukdetect/runall.py", line 140, in main   File "/share/jwaters/anaconda3/envs/eukdetect/lib/python3.6/site-packages/EukDetect-1.0.1-py3.6.egg/eukdetect/runall.py", line 454, in check_readlen   File "/share/jwaters/anaconda3/envs/eukdetect/lib/python3.6/site-packages/Bio/SeqIO/__init__.py", line 611, in parse     for r in i:   File "/share/jwaters/anaconda3/envs/eukdetect/lib/python3.6/site-packages/Bio/SeqIO/QualityIO.py", line 1033, in FastqPhredIterator     for title_line, seq_string, quality_string in FastqGeneralIterator(handle):   File "/share/jwaters/anaconda3/envs/eukdetect/lib/python3.6/site-packages/Bio/SeqIO/QualityIO.py", line 897, in FastqGeneralIterator     line = handle_readline()   File "/share/jwaters/anaconda3/envs/eukdetect/lib/python3.6/encodings/ascii.py", line 26, in decode     return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 12: ordinal not in range(128)
Any advice on how to proceed would be greatly appreciated! I did try the 'export LC_ALL=en_US.UTF-8' that hyphaltip recommended, but that did not resolve my issue and I had the same error.

Thanks for reaching out. I can't reproduce this error and I've not been able to figure out a solution for now, though I'm still trying. It looks like the error is being thrown by BioPython in one of the check steps. Could you try running this as a snakemake pipeline directly? Full instructions are on the github, but in short it's snakemake --snakefile [path_to_install_folder]rules/eukdetect_eukfrac.rules --configfile [config file] --cores [cores] runall. If that works that will be a workaround.

Thanks so much for your prompt reply! My apologies that I missed in the documentation that using snakemake can be a good workaround if running into issues with python.
I've tried running snakemake as you suggested, but I received the following errors while it was running. It did complete, but it seems the output files are empty.
/usr/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = "en_US.UTF-8",
        LANG = "C.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("C.UTF-8").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = "en_US.UTF-8",
        LANG = "C.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("C.UTF-8").

Hi Abigail,
After adding quotes to try and change the local settings export LC_ALL="en_US.UTF-8" that might have worked, or at least I am no longer receiving locale warnings (apologies that I did not correctly understand the error, I promise I tried to google it!).
However, now when I run the snakemake file, I receive this output to the screen and my output files again appear to be empty. Can you advise what may be going on?
Building DAG of jobs...
Nothing to be done.
Complete log: /share/jwaters/Metagenomics_I-Ching/EukDetect/.snakemake/log/2023-01-27T183421.100804.snakemake.log

Hi,
Thanks for your patience with a response. This suggests no steps of eukdetect ran second time because the snakemake process didn't notice any changes (missing the UTF encoding change). The best way to fix this is to delete the eukdetect output files, including the intermediate steps, and rerun the whole thing.

authors: allind hyphaltip jayelldubya

repo: allind/EukDetect


text:
HTML report /favicon.ico 404

You may notice that every time you open up the html report generated by the goaccess in the browser, your browser will request a favicon.ico file from the root uri.
The demo site also have this issue.
When the image/x-icon is not set in the html file, the browser will try to request a favicon.ico file from "https://rt.goaccess.io/favicon.ico" which may cause a 404.And it takes up time.
Many of us won't put the favicon.ico in "/", instead it will be put somewhere else, the path will be set by the html file like
<link rel="icon" href="somewhere/favicon.ico" type="image/x-icon" />

for goaccess maybe a better solution is to base64 the favicon.ico file and embed it into the html report file, the code may looks like this:
<link href="data:image/x-icon;base64,AAABAAEAEBAQAAEABAAoAQAAFgAAACgAAAAQAAAAIAAAAAEABAAAAAAAgAAAAAAAAAAAAAAAEAAA
AAAAAADGxsYAWFhYABwcHABfAP8A/9dfAADXrwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAIiIiIiIiIiIjMlUkQgAiIiIiIiIiIiIiIzJVJEIAAAIiIiIiIiIiIiMyVSRC
AAIiIiIiIiIiIiIRERERERERERERERERERERIiIiIiIiIiIgACVVUiIiIiIiIiIiIiIiIAAlVVIi
IiIiIiIiIiIiIhEREREREREREREREREREREAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" rel="icon" type="image/x-icon" />

That's awesome. Thanks for sharing this. Would you be able to submit a PR for this?

done. the code worked on my mac.

source: gharchive/issue

authors: allinurl dragonandy

repo: allinurl/goaccess

license: mit

license_type: permissive

license_source: bigquery


text:
Blank real-time-html page

Hi,
I am using v.1
Trying to check out the real-time-html feature, however can't get it working.
Command output:
[root@sphere ~]# goaccess -f /var/log/httpd/access_log -o report.html --real-time-html --port=8000 --addr=10.22.0.254
Parsing... [3495] [0/s]

Does it supposed to be like that? Or should there be additional output?
Socket seems to be opened after that:
[root@sphere ~]# netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:445                 0.0.0.0:*                   LISTEN      0          1517011260 929/smbd
tcp        0      0 0.0.0.0:9022                0.0.0.0:*                   LISTEN      0          1517009667 594/sshd
tcp        0      0 10.22.0.254:8000            0.0.0.0:*                   LISTEN      0          1517584169 1584/goaccess

Starting Nmap 7.12 ( https://nmap.org ) at 2016-06-14 05:33 EEST
Nmap scan report for 10.22.0.254
Host is up (0.040s latency).
Not shown: 993 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
80/tcp    open  http
139/tcp   open  netbios-ssn
445/tcp   open  microsoft-ds
3306/tcp  open  mysql
8000/tcp  open  http-alt
10000/tcp open  snet-sensor-mgmt

Nmap done: 1 IP address (1 host up) scanned in 0.46 seconds

I can access the page, but it appears completely blank. Any ideas?
The urls I am using for access:
http://10.22.0.254:8000
http://10.22.0.254:8000/report.html

What you posted above looks right. Have you tried outputting only the report?, e.g.,
goaccess -f /var/log/httpd/access_log -o report.html

Yep, it works just fine.

Also, if goaccess is running on a different machine, you probably want to use --ws-url=server.addr otherwise, it will attempt to connect to localhost. You can also try getting rid of --addr.
One more thing, check the browser console or dev tools, it may be outputting something.

Just to clarify, should I place report.html in web servers root, or it can be anywhere and the report is accessible via socket?

The report can be anywhere, goaccess will send the data via socket. So once the report has been generated, it will attempt to establish a connection to whatever address was given on --ws-url=<addr>.

Ok, so:
10.22.0.254 is simply a server where goaccess is installed. It is accessible via vpn, 10.22.0.254 is its VPN ip.
So I made a report with:
goaccess -f /var/log/httpd/access_log -o report.html --real-time-html --port=8000 --ws-url=10.22.0.254 --addr=10.22.0.254
And downloaded it to my pc. It works, but the report information does not update. The green led near the settings icon in the dashboard is on. If I cancel the command on the server, it goes off :)
Attached is also the screenshot from Firebug. The request to goaccess gets a response, but the info does not update.

Any ideas?

My bad again..Everything works fine!
I did not realise, that the info is refreshed only on new log entry.
The "Last updated field" confused me, so I was refreshing the page and wondering why it is not working.
Wonderfull peace of software. You can close this ! :)

Great to hear that :)
Actually I'm working on an issue similar to what you described above, where in some cases, it is possible for a client not get the data from the server, and therefore, the buffer not being flushed. So I expect having a new minor release soon. Stay tuned!

Sorry I have the exactly the same issue
I did this
goaccess -f /var/log/nginx/access.log -a -o report.html --real-time-html --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u"' --date-format='%d/%b/%Y' --time-format='%T' --ws-url=xxx.xxx.xx.x --addr=xxx.xxx.xx.x --port=7700

I keep get blank page
check in dev tool
Request URL:http://xxx.xxx.xx.x:7700/
Request Method:GET
Status Code:400 Invalid Request
Remote Address:xxx.xxx.xx.x:7700
Any ideas?

@zt1983811 The "Request URL" in your dev tools should look like ws://example.com:9870/.
It's a WebSocket connection and it shouldn't contain HTTP on it. The requested url is set by  --ws-url=example.com --port=9870

@allinurl  I am sorry I try to access ws://example.com:9870/ from dev tool or browser I can not .
Could you explain me how it suppose to work?
Do I need to create client to consume this ws://example.com:9870/ ? or it can be directly access by http and show the report page like you did?
Thanks in advance!

Upon opening the generated report in your browser, it tries to establish a WebSocket connection to the address specified by --ws-url=<address>. Note that the address part should point to the location where GoAccess is running and parsing logs.
If you don't use --ws-url, it will attempt to establish the WebSocket connection to localhost, which means that GoAccess should be running on your local machine (the same machine as the one with the browser)
By default, the WebSocket server listens on port 7890, so the server running GoAccess should have the port open. Optionally, if you need to use a different port you can specify it by using --port=<port_number>
So the following example, assumes that GoAccess is running and parsing logs on a remote server, the port 7890 is opened, and your browser is on your local machine.
goaccess -f access_log -o report.html --real-time-html --ws-url=host.myserver.com

Please note that the address/host that you use in --ws-url should not be of the type http://. If it makes things easier, you can try using the server IP instead.

Hi @allinurl
I think I follow exactly your instructions, however I got 404 for some reason.
goaccess -f /var/log/nginx/access.log -a -o report.html --real-time-html --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u"' --date-format='%d/%b/%Y' --time-format='%T' --ws-url=xxx.xxx.xx.x --addr=xxx.xxx.xx.x --port=7700
the xxx.xxx.xx.x is ip address of remote server, and the log is also in this remote server, the command above is running in that remote server too. And I try use my local computer to access http://xxx.xxx.xx.x
I got 404
I try to access ws://xxx.xxx.xx.x page can not even load from browser
If  you want I can provide you the real ip in private we can test together!
Appreciated your help.

One more thing to note, GoAccess does not have a webserver, only a WebSocket server.
So your report should be behind an Apache or Nginx in order to open it from a remote location. However, since the report is just a plain html file, you can download the generated report to your local machine and open it there. i.e., file:///home/user/report.html.

@allinurl
Could you show us how you did it in your website to make it always updating http://rt.goaccess.io/?20160616115152
I want to do exactly the same , I have nginx server installed here

First, I run goaccess to parse the access log and place the one-time-only generated report under the web server www public folder. Here's how is done:
goaccess -f access.log -o /usr/share/nginx/www/rt.goaccess.io/index.html --real-time-html --ws-url=ws.goaccess.io

Then, when you or anybody else access http://rt.goaccess.io/, your browser will establish a WebSocket connection to ws.goaccess.io.
That's all.
If that doesn't work, shoot me an email to hello  [at] goaccess . io and I'll take a look at it.

Please write this ^ in the README. Took me some time to get here and understand this.

@bilalba Thanks, I've added some details to the README.

@allinurl As a feature suggestion: It would be better if a default HTTP request to the server sent the HTML report and you could use a relative url to form a WS connection with the server.

@bilalba what do you mean with the server should send the HTML report? Currently a JSON object with the latest data is fetched as soon as the connection is opened.

@allinurl Something like detecting if an HTTP request has been made to the server and responding with the report.html. Because currently, if an HTTP request using curl or a browser is made, it responds empty with a 400 error code.

HTTP/1.1 400 Invalid Request
no chunk, no close, no size. Assume close to signal end

Perhaps this should help. http://stackoverflow.com/questions/13791050/is-it-possible-to-enable-tcp-http-and-websocket-all-using-the-same-port

@bilalba I'm not sure I follow. What would be the use case for sending a request to get the HTML report? wouldn't you be able to just open it after it has been generated?

@allinurl This is in case goaccess is running on a remote server; it's more convenient to open(and share) an HTML page through a URL rather than through the file. Right now, sharing the report is only possible by hosting it separately somewhere.

@bilalba Got it. I'll check on this.

I think the confusion comes in, i know it did with me - in that I thought the web server is serving the HTML page and you visit it and get the report. Which gives me a blank page.
Whereas the web server is just populating data for the HTML file it creates, which you download and open on your browser and it auto-updates via the web service.

authors: allinurl bilalba rurapenthe tomasliumparas zt1983811


text:
select下拉框应可以支持输入值

虽说下拉框是直接选择进行输入，但还是有输入和选择共同进行的输入方式的需求的

你这个是不是跟select2那种风格的 或者chosen那种风格的？

可以把 select 和 https://github.com/amazeui/tagsinput 结合使用，下拉框选中以后把值自动同步到 tagsinput 里面。如果需要用户直接输入的，就在 tagsinput 里面输入。

Hello!
You have a new message, please read http://watersedgemedia.com/landmark.php?zhnie
Again_moon@sohu.com

authors: 2070019559ysh againshao hainuo minwe

repo: allmobilize/amazeui

license_source: github-api


text:
feat(protocol): Batch Reader

Description
Introduces the batch reader type.
Fixes broken batch decoding.

📚 $\text{Stack Overview}$
Pulls submitted in this stack:

#234 👈
main

This comment was automatically generated by st.

Will come back to this once more batch type cleaning is done

Closing this for now - batch reading will strictly be used in the derivation pipeline so it shouldn't need to be used in other op stack components.

source: gharchive/pull-request

repo: alloy-rs/op-alloy


text:
Abilty to add parameters to step

I'm using allure-java-adaptor-api for gathering information about tests being running in Robot Framework.
I have steps added for every scenario's keyword in my report. Most keywords contains number of arguments. I have to list arguments in step title label. Looks not very good for me. Of course, I can use step attachments for arguments listing. But its looks like some kind of overkill.
Is it possible to add parameters to steps like it can be done for test cases? If not, have you any plans to implement extra labels for step?

@r-coder thanks for suggestion!

Is it possible to add parameters to steps like it can be done for test cases?

Nope, only using title =(

If not, have you any plans to implement extra labels for step?

Yep, in the future we plan to implement such feature.

+1 on our backlog as well

Almost done in allure2. See https://github.com/allure-framework/allure2/issues/93

authors: baev buzdin r-coder


text:
Collapse Menu Save State

Even I found issues of Collapse Menu Save State i found this sollution of my code
		<script type="text/javascript">
		if (Boolean(sessionStorage.getItem("sidebar-toggle-collapsed"))) {
			$("body").removeClass('sidebar-collapse');
			sessionStorage.setItem("sidebar-toggle-collapsed", "1");
		}else{
			$("body").addClass('sidebar-collapse');
			sessionStorage.setItem("sidebar-toggle-collapsed", "");
		}

		$('.sidebar-toggle').click(function() {
			event.preventDefault();
			if (Boolean(sessionStorage.getItem("sidebar-toggle-collapsed"))) {
				sessionStorage.setItem("sidebar-toggle-collapsed", "");
			}else{
				sessionStorage.setItem("sidebar-toggle-collapsed", "1");
			}
		});
		</script>

My question what is  sessionStorage as support most browser. thanks.

As discussed multiple times, this solution causes an issue in loading on IE10|11 where the sidebar would show on load and hide immediately. The best solution to this is to implement a server side state mechanism to store the current status of the sidebar.
Thanks!


text:
javascript parse data to modal issue

  $(document).on( "click", '.edit_button',function() {

        var rno = $(this).data('rno');
        var rname = $(this).data('rname');
        var rpoint = $(this).data('rpoint');
        var rremarks = $(this).data('rremarks');
        var rewcreate = $(this).data('rewcreate');
        var rewupdate = $(this).data('rewupdate');

        $(".modal-body #rno").val(rno);
        $(".modal-body #rname").val(rname);
        $(".modal-body #rpoint").val(rpoint);
        $(".modal-body #rremarks").val(rremarks);
        $(".modal-body #rewcreate").val(rewcreate);
        $(".modal-body #rewupdate").val(rewupdate);
    });

    <button class="btn btn-info edit_button" type="submit" name="submit" data-toggle="modal" data-target="#myModal" data-rno="'.$rno.'" data-rname="'.$rewname.'" data-rpoint="'.$rewpoint.'" data-rremarks="'.$rewremarks.'" data-rewcreate="'.$rewcreate.'" data-rewupdate="'.$rewupdate.'"></button>

i use the above method to parse data to modal... it works pretty well on other template...only in AdminLTE the script doesnt work..i've been googling for awhile but cant find a clue... can anyone point me to a proper direction where i can solve this issue?

Hello,
Could you provide the HTML code for your page?

here you go... dont comment on my php skill..im quiet new in php
http://pastebin.com/Ds2w5ERP

From the link you submitted, you forgot to add </script>.

ups sorry but after i put  it also not working...im using the clean bootstrap and its working but in your admin lte is not... im wondering if your other templates also like this? since i plan to get the premium template

This isn't a template problem. It's a bug in the code. Did you check the browser's console for any errors?

i got this error in console...  the 1st error is the error in $(document) in the javascript
2nd one is from moment.js
3rd is from DataTables.js

This says you didn't load jQuery. Could make sure you have all the resources loaded?

http://pastebin.com/HrbFVGzR
yup its loaded on the footer file :D

okay...i solved it already... i included hte JQuery librabry once again on the top of the file... so i included twice once on the footer together with others and then on the top of the file

You should move all of your JavaScript code to the bottom of the file after loading jQuery. Also, you should wrap your code in an anonymous function or jQuery's document.ready() function.
Good luck!


text:
Does not work st7789 on esp32

Logging this as a new bug (although in a way I suspect it might be my own fault). I apolagize in advance if I am misisng some small thing, and this is more a request for help than a bug report
I do have an ESP32 with the st7789 attached to it. Cirtcuit configuration can be found here: https://raw.githubusercontent.com/Fri3dCamp/badge-2020/master/media/Badge_Block.png
Some (setup) example code with Arduino can be found here https://github.com/Fri3dCamp/Badge2020_arduino/blob/main/Badge2020_TFT.h / https://github.com/Fri3dCamp/Badge2020_arduino
This code example works (just as sanity check that my hardware is working)
I am pretty sure my pin config is correct (5 for CS, 33 for DC and 26 for RST), as configured in example code from before. (Note 26 does not appear on the schema, maybe it does not have RST, but I tried it with st7789_without_rst as well.., also in working arduino code it has been set as wel
In all cases I just get a black screen and I am not sure why as I expect a cleaned red screen
I notice with this I have to configure my own SPI config which is somehow "magically"? handled when using Arduino/Adadruit library so maybe I am doing something wrong here... (I took inspiration from https://github.com/mutantbob/lcd-esp32)
  # Tried both MODE_3 and MODE_0
    let config = <esp_idf_hal::spi::config::Config as Default>::default().data_mode(embedded_hal::spi::MODE_3);
    let spi = esp_idf_hal::spi::Master::<esp_idf_hal::spi::SPI2, _, _, _, esp_idf_hal::gpio::Gpio23<esp_idf_hal::gpio::Unknown>>::new(
        peripherals.spi2,
        esp_idf_hal::spi::Pins {
            sclk: peripherals.pins.gpio18,
            sdo: peripherals.pins.gpio19,
            sdi: Some(peripherals.pins.gpio23),
            cs: None,
        },
        config,
    )
    .unwrap();
    let lcd_dc = peripherals.pins.gpio33.into_output().unwrap();
    let lcd_rst = peripherals.pins.gpio26.into_output().unwrap();
    let lcd_cs = peripherals.pins.gpio5.into_output().unwrap();
    let mut led_bl = peripherals.pins.gpio12.into_output().unwrap();
    led_bl.set_high();
    let di = display_interface_spi::SPIInterface::new(spi, lcd_dc, lcd_cs);
    println!("SPI CONFIG DONE");

    let mut lcd = mipidsi::Display::st7789(di, lcd_rst);
    let mut delay = esp_idf_hal::delay::Ets;
    println!("INIT S");

    lcd.init(
        &mut delay,
        mipidsi::DisplayOptions {
            orientation: mipidsi::Orientation::PortraitInverted(false),
            invert_vertical_refresh: false,
            color_order: Default::default(),
            invert_horizontal_refresh: false,
        },
    ).unwrap();

    println!("INITED");

    lcd.clear(Rgb565::RED);
    println!("CLEAR");

   # I am joining on another thread here to make sure my program does not exit

What is your SPI speed setting (frequency) in both C and Rust code? That's the one thing I don't see mentioned here. It's possible you're running it at too high of a speed. The ST7789 can do up to 60Mhz at times AFAICS but it gets flakey there, depending on the board you use tho.
If not that we could try comparing the init and if you have a logical analyzer capturing the SPI output to see if there's a logical or even a SPI level problem.

"What is your SPI speed setting (frequency) in both C and Rust code?"
C code => I have no idea, in my userland code I am not setting SPI settings, this seems all handled by Adafruit library (and even in there I could not find anything really)
Rust code => I stayed with defaults, so I dont specifically set anything.
At some point I tried "let config = <spi::config::Config as Default>::default().baudrate(24.MHz().into());" but that did not work either...
Note the example rust application esp32 https://github.com/ivmarkov/rust-esp32-std-demo which uses the st7789 driver also doesn't work for ....
It surely looks maybe I am just using wrong pins or something (or some other bad configuration, or both the st7789 driver and this driver have same bug)
Ill try a few more things as well, but if anyone has an idea where to look please let me know

Okay I finally found something
Look in https://gist.github.com/nathansamson/8ff4efbcd11043bcfda6a3e7cf2d01ea for details
There are 3 files in there

Cargo.toml (this is for the st7789 version but can easily be modified by swapping dependencies for mipidsi version)
mipidsi.rs  this version does NOT work, it shows some greyish background on the screen but nothing more
st7789.rs It differs only to the previous version how it initializes the lcd, inthis case via st7789. All drawing or SPI operations are the same. This WORKS (displays blue background with 2 squares, green with a red border, as expected)

So the breaktrough since previous time was that I was usingt he wrong SPI (2 vs 3) and my pins where out of order...
So now I know my config finally works I think this should fall under bug as with same config st7789 library works but this one doesn't (although it claims it should).
Funliy enough it does not seem to be exactly like https://github.com/almindor/mipidsi/issues/22 there using MODE_3 solved the problem, in my case MODE3 with mipidsi gives a blank screen, and with MODE0 a grayish screen (but certainly not the blue with 2 squares as I expect it)...

Okay I finally found something
Look in https://gist.github.com/nathansamson/8ff4efbcd11043bcfda6a3e7cf2d01ea for details
There are 3 files in there
* Cargo.toml (this is for the st7789 version but can easily be modified by swapping dependencies for mipidsi version)

* mipidsi.rs  this version does NOT work, it shows some greyish background on the screen but nothing more

* st7789.rs It differs only to the previous version how it initializes the lcd, inthis case via st7789. All drawing or SPI operations are the same. This WORKS (displays blue background with 2 squares, green with a red border, as expected)

So the breaktrough since previous time was that I was usingt he wrong SPI (2 vs 3) and my pins where out of order...
So now I know my config finally works I think this should fall under bug as with same config st7789 library works but this one doesn't (although it claims it should).
Funliy enough it does not seem to be exactly like #22 there using MODE_3 solved the problem, in my case MODE3 with mipidsi gives a blank screen, and with MODE0 a grayish screen (but certainly not the blue with 2 squares as I expect it)...

Ok so if I understand correctly, using the "proper" SPI instance works with the ST7789 driver, but still fails with the MipiDSI one.
When it comes to ST7789, which MODE works for you? Can you try lowering your SPI clock to something super slow like 100khz (you'll see individual pixels being pushed, should it work out)?
AFAICS the only difference between the inits of those two drivers (I wrote both of them :D) is the MADCL/Rgb-Bgr flip (if using defaults).

Normal Hz (32Mhz)
ST7789 driver

MODE 0 works
MODE 3 works

mipidsi driver

MODE 0 displays a grayish background but no colors are displayed
MODE 3 keeps black screen

Slow Hz (20000.Hz().)
ST7789 driver

MODE 0 works, pixels are drawn slowly from top to bottom, starting on grayish background (same as non working MODE 0 for mipdsi above)
MODE 3 works, same as mode 0

mipidsi driver

MODE 0 displays a grayish background but no colors are displayed
MODE 3 keeps black screen

Recap of initialization (st7789)
    let mut lcd = st7789::ST7789::new(di, lcd_rst, 240, 240);
        lcd.set_orientation(st7789::Orientation::Portrait).unwrap();

        let mut delay = esp_idf_hal::delay::Ets;
        lcd.init(&mut delay).unwrap();

initialization of mdpsi, note the settings that I kind of guessed but for me should worst case result in inverted image and/or wrong colors
let mut lcd = mipidsi::Display::st7789(di, lcd_rst);
    let mut delay = esp_idf_hal::delay::Ets;
    println!("INIT S");

    lcd.init(
        &mut delay,
        mipidsi::DisplayOptions {
            orientation: mipidsi::Orientation::PortraitInverted(false),
            invert_vertical_refresh: false,
            color_order: Default::default(),
            invert_horizontal_refresh: false,
        },
    ).unwrap();

Could you try st7789@v0.7.0 as well? I just want to make sure it still works.
This is really perplexing, the only init difference AFAICS is that mipidsi has a "dynamic" MADCTL based on config (e.g. orientation, color swaps etc.) and "inverts" the RGB by default (which might or might not be needed, it's another odd point).
I'm comparing https://github.com/almindor/st7789/blob/master/src/lib.rs#L127-L151 to this
The only diffs I can see:

let madctl = options.madctl() ^ 0b0000_1000; line which should at worst confuse the colors.
hard_reset being used in all cases with ST7789 while it's a union with sw reset on mipidsi (should be correct that way)
Longer final delay after DISPON in mipidsi

If this is an init issue (without any kind of odd timing or other side-effect problem) then changing these in the mipidsi driver to match should make things work. Everything else is "generic" code that is the same.

I can confirm st7789 v0.7.0 works too
If you prepare a branch with the fixes to match st7789 I am happy to try it out to see if that one works and if indeed thats the cause...
If you don't have tine for that I might try that later (might only be in weekend)

I can confirm st7789 v0.7.0 works too
If you prepare a branch with the fixes to match st7789 I am happy to try it out to see if that one works and if indeed thats the cause... If you don't have tine for that I might try that later (might only be in weekend)

Could you please try with https://github.com/almindor/mipidsi/tree/st7789init ? Use DisplayOptions::default() for the options which should match madctl to 0x0 (same as st7789). This branch unifies everything else with the ST7789 driver, so if it still doesn't work we're probably seeing some deeper issue that might be semi-random.
One other thing that I didn't touch upon is the backlight pin. I suspect you don't use it though.

Ill give it a try later
Backlight pin: that shouldn't matter. As far as I understand its hardlinked with a physical switch (although the example arduino code for my board defines pin 12) -- correction: there has been some activity in that repo which fixes that misconception.
Note I also see some noise about the RST button, so maybe that also has an impact... Ill check it out later

I tried it out.
So couple of things

The main inprovement came from using "DisplayOptions::default()" and not

mipidsi::DisplayOptions {
            orientation: mipidsi::Orientation::PortraitInverted(false),
            invert_vertical_refresh: false,
            color_order: Default::default(),
            invert_horizontal_refresh: false,
        }

Once I used the defaults I could see the 2 small rectangles I was drawing.
Note the background color stayed grey while it should have been Blue lcd.clear(Rgb565::BLUE)

When using the 0.2.1 version with DisplayOptions::default() the colors are wrong though. I see a blue outline with a green background for each rectangle while it should be red outline and green background (Red & Green are switched but that does not fully surprise you so it seems)

So to me it seems 'lcd.clear' is not (properly) implemented?
And in the 0.2.1 version the color initialization is not "correct" (maybe there is no one correct and it really depends on the version you are getting...)
I am not sure  why my original DisplayOptions where incorrect though (I copied them from another example), looks to me worst case it would do drawing in the wrong orientation?

I tried it out.
So couple of things
* The main inprovement came from using "`DisplayOptions::default()`" and not

mipidsi::DisplayOptions {
            orientation: mipidsi::Orientation::PortraitInverted(false),
            invert_vertical_refresh: false,
            color_order: Default::default(),
            invert_horizontal_refresh: false,
        }

Once I used the defaults I could see the 2 small rectangles I was drawing. Note the background color stayed grey while it should have been Blue lcd.clear(Rgb565::BLUE)
* When using the 0.2.1 version with `DisplayOptions::default()` the colors are wrong though. I see a blue outline with a green background for each rectangle while it should be red outline and green background (Red & Green are switched but that does not fully surprise you so it seems)

Yes this is due to the madctl XOR operation I added (wrongly) in mipidsi thinking the ST7789 had the RGB bit wrong for some reason.

So to me it seems 'lcd.clear' is not (properly) implemented? And in the 0.2.1 version the color initialization is not "correct" (maybe there is no one correct and it really depends on the version you are getting...)

I'm not sure why clear doesn't work. It's odd if a rect works, clear should too.

I am not sure why my original DisplayOptions where incorrect though (I copied them from another example), looks to me worst case it would do drawing in the wrong orientation?

Using a different orientation changes the MADCTL register value. I suspect that there either is a bug or that the given setting draws "in the wrong direction" (off display area).
Just to be 100% sure I understand the results here. The background clear doesn't work in either case, but the drawable rect with border is drawn with wrong colors in both the branch and master if you use the default options, with inverted colors in the master branch, correct?

Just to be 100% sure I understand the results here. The background clear doesn't work in either case, but the drawable rect with > border is drawn with wrong colors in both the branch and master if you use the default options, with inverted colors in the master > branch, correct?

0.2.1 has wrong colors, st7789init branch has correct colors. (Both using default drawing options)
Neither of them have a full size background (created with "clear")

Just to be 100% sure I understand the results here. The background clear doesn't work in either case, but the drawable rect with > border is drawn with wrong colors in both the branch and master if you use the default options, with inverted colors in the master > branch, correct?

0.2.1 has wrong colors, st7789init branch has correct colors. (Both using default drawing options) Neither of them have a full size background (created with "clear")

Thanks for all the testing. Could you do one more thing please? Add a delay after your last render, and then call clear again to see if that works. I want to exclude "timing" issues with the clear call.

No that doesnt work either...
So I am initting, clearing (which doesnt do much), drawing 2 rectangles, sleeping 2 seconds, clearing again and still 2 rectangles show
Note using fill_solid works (and rectangle of 240x240)
Note:
fill_solid will basically do a set_pixels(0, 0, 240, 240, array of 240 * 240)
while clear will do a
set_prixes(0, 0, 240, 320, array of 240 * 320) (or set_prixes(0, 0, 320, 240, array of 240 * 320)
and maybe thats the problem that there is problem with set_pixels that its going out of boumds (because wrong orientation)

Yes thats it , when I say that its in Landscape mode clearing works. Note that for me this feels the wrong orientation (my small rectangles are now oriented differently and I feel incorrectly)
My best guess is that clear should be independent from the orientation I want things to be in??? Which in a way makes sense since the orientation is something that is not hardware bound/set (I think), so regardless of how I see my display the clearing should happen the same way?

No that doesnt work either... So I am initting, clearing (which doesnt do much), drawing 2 rectangles, sleeping 2 seconds, clearing again and still 2 rectangles show
Note using fill_solid works (and rectangle of 240x240)
Note: fill_solid will basically do a set_pixels(0, 0, 240, 240, array of 240 * 240) while clear will do a set_pixels(0, 0, 240, 320, array of 240 * 320) (or set_prixes(0, 0, 320, 240, array of 240 * 320)
and maybe thats the problem that there is problem with set_pixels that its going out of boumds (because wrong orientation)
Yes thats it , when I say that its in Landscape mode clearing works. Note that for me this feels the wrong orientation (my small rectangles are now oriented differently and I feel incorrectly)
My best guess is that clear should be independent from the orientation I want things to be in??? Which in a way makes sense since the orientation is something that is not hardware bound/set (I think), so regardless of how I see my display the clearing should happen the same way?

I think you're right. I'll have to go over the orientation MADCTL values vs clear to see where the actual problem lies. I won't be able to do anything until after Sep 10 tho since I'm abroad atm.

@nathansamson if you can please check out #32 to see if this works with those changes now.

authors: almindor nathansamson

repo: almindor/mipidsi

license: MIT


text:
Update ammonite-repl to 1.6.5

Updates com.lihaoyi:ammonite-repl from 1.6.4 to 1.6.5.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a nice day!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [{ groupId = "com.lihaoyi", artifactId = "ammonite-repl" }]

Handled from https://github.com/almond-sh/almond/pull/338.

authors: alexarchambault scala-steward

repo: almond-sh/almond


text:
Convert html.erb templates to haml.haml

for issue #28
The files were converted with
for f in *.html.erb; do html2haml $f ${f/\.html\.erb/.html\.haml}; done

and only the comment in app/views/base_wiki_pages/edit.html.erb had to be manually copied over.

I don't think it's worth adding haml dependency just for bundled views - they are really scaffolds, and are supposed to be reworked by users to match their application.

authors: alno christophermanning

repo: alno/irwi


text:
mycms-tool sign/encrypt with pkcs11 and keyopt

mycms-tool version 0.2.0  sign/encrypt with pkcs11 and keyopt fails with error:
ERROR: 00000005: Invalid keyopts 'rsa_padding_mode=pkcs1' no separator
$ mycms-tool ...... --keyopt="rsa_padding_mode=pkcs1'"
Test scripts with pkcs11 don't run mycms-tool keyopt options.
With version 0.1.1 it's ok

Working for me with the following, I think that you can do better, please find someone that is capable of finding the issue and submit patches or at least refer to the place in code where there is an issue, you should maintain this package, it is not a general use.
diff --git a/test/mycms-tool/sign/test-pkcs11.sh b/test/mycms-tool/sign/test-pkcs11.sh
index c1f66b0..8acbf5e 100755
--- a/test/mycms-tool/sign/test-pkcs11.sh
+++ b/test/mycms-tool/sign/test-pkcs11.sh
@@ -114,6 +114,7 @@ test_sanity() {
                --data-in="${DATA}" \
                --signer-cert="pkcs11:module=${SOFTHSM2_MODULE}:token-label=token1:cert-label=test1" \
                --signer-cert-pass="token=pass=secret" \
+               --keyopt="rsa_padding_mode:pkcs1" \
                || die "sanity.sign.test1"

        echo "List signers test1"

Thanks,
It worked for me too with this change (openssl 3.10).
The issue was with openssl 1.1.1.
I'll try to find the issue.
On Mon, Feb 26, 2024 at 5:47 PM Alon Bar-Lev @.***>
wrote:

Working for me with the following, I think that you can do better, please
find someone that is capable of finding the issue and submit patches or at
least refer to the place in code where there is an issue, you should
maintain this package, it is not a general use.
diff --git a/test/mycms-tool/sign/test-pkcs11.sh b/test/mycms-tool/sign/test-pkcs11.sh
index c1f66b0..8acbf5e 100755--- a/test/mycms-tool/sign/test-pkcs11.sh+++ b/test/mycms-tool/sign/test-pkcs11.sh@@ -114,6 +114,7 @@ test_sanity() {
--data-in="${DATA}"
--signer-cert="pkcs11:module=${SOFTHSM2_MODULE}:token-label=token1:cert-label=test1"
--signer-cert-pass="token=pass=secret" +               --keyopt="rsa_padding_mode:pkcs1"
|| die "sanity.sign.test1"
    echo "List signers test1"

—
Reply to this email directly, view it on GitHub
https://github.com/alonbl/mycms/issues/12#issuecomment-1964463757, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ASSJEC2Y2MYDTWLJZ323RYLYVSVBRAVCNFSM6AAAAABD2NNEM2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRUGQ3DGNZVG4
.
You are receiving this because you authored the thread.Message ID:
@.***>

I found that the problem was with the parameter.
In the past "rsa_padding_mode=pkcs1" worked and now the parameter should be "rsa_padding_mode:pkcs1"

authors: alonbl gilweis

repo: alonbl/mycms


text:
How to use Azurite (emulator) for Azure Blob Storage.

Hi 👋
The previous version Storage.Net is supporting the emu account while in development. It doesn't seem like stowage is supporting it. I looked at the source code and couldn't find any way of connecting, easily, to the emulator. Therefore, I revised the current implementation and do have one issue.
To address the issue, I used the same approach as @iamkoch which is allowing to override the azure endpoint. It worked great; however, it looks like using the http://127.0.0.1:10000/devstoreaccount1 endpoint with a container (emu in this example) lead to an issue with the emulator not looking at the right account. To address that, I had to set my ContainerName to devstoreaccount1/emu.
Here is the code I changed to get things working. If you prefer, I can create a pull request to start a discussion.
Files.cs
public static IFileStorage AzureBlobStorage(this IFilesFactory _, string accountName, string sharedKey, string containerName = null)
{
   return AzureBlobStorage(_, new Uri($"https://{accountName}.blob.core.windows.net"), accountName, sharedKey, containerName);
}

public static IFileStorage AzureBlobStorage(this IFilesFactory _, Uri endpoint, string accountName, string sharedKey, string containerName = null)
{
   return new AzureBlobFileStorage(endpoint, containerName, new SharedKeyAuthHandler(accountName, sharedKey));
}

AzureBlobFileStorage.cs
public AzureBlobFileStorage(Uri endpoint, string containerName, DelegatingHandler authHandler) : base(endpoint, authHandler)
{
   if(endpoint is null)
      throw new ArgumentNullException(nameof(endpoint));
   if(string.IsNullOrEmpty(containerName))
      throw new ArgumentException($"'{nameof(containerName)}' cannot be null or empty", nameof(containerName));
   if(authHandler is null)
      throw new ArgumentNullException(nameof(authHandler));

   _containerName = containerName;
}

I guess I could keep it that way, but I find it a bit confusing to put the storage account in the container name when in development. Haven't tested in production with Azure Storage yet.
Thanks,

Sounds good to me, please raise a PR. I haven't used Azurite yet other than from azure functions emulator indirectly so can't confirm how it works - you have my full trust and support, should be a great addition.

@YannickRondeau are you doing a PR?
If not I can look at doing one, looking to provide more storage options for the thing I'm developing and this NuGet is perfect, but I use Azurite for local development.

@IeuanWalker I haven't work on anything yet. For our project, we've decided to go directly with the Azure Storage SDK which was easier in our case.

authors: IeuanWalker YannickRondeau aloneguid

repo: aloneguid/stowage


text:
AssertionError when using PipeshardParallel

Problem
Hello, I am trying to use PipeshardParallel method with ResNet18 model in a cluster using two nodes. However, when the model.apply function is invoked, Alpa triggers an assertion error inside the function slice_closed_jaxpr_by_full_pipeline_marks. Here is the full log with stack trace. I am not sure if I am using the API wrong or if this is indeed a bug, but I am assuming a bug since it is an assertion error
WARNING:alpa.pipeline_parallel.apply_grad:Missing microbatch_bound between compute and apply. Assume there is no apply gradient step. Hint: replace jax.grad by alpa.grad.
Traceback (most recent call last):
  File "main_simple.py", line 40, in <module>
    result, mutated_vars = apply(params, x)
  File "/usr/local/lib/python3.8/dist-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/alpa/api.py", line 121, in __call__
    self._decode_args_and_get_executable(*args))
  File "/usr/local/lib/python3.8/dist-packages/alpa/api.py", line 191, in _decode_args_and_get_executable
    executable = _compile_parallel_executable(f, in_tree, out_tree_hashable,
  File "/usr/local/lib/python3.8/dist-packages/jax/linear_util.py", line 309, in memoized_fun
    ans = call(fun, *args)
  File "/usr/local/lib/python3.8/dist-packages/alpa/api.py", line 218, in _compile_parallel_executable
    return method.compile_executable(fun, in_tree, out_tree_thunk,
  File "/usr/local/lib/python3.8/dist-packages/alpa/parallel_method.py", line 233, in compile_executable
    return compile_pipeshard_executable(
  File "/usr/local/lib/python3.8/dist-packages/alpa/pipeline_parallel/compile_executable.py", line 93, in compile_pipeshard_executable
    pipeshard_config = compile_pipeshard_executable_internal(
  File "/usr/local/lib/python3.8/dist-packages/alpa/pipeline_parallel/compile_executable.py", line 158, in compile_pipeshard_executable_internal
    jax_pipeline_layers = slice_closed_jaxpr_by_full_pipeline_marks(
  File "/usr/local/lib/python3.8/dist-packages/alpa/pipeline_parallel/computation.py", line 395, in slice_closed_jaxpr_by_full_pipeline_marks
    assert current_computation is not None
jax._src.traceback_util.UnfilteredStackTrace: AssertionError

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main_simple.py", line 40, in <module>
    result, mutated_vars = apply(params, x)
  File "/usr/local/lib/python3.8/dist-packages/alpa/pipeline_parallel/compile_executable.py", line 93, in compile_pipeshard_executable
    pipeshard_config = compile_pipeshard_executable_internal(
  File "/usr/local/lib/python3.8/dist-packages/alpa/pipeline_parallel/compile_executable.py", line 158, in compile_pipeshard_executable_internal
    jax_pipeline_layers = slice_closed_jaxpr_by_full_pipeline_marks(
  File "/usr/local/lib/python3.8/dist-packages/alpa/pipeline_parallel/computation.py", line 395, in slice_closed_jaxpr_by_full_pipeline_marks
    assert current_computation is not None
AssertionError

Source Code
Here is the code I used to reproduce this bug. The ResNet18 model  used is identical to the one found in the Alpa examples.
import jax
import alpa
import models
import numpy as np

import jax.numpy as jnp

# Alpa Initialization
# ==============================================================================

alpa.init(cluster="ray")

# Data Initialization
# ==============================================================================

dim = 4096
batch_size = 32
num_classes = 128

k0 = jax.random.PRNGKey(0)
k0, k1 = jax.random.split(k0)
k0, k2 = jax.random.split(k0)
k0, k3 = jax.random.split(k0)

x = jax.random.normal(k1, (batch_size, dim, dim, 3))

# Model Initialization
# ==============================================================================

model = models.ResNet18(num_classes=num_classes, dtype=jnp.float32)
params = model.init(k3, x)

@alpa.parallelize(method=alpa.PipeshardParallel())
def apply(*args):
  return model.apply(*args, mutable=["batch_stats"])

# Feed forward
# ==============================================================================

result, mutated_vars = apply(params, x)

print(result.shape)
print(result)

Reproducing
I am running this code on Slurm, and my sbatch script is given below. I have reproduced this bug using both Python virtualenvs and a Singularity (Apptainer) container image. The command to run is: sbatch -A <account> -p <partition> run.sh.
#!/usr/bin/env bash

#SBATCH --job-name=alpa
#SBATCH --nodes=2
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=1GB
#SBATCH --time=00:30:00

module purge
module load python/3.9.6
module load cuda/11.5
module list

source venv/bin/activate

which python python3 pip pip3
pip list

# RAY INIT
# ------------------------------------------------------------------------------

# Get names of nodes assigned
nodes=$(scontrol show hostnames "$SLURM_JOB_NODELIST")
nodes_array=($nodes)

# By default, set the first node to be head_node on which we run HEAD of Ray
head_node=${nodes_array[0]}
head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address)

# Setup port and variables needed
gpus_per_node=4
port=6789
ip_head=$head_node_ip:$port
export ip_head
# Start HEAD in background of head node
srun --nodes=1 --ntasks=1 -w "$head_node" \
        ray start --head --node-ip-address="$head_node_ip" --port=$port \
        --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus $gpus_per_node --block &

# Start worker nodes
# Number of nodes other than the head node
worker_num=$((SLURM_JOB_NUM_NODES - 1))
# Iterate on each node other than head node, start ray worker and connect to HEAD
for ((i = 1; i <= worker_num; i++)); do
    node_i=${nodes_array[$i]}
    echo "==> Starting WORKER $i at $node_i"
    srun --nodes=1 --ntasks=1 -w "$node_i" \
        ray start --address "$ip_head" --num-cpus "${SLURM_CPUS_PER_TASK}" \
        --num-gpus $gpus_per_node --block &
    sleep 5
done

# USER CODE
# ------------------------------------------------------------------------------

python3 main.py

# RAY STOP
# ------------------------------------------------------------------------------

ray stop

System Information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04, docker):
Python version: 3.9.6
CUDA version: 11.5
NCCL version: 2.7.8
cupy version: 11.5.0
GPU model and memory: Two cluster nodes with 4x NVIDIA-V100-PCIE-32GB each.
Alpa version: 0.2.2
TensorFlow version: None
JAX version: 0.3.22

I am willing to provide further information if needed. Thanks in advance!

Similar issue here. I'm using Slurm, too. Even a single 1-layer layer MLP network seems to re-create this. ShardParallel, Zero2, and Zero3 are able to start training on the same code.

If it is inference only, please see examples here https://github.com/alpa-projects/alpa/blob/66a77da2fa4dfc15dcc0045976dcb64537c88512/tests/pipeline_parallel/test_inference_only.py#L88.
You should use a slightly different option

Thanks @merrymercy, the original problem indeed went away. Now, however, I get a timeout error. Do you have any idea what might be causing this? It seems to be more related to Ray. I can move this discussion to a new issue if you prefer.
(scheduler +9m24s) Tip: use `ray status` to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0.
(scheduler +9m24s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +9m59s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +10m34s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +11m9s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +11m44s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +12m19s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +12m54s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +13m29s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +14m5s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +14m40s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +15m15s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +15m50s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +16m25s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +17m0s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
(scheduler +17m35s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.
Traceback (most recent call last):
  File "/home/gustavo.leite/alpa/examples/conda/main.py", line 81, in <module>
    result, mutated_vars = apply(x)
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/jax/_src/traceback_util.py", line 162, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/api.py", line 121, in __call__
    self._decode_args_and_get_executable(*args))
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/api.py", line 191, in _decode_args_and_get_executable
    executable = _compile_parallel_executable(f, in_tree, out_tree_hashable,
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/jax/linear_util.py", line 309, in memoized_fun
    ans = call(fun, *args)
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/api.py", line 218, in _compile_parallel_executable
    return method.compile_executable(fun, in_tree, out_tree_thunk,
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/parallel_method.py", line 233, in compile_executable
    return compile_pipeshard_executable(
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/compile_executable.py", line 93, in compile_pipeshard_executable
    pipeshard_config = compile_pipeshard_executable_internal(
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/compile_executable.py", line 237, in compile_pipeshard_executable_internal
    xla_stages, total_flops = shard_each_stage(
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/compile_executable.py", line 361, in shard_each_stage
    stage_plan) = compile_workers.get_next_unordered()
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/stage_profiling.py", line 85, in get_next_unordered
    return self.pool.get_next_unordered(
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/util/actor_pool.py", line 254, in get_next_unordered
    raise TimeoutError(timeout_msg)
jax._src.traceback_util.UnfilteredStackTrace: TimeoutError: Timed out waiting for result

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gustavo.leite/alpa/examples/conda/main.py", line 81, in <module>
    result, mutated_vars = apply(x)
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/compile_executable.py", line 93, in compile_pipeshard_executable
    pipeshard_config = compile_pipeshard_executable_internal(
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/compile_executable.py", line 237, in compile_pipeshard_executable_internal
    xla_stages, total_flops = shard_each_stage(
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/compile_executable.py", line 361, in shard_each_stage
    stage_plan) = compile_workers.get_next_unordered()
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/stage_profiling.py", line 85, in get_next_unordered
    return self.pool.get_next_unordered(
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/util/actor_pool.py", line 254, in get_next_unordered
    raise TimeoutError(timeout_msg)
TimeoutError: Timed out waiting for result
2023-02-13 11:49:46,879	ERROR services.py:1403 -- Failed to start the dashboard: 'NoneType' object is not iterable
2023-02-13 11:49:46,879	ERROR services.py:1404 -- 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/services.py", line 1288, in start_api_server
    minimal = not ray._private.utils.check_dashboard_dependencies_installed()
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/utils.py", line 1291, in check_dashboard_dependencies_installed
    import ray.dashboard.optional_deps  # noqa: F401
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/dashboard/optional_deps.py", line 8, in <module>
    import opencensus  # noqa: F401
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 982, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1414, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1380, in _get_spec
TypeError: 'NoneType' object is not iterable
Exception ignored in: <function BaseWorkerPoolWrapper.__del__ at 0x7f62c954d670>
Traceback (most recent call last):
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/stage_profiling.py", line 100, in __del__
    self.shutdown()
  File "/home/gustavo.leite/alpa/examples/conda/alpa/alpa/pipeline_parallel/stage_profiling.py", line 92, in shutdown
    ray.kill(w)
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 100, in wrapper
    if client_mode_should_convert(auto_init=auto_init):
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 124, in client_mode_should_convert
    ray.init()
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/worker.py", line 1429, in init
    _global_node = ray._private.node.Node(
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/node.py", line 309, in __init__
    self.start_ray_processes()
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/node.py", line 1112, in start_ray_processes
    resource_spec = self.get_resource_spec()
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/node.py", line 471, in get_resource_spec
    self._resource_spec = ResourceSpec(
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/resource_spec.py", line 175, in resolve
    num_gpus = _autodetect_num_gpus()
  File "/home/gustavo.leite/alpa/examples/conda/venv/lib/python3.9/site-packages/ray/_private/resource_spec.py", line 277, in _autodetect_num_gpus
    if importlib.util.find_spec("GPUtil"):
  File "/opt/share/python/3.9.6/lib/python3.9/importlib/util.py", line 103, in find_spec
    return _find_spec(fullname, parent_path)
  File "<frozen importlib._bootstrap>", line 925, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1414, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1380, in _get_spec
TypeError: 'NoneType' object is not iterable

When the program is stuck waiting, what does running "ray status" show? Does it see your GPUs? It looks like the profiling is trying to create more processes via Ray than the amount of available logical CPUs on your machine.

@samblouir I ran watch -n5 ray status on one of the compute nodes while the example was running and a sample of the output is given below. For this job I allocated 8 nodes with 4 GPUs (V100) each. From the output we can see all of the 32 GPUs---but the usage of all resources remained at zero for the entirety of the execution.

It looks like the profiling is trying to create more processes than the amount of available logical CPUs on your machine via Ray, resulting in hanging.

Is there a way to run with profiling disable just to see if it works?
I also ran some other (very simple) tests without Alpa where I launch a bunch of tasks or actors and it seems to be running fine.
======== Autoscaler status: 2023-03-13 11:34:48.020321 ========
Node status
---------------------------------------------------------------
Healthy:
 1 node_4ea936f8a79487627b429676dfeeee82d8aa951396450e81fccbde70
 1 node_2bb257a71e82b39a12488c7414f21cbb08ebf7073398b2cd84eaa15d
 1 node_64fcc89dee05bf7e5bee0fe51ffc1bd919a108eebdd4b2dbb8e7b9af
 1 node_88755f5a51ebbdb856854b1f4ef82806a561eb8a008d038cbbe72a33
 1 node_03789627b3f61992abf6a807bbd186490c03838e71293f9018f71e0e
 1 node_157f908f31caa76661eeb887e7c0ac9f6cebf09e8f1616a3ab0a2958
 1 node_836d00c494cd05dbbff7136ee2b893bbd49fdcfe59cb67fb6642517d
 1 node_5deb78ca652a15ea1afc08b720ecbe01bacbdf5929b40772a3bc1874
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/8.0 CPU (0.0 used of 8.0 reserved in placement groups)
 0.0/32.0 GPU (0.0 used of 32.0 reserved in placement groups)
 0.0/8.0 accelerator_type:V100
 0.00/2078.250 GiB memory
 0.00/894.670 GiB object_store_memory

Demands:
 {'CPU': 1.0}: 2+ pending tasks/actors

I finally found the problem. The Slurm script I posted above was taken from the documentation page Using Alpa on Slurm and it defines the sbatch flag --cpus-per-task=1. This made only one core available per Ray process, causing Alpa to hang during the distributed compilation of the model due to the lack of CPU resources available for scheduling.
The solution consisted in not specifying the amount of CPUs per task in the sbatch script and let the Ray process use all of the CPUs and GPUs available (which is the default behavior).
To my understanding, the example in the documentation is incorrect. Should I send a patch to fix it?
--- a.sh	2023-03-15 10:53:58.315148470 -0300
+++ b.sh	2023-03-15 10:54:24.661178461 -0300
@@ -2,14 +2,13 @@
 #SBATCH --partition=GPU
 #SBATCH --nodes=2
 #SBATCH --tasks-per-node=1
-#SBATCH --cpus-per-task=1
 #SBATCH --mem-per-cpu=1GB
 #SBATCH --gpus-per-node=v100-16:8
 #SBATCH --time=00:10:00

 # ...[snip]...

 srun --nodes=1 --ntasks=1 -w "$head_node" \
         ray start --head --node-ip-address="$head_node_ip" --port=$port \
-        --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus $gpus_per_node --block &
+        --block &

 # ...[snip]...

 worker_num=$((SLURM_JOB_NUM_NODES - 1))

@@ -17,7 +16,6 @@
     node_i=${nodes_array[$i]}
     echo "Starting WORKER $i at $node_i"
     srun --nodes=1 --ntasks=1 -w "$node_i" \
-        ray start --address "$ip_head" --num-cpus "${SLURM_CPUS_PER_TASK}" \
-        --num-gpus $gpus_per_node --block &
+        ray start --address "$ip_head" --block &
     sleep 5
 done

authors: leiteg merrymercy samblouir

repo: alpa-projects/alpa


text:
Added support for matlab/octave

Minor changes to include matlab and octave as per #17
Included the update information in the changelog and readme.

Thank you :blush:

authors: alpertuna asclepius

repo: alpertuna/vim-header


text:
Add ability to prevent a flash of unstyled content

Add js-enabled CSS for the select which allows users to stop fouc if they add js-enabled to the html element.  Also update form example to reflect this.

What happens if js is enabled but fails to load successfully?

@edwardhorsford With this, if JavaScript is enabled but it the autocomplete fails to render (connection problems, JS errors), the <select> will look like a prefilled input field:

Which when clicked, is actually a <select> element:

This isn't ideal, but not broken either.
Having the caret from showAllValues that Joe built could potentially help with this; is that something worth trying out?

We'd have to create an error boundary around this component that can apply a class to override this.
e.g.

Add js-enabled in the
Try to init component but wrap it in try {} catch {}
If it fails in the try {} catch {} add 'is-init-failed' class to element

Mind dump so might be a more sane way, on GOV.UK we just do a general check at the end of the page.

An idea I had previously (which would help in most, but not all cases) is to hide the input on page-load, but display it using css transition after 2 seconds. If js fails to run, the input will become visible, but in most cases (assuming js loads in under 2 seconds), we'll just see the new autocomplete. The advantage of this method is it feels more robust - but at the downside that slower browsers or those not supporting css transitions will still get a FOUC.

I'm closing this PR as it has been open for a considerable amount of time and we're trying to cut down the number of orphaned PRs in alphagov.

authors: carolinegreen edwardhorsford nickcolley tlwr tvararu

repo: alphagov/accessible-autocomplete


text:
Insert file attachments via modal

For https://trello.com/c/gPfQ2lBT/819-interface-with-a-documents-attachments-via-a-modal
This enables users with Javascript enabled to auto-insert file attachment markdown via the modal.
I've tested inserting the markdown before text, after text and within text for both attachment types. All renders correctly bar the attachment link when inserted before text, however this is being dealt with here https://github.com/alphagov/govspeak/pull/154

All looks good to me! My only comment would be that if we have a no js spec for inserting an inline attachment then we should probably have one for images too but thats not concerning this pr! :) ⭐

@1pretz1 well spotted with the missing spec. We planned to add it, but forgot - @emmabeynon, do you think you could do a quick PR for this?

authors: 1pretz1 benthorner emmabeynon

repo: alphagov/content-publisher


text:
Refactor taxon controller & views

The TaxonsController has become too big and has contains a lot of logic. This refactors the controller and views.
The guiding principle is that any view should only be passed one object from the controller. See Sandi's rule No. 4:

"Only instantiate one object in the controller"

https://robots.thoughtbot.com/sandi-metz-rules-for-developers#only-instantiate-one-object-in-the-controller
To do this we create 3 view objects, IndexPage, ShowPage, and EditPage. These objects know the data to render the page. It simplifies the controller a lot. We could probably also simplify the views a bit, but I haven't gotten to that in this PR.
Commit also inlines some HTML in the deletion warning template to make it easier to change.
cc @carvil

👍

authors: carvil tijmenb

repo: alphagov/content-tagger


text:
Redesign the management of root taxons

https://trello.com/c/AHfdTD94/188-create-visualisations-to-show-how-much-is-tagged-to-where
Move the previous index page for root taxons to be an explicit edit
page, and change the index page to a show page with a table. This show
page is now used to link out to the pages for the individual root
taxons.
This will allow users to navigate to the show pages for root taxons.

@tijmenb I've now restructured the routes, controller and views around a plural resource (rather than a singular one). Let me know what you think?


text:
Add script to create missing log alerts in Hosted Graphite

https://trello.com/c/tZghClQt/354-add-a-hosted-graphite-alert-for-missing-logs
We want to know if our logging infrastructure breaks. We can do
this by monitoring AWS incomingLogEvents metrics. If no logs
are being added to AWS Cloudwatch log groups for a period of time
then we can trigger an alert.
Note, we looked into setting these up as AWS Cloudwatch alerts
but ended up not going down that path. This was because

it would be another type of infrastructure to use
slack alerts would be extra work to set up than emails which
differs from our current alerts
terraform doesn't let you set up cloudwatch alerts that go to
an email address (email is an unsupported protocol as described
https://www.terraform.io/docs/providers/aws/r/sns_topic_subscription.html)
unless the email alert is set up manually without using terraform

Relies on metric shipping PR - https://github.com/alphagov/digitalmarketplace-cloudwatch-to-graphite/pull/7
Note, this is currently live on preview as part of my testing but not for staging/production.
I have also taken this opportunity to put into code the current 4 alerts that we have on Hosted Graphite already.

I've looked to set up the rest of our alerts using this script. During that I've uncovered a bug with the hosted graphite API. I'm in contact with their support team and they say they are going to fix it (no idea on time frame but sounded like in the short term).
I've got a commit waiting on my local to do it but until they fix their API it will have a slightly broken interface for our alerts so I'll hold off pushing it up for the moment...

Funny you should say that... think it's just managed to do that for itself...

So @risicle I looked into the list of app names.
We do have some similar but different lists that already exist:

vars/common.yml contains all the apps but not as a list and there are extra keys that we would not want such as db-backup and maintenance_mode
some of our terraform modules do define them, for example terraform/modules/application-logs/variables.tf.  However this one doesn't include the router app for example and also isn't some nice python file to import.

So it looks like there isn't a list we could just use as is. This is the first time we are using this exact list I think. As this is the first time I'm probably not going to extract out the list and put it somewhere common although I can do if you feel strongly (if so, feel free to make a recommendation of where it could go if you have one).

No I don't feel strongly about it - it might be an interesting thing to do, having a more central place where the different categories of application are listed & defined...

authors: idavidmcdonald risicle

repo: alphagov/digitalmarketplace-aws


text:
Tweaks for opportunities data csv

Show all opportunities
Add default to requirementsLength

(Merged with Ben looking over my shoulder!)

authors: benvand katstevens

repo: alphagov/digitalmarketplace-scripts


text:
Remove custom version of gulp-jasmine-phantom

We made a custom version to deal with the fact
that the version of Node that Travis uses in the VM
image for Python projects was below that required
for the NPM module (see related stackoverflow question).
This is no longer the case (it's currently running
at 4.2 (using nvm) so the custom version is no
longer needed.

:+1:

authors: TheDoubleK tombye


text:
Add supplier update errors

Remove redundant _get_fields method
This is exactly the same as get_question_ids but without using the same language as the rest of the class.
Allow question IDs to be filtered by question type
This is useful for doing actions on a questions of a particular type.
Move upload_documents into dmutils
This was previously implemented in the supplier and admin frontends. This also introduces a couple of tests for it.

:+1:

Ooops, yea, changed title and added version bump.

authors: TheDoubleK robyoung

repo: alphagov/digitalmarketplace-utils


text:
Run RuboCop as part of bundle exec rake defaults

This is a prerequisite for moving to GitHub Actions eventually: we
want to be able to run one single bundle exec rake command and have it
to tests, linting, etc.
Also remove old excludes that are dealt with now by upstream gem
configs.

https://trello.com/c/5Gob6WUL/147-for-everyone-ensure-that-rubocop-runs-as-part-of-bundle-exec-rake-as-one-of-the-default-tasks

I think this change has broken the deploys:
16:02:12  ** [out :: ip-10-12-4-217.eu-west-1.compute.internal] I, [2020-05-20T15:02:11.990725 #19605]  INFO -- sentry: ** [Raven] Raven 3.0.0 ready to catch errors
16:02:12 *** [err :: ip-10-12-4-217.eu-west-1.compute.internal] rake aborted!
16:02:12 *** [err :: ip-10-12-4-217.eu-west-1.compute.internal] Don't know how to build task 'default' (See the list of available tasks with `rake --tasks`)
16:02:12 *** [err :: ip-10-12-4-217.eu-west-1.compute.internal] /data/vhost/email-alert-api/releases/20200520150209/Rakefile:10:in `<top (required)>'
16:02:12 *** [err :: ip-10-12-4-217.eu-west-1.compute.internal] /data/vhost/email-alert-api/shared/bundle/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
16:02:12 *** [err :: ip-10-12-4-217.eu-west-1.compute.internal] /usr/lib/rbenv/versions/2.6.6/bin/bundle:23:in `load'
16:02:12 *** [err :: ip-10-12-4-217.eu-west-1.compute.internal] /usr/lib/rbenv/versions/2.6.6/bin/bundle:23:in `<main>'
16:02:12 *** [err :: ip-10-12-4-217.eu-west-1.compute.internal] (See full trace by running task with --trace)

Example: https://deploy.blue.staging.govuk.digital/job/Deploy_App/5355/console

Indeed. We spotted a bunch of these too late. We need to add unless Rails.env.production? in the Rakefile.

I've opened https://github.com/alphagov/email-alert-api/pull/1239, just going to check the deploy works in integration.

authors: benthorner issyl0 thomasleese

repo: alphagov/email-alert-api


text:
Add some more calendars routes

We have these routes in router-api, which look a bit weird:
[#<Route _id: 5277d004fa5ee55bbe000018, created_at: nil, updated_at: nil, incoming_path: "/bank-holidays.json", route_type: "exact", handler: "backend", disabled: false, backend_id: "calendars", redirect_to: nil, redirect_type: nil, segments_mode: nil>,
 #<Route _id: 5277d004fa5ee55bbe000019, created_at: nil, updated_at: nil, incoming_path: "/gwyliau-banc", route_type: "prefix", handler: "backend", disabled: false, backend_id: "calendars", redirect_to: nil, redirect_type: nil, segments_mode: nil>,
 #<Route _id: 5277d004fa5ee55bbe00001a, created_at: nil, updated_at: nil, incoming_path: "/gwyliau-banc.json", route_type: "exact", handler: "backend", disabled: false, backend_id: "calendars", redirect_to: nil, redirect_type: nil, segments_mode: nil>,
 #<Route _id: 5277d004fa5ee55bbe00001c, created_at: nil, updated_at: nil, incoming_path: "/when-do-the-clocks-change.json", route_type: "exact", handler: "backend", disabled: false, backend_id: "calendars", redirect_to: nil, redirect_type: nil, segments_mode: nil>]

No created_at or updated_at?  Do these routes predate our current way of doing things?  They're definitely not in the content items.

Trello card

Hmm, I've got myself a bit confused by this.  Publishing with the cy locale got me this:
{"request_uri":"http://publishing-api.dev.gov.uk/v2/content/58f79dbd-e57f-4ab2-ae96-96df5767d1b2/publish","start_time":1588587884.789642,"status":409,"end_time":1588587884.9564757,"body":"{\"error\":{\"code\":409,\"message\":\"Cannot publish an already published edition\",\"fields\":{}}}"}

So I came up with a new content id, and got this:
{"error":{"code":404,"message":"Item with content_id 1cf6dcbc-4bb9-4f64-b964-d37b8d05fc7b and locale en does not exist","fields":{}}}

I'll keep digging

Alright, managed to get this working on integration

repo: alphagov/frontend


text:
Update tour page

Lots of this hasn't been modified in ages. I've updated the bits which sound the most old.

LGTM :+1:

authors: alexmuller fofr


text:
[Do not merge] Add Northern Ireland authorities (LGD) to the tier lookup

For some reason LGD (Northern Ireland) authorities were added to the list of
area types but not the tier lookup.
As part of the fix, we'll try and reduce duplication of lists of types.  It's also hard
to check which ones are in the list if they're randomly ordered.
The ordering of the list of authority_types is important as licences need to
be able to select the most specific match in the case when two local
authorities are returned for a postcode.  This means that districts must come
before counties.  It doesn't really matter where unitary authorities come in the list.
This test describes the case

I think there are still more things we'd need to fix apart from this to be able to support Northern Ireland fully for local transactions. What's the effect for users of making this change in isolation?

Seems fair, I'll withdraw this

authors: jennyd sihugh


text:
Adds govuk-chat repo to available apps

Description
Adds govuk-chat repo to the available govuk-docker apps

At the moment it only links the app and postgresSQL 13 for a basic dev environment

Trello card
https://trello.com/c/h83meZak/1183-add-govuk-chat-to-govuk-docker

Changes look good, do you want to squash the commits?

authors: AlexAvlonitis kevindew

repo: alphagov/govuk-docker


text:
Swap in draft-content-store-proxy for draft-content-store via service app selector

We have the draft-content-store-proxy up and running on integration, now we need to manipulate the cluster DNS-resolution into making http://draft-content-store/ resolve to the proxy (as per this doc)
This PR:

allows an app to override the service -> selector -> app value
makes the draft-content-store-proxy provide a selector of draft-content-store
renames the draft-content-store app to draft-content-store-mongo-main
alters the PRIMARY_UPSTREAM env var for the proxy to http://draft-content-store-mongo-main/

Note - I'm hoping that this will achieve the desired result, but beyond linting correctly I can't really test this locally (unless I install kubernetes and replicate the cluster setup on my laptop, etc etc  which would be a big job). It's quite possible I'm going about this the wrong way, and I'm totally open to alternative ways of achieving the result.
Trello card, part of rolling out the content-store mongo->postgresql migration  on integration

fwiw there are some other ways you could do this but I think this is as good as any other 👍

authors: aldavidson sengi

repo: alphagov/govuk-helm-charts


text:
Extract phone numbers

https://trello.com/c/mSjX7Dq0/2353-think-about-phone-numbers
Users seem to often search for phone numbers, but they might not be getting a complete set of results.  This could lead to outdated phone numbers remaining on GOV.UK.
We ought to consider mitigating this problem.
Evidence of user need: https://docs.google.com/document/d/1auqzEXTiwAgNPG6PfxDDG7rSt7ImShKxEOo_zBaUsAE/edit#heading=h.aowahbaamoq
One way would be to use Google's libphonenumber to detect phone numbers in GOV.UK content, and format them in a standard form.  Similarly, format searched-for phone numbers in a standard form, and then do a lookup.
The script below uses a python implementation of libphonenumber.  I wrote my own python bindings to Google's C++ implementation, and it had exactly the same performance :smiling_face_with_tear:
# Extract phone numbers from a text column of a CSV file.

import argparse
import sys
import csv
import phonenumbers
import phonenumbers.geocoder

# Allow the largest field size possible.
# https://stackoverflow.com/a/15063941
maxInt = sys.maxsize
while True:
    # decrease the maxInt value by factor 10
    # as long as the OverflowError occurs.
    try:
        csv.field_size_limit(maxInt)
        break
    except OverflowError:
        maxInt = int(maxInt / 10)
csv.field_size_limit(maxInt)

writer = csv.DictWriter(
    sys.stdout,
    fieldnames=["url", "phonenumber", "start", "raw_string", "type", "country", "location"],
)

region = "GB"
locale = "en"
number_format = phonenumbers.PhoneNumberFormat.E164

with open("../body_content.csv", newline="") as csvfile:
    reader = csv.DictReader(csvfile, delimiter=",", quotechar='"')
    for line in reader:
        row_dict = {"url": line["url"]}
        text = line["text_without_blank_lines"]
        for match in phonenumbers.PhoneNumberMatcher(text, region):
            if phonenumbers.is_valid_number(match.number):
                row_dict["phonenumber"] = phonenumbers.format_number(
                    match.number, number_format
                )
                row_dict["start"] = match.start
                row_dict["raw_string"] = match.raw_string
                row_dict["type"] = phonenumbers.PhoneNumberType.to_string(
                    phonenumbers.number_type(match.number)
                )
                row_dict["country"] = phonenumbers.geocoder.country_name_for_number(
                    match.number, locale
                )
                row_dict["location"] = phonenumbers.geocoder.description_for_number(
                    match.number, region
                )
                writer.writerow(row_dict)

url
phonenumber
start
raw_string
type
country
location

https://www.gov.uk/ad-dalu-gordaliadau-budd-dal-plant
+443002001900
3018
0300 200 1900
UAN

https://www.gov.uk/adrodd-pryder-am-atwrnai-dirprwy-warcheidwad
+441159342777
1537
0115 934 2777
FIXED_LINE
United Kingdom
Nottingham

https://www.gov.uk/adrodd-pryder-am-atwrnai-dirprwy-warcheidwad
+441159342778
1564
0115 934 2778
FIXED_LINE
United Kingdom
Nottingham

https://www.gov.uk/anti-money-laundering-registration
+441415822000
1557
0141 582 2000
FIXED_LINE
United Kingdom
Glasgow

https://www.gov.uk/anti-money-laundering-registration
+442073973008
1688
020 7397 3008
FIXED_LINE
United Kingdom
London

https://www.gov.uk/anti-money-laundering-registration
+441914930272
1897
0191 4930272
FIXED_LINE
United Kingdom
Tyneside

https://www.gov.uk/anti-money-laundering-registration
+442073400551
2093
020 7340 0551
FIXED_LINE
United Kingdom
London

https://www.gov.uk/anti-money-laundering-registration
+441234845777
2381
01234 845777
FIXED_LINE
United Kingdom
Bedford

https://www.gov.uk/anti-money-laundering-registration
+442073400550
2990
020 7340 0550
FIXED_LINE
United Kingdom
London

If this can be done in javascript instead of Ruby or Python, then we don't even need to create cloud functions.  We can define them directly in BigSQL.  Google's own javascript library isn't useable for this, because it's difficult to install, and anyway it doesn't support find() to find numbers in general text. But Google themselves link to two forks that we could use instead.

https://github.com/catamphetamine/libphonenumber-js supports find() and is bundled in a way that we could host in a bucket
https://github.com/ruimarinho/google-libphonenumber is bundled for node, so we could use it in GovSearch to inform the user about how the number that they input will be interpreted.

Done by #571

repo: alphagov/govuk-knowledge-graph-gcp


text:
Shakedown testing of v13

What
Do some last-minute testing of v13 of the Prototype Kit before release.
Why
Make sure there are no major issues in the v13.0.0 release.
Who needs to work on this
Whole team
Done when

[ ] Users can do all the things that we expect them to be able to do without issues

[x] Create a prototype
[x] Make a juggling licence app
[ ] Publish on Heroku

[ ] Users doing things we don't expect them to do won't have massive issues
[ ] We are happy to release v13.0.0

Using a fresh Linux machine, I have succsessfully built a juggling license prototype using v13 and the new instructions.

I have now succsessfully deployed to heroku.

Some small issues:

When deleting the config.json file as I think users could possibly do by accident, the prototype does not report an error in the log but continues on with the service name changed to "GOV.UK Prototype Kit".  I agree with @joelanman that the users may not necessarily notice this.
When the config.json is malformed, the application displays the fact the config.json is malformed several times but carries on with the default name changed as above to "GOV.UK Prototype Kit".  I think can cause the same confusion as above especially if the user is not monitoring the logs when their app "appears" to be running ok.

A few suggestions for pages not found:

I notice that if a user has deleted their index.html page and deploy to Heroku, the default page telling the user how to create their home page is displayed.  Perhaps when deployed to Heroku, this should instead be "Page not found" like it is for any other page not defined.
Now that we have the manage-prototype pages available when the prototype is in development it would be better to replace the code in our default homepage with a link to the /manage-prototype/templates page with an example of an index page for them to create it from.
Perhaps while in development, we should replace the "Page not found" message in the browser with a formatted page specifiying the page is not found and if they expect it to exist they can go to the /manage-prototype/templates page to create one.

authors: BenSurgisonGDS lfdebrux

repo: alphagov/govuk-prototype-kit


text:
Use of MD5, use of HTTP, and empty passwords

Greetings,
I am a security researcher, who is looking for security smells in Puppet scripts. I noticed instances of MD5 uses in one of the Puppet scripts. MD5 is vulnerable to attacks, and should be avoided. The Common Weakness Enumeration organization recommends against usage of weak cryptographic algorithms such as MD5. Reff: https://cwe.mitre.org/data/definitions/327.html.
I suggest the use of SHA512 , which is more secure. Any feedback is appreciated.
Source: https://github.com/alphagov/govuk-puppet/blob/master/modules/govuk/manifests/node/s_transition_postgresql_standby.pp

Along with use of MD5, I also noticed instances of empty passwords. Empty passwords increase the guessability of passwords. The Common Weakness Organization (CWE) identifies use of empty passwords as a security weakness (https://cwe.mitre.org/data/definitions/258.html).
I suggest that to follow the strong password guidelines, and manage passwords with hiera.
Source: https://github.com/alphagov/govuk-puppet/blob/master/modules/govuk_crawler/manifests/init.pp

I also found instances where the HTTP protocol is used instead of HTTPS (HTTP with TLS). According to the Common Weakness Enumeration organization this is a security weakness (https://cwe.mitre.org/data/definitions/319.html).  I was wondering why HTTP is used? Is it because of lack of tool support?
I am trying to find out if developers are forced to adopt bad practices due to lack of tool support when it comes to the HTTPS protocol. Maybe it is due to dependency on a resource that uses HTTP?
Any feedback is appreciated.
Source: https://github.com/alphagov/govuk-puppet/blob/master/modules/nodejs/manifests/repo.pp

Thank you for raising these and apologies for the lack of activity on the issue.
We have a separate, private, system that replaces the empty passwords with securely generated values that are loaded via Hiera. Due to the nature of the public/private split it can appear we have empty passwords in places where we actually don't. In regards to the postgres issue we've moved to RDS now and changed the way this authentication works.
Thank you again for flagging these

authors: akondasif deanwilson

repo: alphagov/govuk-puppet


text:
Fix attachable for campaigns

This updates attachable to work with the new GDS API Adapters response, which is now a hash.

LGTM

authors: emmabeynon whoojemaflip


text:
Remove alias for footer background

This Sass variable $footer-background should remain with the GOV.UK template’s colour
variables.
cc. @dsingleton.

:+1:
We agreed we're going to start splitting non-template specific helpers out into the toolkit, but for anything specific to template, eg this grey is specifically for use with templates' footer, they'll stay in template.

authors: dsingleton gemmaleigh


text:
Put the current version of the kit in the footer

This would help with debugging as you would be able to tell which version of the kit is running at a glance.

good idea

It's currently inserted as an html comment on each page. Are you suggesting it be visibly printed too?

@edwardhorsford yes because that wasn't apparent to me. I am suggesting it's visible in the page footer.

authors: alextea edwardhorsford joelanman


text:
PP-10639: Map event types to AgreementStates

We are going to remove the status field from all AgreementEvent implementations, so instead, map SalientEventTypes to AgreementStates.

I think AGREEMENT_SETUP (noun) is misnamed and should be AGREEMENT_SET_UP (verb phrase), but it needs to be kept in sync with what's in Connector, so for the moment I'll go with AGREEMENT_SETUP.  I'd like to rename it in a subsequent PR though.

repo: alphagov/pay-ledger


text:
BAU Allow users to be added to internal services

Add page to allow support and pay administrators to add users to
internal services.
It is often useful to use an existing internal service to test a
specific feature in each of the test, staging and production
environments. Access to those accounts currently involves asking someone
else to sign in and invite you through the admin tool.
Toolbox will restrict the operation to:

services that have the internal flag set to true

Admin users will throw an appropriate error if:

the service doesn't exist
the user already has access to the servic

On balance if this hasn't been necessary the last 2 years it shouldn't be needed, the placeholder code is here if the need comes back up!

repo: alphagov/pay-toolbox


text:
Reduce Sidekiq log verbosity

We're generating a lot of data by having sidekiq logging set to verbose. On the 26th of October we produced over 20G of log data on integration, the vast majority were a log of severity: DEBUG.
I don't believe we're making any use of the data it produces as we already log in the Rails application the parameters of a Content Item being sent downstream and if an error occurs we will still get all the information.
I've added a job id to our rails logging which should allow us to link our rails queuing to sidekiq logs.

Great, thanks @kevindew :)

authors: cbaines kevindew

repo: alphagov/publishing-api


text:
[DO NOT MERGE]Edit marriage abroad booking links

This is part of the work to update some links to the embassy/consulate booking service for Marriage Abroad smart answer outcomes.
FCDO are changing the supplier they use for these bookings so URLs are changing.
Trello card: https://trello.com/c/yoorhwiQ/2244-2-change-urls-in-marriage-abroad-smart-answer
⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠

Looks good to me

authors: AlanGabbianelli nicholsj

repo: alphagov/smart-answers


text:
[CLOSED] Remove use of exceptions for flow control.

 Issue by kalleth
Monday 16 June 2014 at 15:29 GMT
Originally opened as https://github.com/alphagov/specialist-publisher/pull/114

Previously, an error was being thrown by the fetchable gem deep within the call stack which was then being rescued way up the chain in the SpecialistDocumentsController.
By creating a DocumentFinder to be used by the various helpers, which prevents calling the service and instead returns nil in cases where we want to render a 404, we're able to remove this use of exceptions.
[+ @bestie]

kalleth included the following code: https://github.com/alphagov/specialist-publisher/pull/114/commits

 Comment by evilstreak
Wednesday 25 June 2014 at 14:41 GMT

I don't think this approach makes anything clearer. I'm going to close the pull request for now, but let's talk about it after we've cleared down the decks a bit.

repo: alphagov/specialist-publisher-rebuild


text:
Test that gem_layout renders feedback component

Trello card
This imports a test from Smokey^1 that confirms that the feedback component is present & functional.
The Smokey test does not currently meet the eligibility criteria for inclusion in the repository^3, but by adding this test to Static we can meet the "Second critical functional check" criterion.
n.b. I'm new to GDS, so please do look over it critically and let me know if I've made any mistakes 🙏

Thanks for the review & the great suggestions @ChrisBAshton 🙏

repo: alphagov/static


text:
Permit start date should relate to today's date

The permit start date that is shown when after a user chooses 'as soon as possible' should be 5 working days after today's start date.

This is fixed by #289

authors: petegale sanjaypoyzer

repo: alphagov/verify-local-patterns


text:
TT-1368 run CI for verify-saml-extensions on travis in parallel

Note: this needs a travis config change, ideally before merging
As part of a program to move PR builders to travis.
This may need to move to travis.com or to hosted travis later, for now this is a safe (the project is open source anyway and we are not using build artefacts yet) way to run tests on master and branches on travis, as a simple demonstration.
Tested here: https://travis-ci.org/swilson96/verify-saml-extensions
Before merging, we need someone with admin access to this repo to enable it in travis-ci.org. @kornysietsma is that you?

Closing, doesn't actually work on Travis for some reason, needs more work

repo: alphagov/verify-saml-extensions


text:
Add feed and email subscriptions for statistics page

This only adds feed and email for published statistics (ie https://www.gov.uk/government/statistics) and not for announcements (ie https://www.gov.uk/government/statistics/announcements)
Follow up to #2120.
Ticket: https://www.pivotaltracker.com/n/projects/1261204/stories/87099378

Makes sense to me :+1:

authors: boffbowsh evilstreak

repo: alphagov/whitehall


text:
Update past foreign secretaries

What
Update Past Foreign Secretaries Page to apply new Design + Design system + use Components
Why
On-going work to align architecture, have pages benefit from Accessibility improvements
Visuals

Before
After

Done when

[x] Release merged
[x] Design Reviewed
[ ] QA Content

Grouped related PRs
https://github.com/alphagov/whitehall/pull/6236
https://github.com/alphagov/whitehall/pull/6243
https://github.com/alphagov/whitehall/pull/6241
Anything else?
Add link to image card heading - this is not dependant
https://github.com/alphagov/govuk_publishing_components/pull/2277
Implementing a semantic list in this context + the Design System layout could be done various different ways.  This might prompt a change during review.

All looks good to me. Just one small comment to add with relation to -
https://github.com/alphagov/whitehall/pull/6243/files#diff-4c36183021f514ff3edc854264a1c1c1d3b5a954d499f383bd9a56cd5729cfeaR21
  .govuk-heading-s--foreign-secretary-profile {
    margin-bottom: 0;
  }

I wondered if an alternative approach could be to replace this CSS with the govuk-!-margin-bottom-0 class?

I wondered if an alternative approach could be to replace this CSS with the govuk-!-margin-bottom-0 class?

Good call, should work the same, i'll update and remove those extra lines.

@jon-kirwan updated! Tests were failing but needed a rebuild.

Moving into Draft (aim to close) as combined PR raised: https://github.com/alphagov/whitehall/pull/6264

authors: chris-gds jon-kirwan


text:
Update BreachDirectory

Updated BreachDirectory output formatting and README to reflect update

There is a small bug in the PrintBreachDirectory() function. The key passed to the TitleMap[] map is not correct ("Breachdirectory" instead of "BreachDirectory"). I think it will be a nice fit to this PR.

authors: MstchCmBck rohan-patra

repo: alpkeskin/mosint


text:
🛑 roothouse-2 is down

In 7366ff7, roothouse-2 (https://rhstatus-proxy.alterine0101.id/?deviceId=roothouse-2) was down:

HTTP code: 404
Response time: 264 ms

Resolved: roothouse-2 is back up in 2f4c42e.


text:
App::FilterEvent(wxEvent& e) should not use many ifs

This greatly slows the application, as each message is checked against 20 or more ifs

For now I am closing this issue as now events are processed faster. Later we could use array for mouse events in order to speedup their processing

repo: alternetsoft/AlternetUI