text:
The ultimate HIBP downloader: curl

Hi!
Thanks for HIBP and this downloader. At first I was considering using it, but the API of HIBP passwords is so easy that I wrote a small shell script for it. It was slow as hell, because it had no parallelism at all. It was far too slow for my taste, thus I was thinking about adding parallelism. And that's when I stumbled on curl's URL globbing feature.
curl is the swiss army knife of HTTP downloading and it supports patterns/globbing and massive parallelism, and pretty much every aspect of HTTP downloads (proxies, HTTP1/2/3, all SSL/TLS versions, etc.).
Here's a single curl commandline that downloads the entire HIBP password hash database into the current working directory:
curl -s --retry 10 --retry-all-errors --remote-name-all --parallel --parallel-max 150 "https://api.pwnedpasswords.com/range/{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}"

If you want to be able to check the curl output/log, remove the -s option and redirect to a file:
curl --retry 10 --retry-all-errors --remote-name-all --parallel --parallel-max 150 "https://api.pwnedpasswords.com/range/{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}" > curl.log 2>&1

To debug issues, you can add the -v (or the --trace-ascii -) option to increase verbosity.
By default curl does not retry any requests
On an always-free Oracle Cloud VM this finished (for me) in 13.5 minutes. ;-)
The URL globbing and the --remote-name-all options have been around in curl for ages (i.e. for over a decade) and the --parallel* options have been added in Sep 2019 (v7.66.0). So pretty much all "recent" Linux distros already contain a curl version that fully support this commandline.
Curl is cross-platform, e.g. you can download a Windows version too.
If you don't have the necessary 30-40 GB free space for the entire hash dump, you can get away with less by downloading in smaller batches and instantly compressing them.
Here's a command that you can fire and forget (i.e. disconnect / log out) on any Linux PC/server and it'll do the job in batches:
d="$(date "+%F")"
nohup bash -c 'd="'$d'"; chars=(0 1 2 3 4 5 6 7 8 9 A B C D E F); printf -v joined "%s," "${chars[@]}"; charscomma="${joined%,}"; hibpd="$(pwd)/hibp_${d}"; for c in "${chars[@]}"; do prefix="hibp_${d}_${c}" dir="${hibpd}/${prefix}"; mkdir -p "$dir"; cd "$dir"; date; echo "Starting in $dir with prefix $c ..."; curl -s --remote-name-all --parallel --parallel-max 150 -w "%{url}\n" "https://api.pwnedpasswords.com/range/${c}{$charscomma}{$charscomma}{$charscomma}{$charscomma}"; cd "$hibpd"; BZIP2=-9 tar cjf "${prefix}.tar.bz2" "$prefix" && rm -r "$prefix"; done; echo "Finished"; date' > "hibp_$d.log" 2>&1 &!

This doesn't support suspend and resume of the download job (other HIBP downloaders do), but since it finishes pretty quickly (if you have a good enough internet connection), I don't see any reason for this feature.
You can easily assemble the server responses into a single ~38 GB file with the following commandline (on Linux):
find . -type f -print | egrep -ia '/[0-9a-f]{5}$' | xargs -r -d '\n' awk -F: '{ sub(/\r$/,""); print substr(FILENAME, length(FILENAME)-4, 5) $1 ":" $2 }' > hibp_all.txt

You can sort it easily based on the second field to get the most "popular" hashes:
sort -t: -k2 -rn hibp_all.txt | head -n100

To get just the most popular hashes:
sort -t: -k2 -rn hibp_all.txt | head -n100 | cut -d: -f1

You can feed these into a pre-computed lookup table (like what crackstation.net provides) to get the most popular breached passwords. :)

Hey @muzso, that is an awesome approach and I'm kind a wondering why we never did that in the first place! I've added a reference to the readme, well done!

@troyhunt
Thanks for the mention in the readme.

makes use of a regular expression

It's not a regular expression. Curl calls it "globbing", which is a simple method for specification of string patterns.
See:

https://en.wikipedia.org/wiki/Glob_(programming)
https://curl.se/docs/url-syntax.html

Fixed!

Thanks.

For some reason this skipped a single file, A1FBF, for me. After downloading that missing file manually I had the complete set of 1048576 files.

@dextercd
Thanks for the feedback.
I've added some explanation to this issue's description to aid debugging any problems with the download.

I've tested this (with the --retry 10 and --retry-all-errors options) on a pretty different platform to double-check that it works as advertised: I ran it on my Google Pixel 6a phone (Android 15) via Termux and sshd.
It ran for 34 minutes and finished successfully, all 1048576 files were downloaded (with OK looking file sizes). :)

Nice work. In my experience awk is slow. So I tried with sed which is usually faster. However, sed can't open multiple files at once and be aware of their filename. So xargs with -I{} will call sed once per file, which ends up being slightly slower than your awk.
However I managed to get 25-30% speedup by moving the FILENAME manipulation and the removal of the \r from awk:
oliver@alpha:~/hibp/data$ time find . -type f -printf '%f\n' | egrep -ia '^[0-9a-f]{5}$' | head -n10000 | xargs awk -F: '{ print FILENAME $1 ":" $2 }' | tr -d '\r' > hibp_all2.txt

real	0m13.353s
user	0m13.062s
sys	0m1.846s
oliver@alpha:~/hibp/data$ time find . -type f -print | egrep -ia '/[0-9a-f]{5}$' | head -n10000 | xargs -r -d '\n' awk -F: '{ sub(/\r$/,""); print substr(FILENAME, length(FILENAME)-4, 5) $1 ":" $2 }' > hibp_all.txt

real	0m18.645s
user	0m17.648s
sys	0m0.917s
oliver@alpha:~/hibp/data$ diff hibp_all.txt hibp_all2.txt

[ no differences ]

above shown for 10000 files. So the suggested "improved" command line is:
find . -type f -printf '%f\n' | egrep -ia '^[0-9a-f]{5}$' | xargs awk -F: '{ print FILENAME $1 ":" $2 }' | tr -d '\r' > hibp_all.txt

@oschonrock

In my experience awk is slow. So I tried with sed which is usually faster. However, sed can't open multiple files at once and be aware of their filename. So xargs with -I{} will call sed once per file, which ends up being slightly slower than your awk.

That's the exact reason (1 million invocations of sed) why I didn't even give it (sed) a try.

However I managed to get 25-30% speedup by moving the FILENAME manipulation and the removal of the \r from awk

Great! :)
You've motivated me to spend some time/effort on performance optimization too. :)
As a first step, I've collected the input filenames into a file so the measurements are as little influenced by other factors as possible.
find . -type f -printf "%f\\n" | egrep -a "^[0-9A-F]{5}\$" | LC_ALL=C sort > ../filenames.txt

I've also created a ramdisk and copied the input files to it, so my SSD won't be a factor either.
In practise this didn't make much difference though as you'll see from the measurements.
I've included a run for the same code and both using my SSD and using the ramdisk, but the difference is within the margin of error (of measurement accuracy).
For the other/later tests I only used the ramdisk.
I've done quite a few variations.
# My original variant.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" awk -F: "{ sub(/\\r\$/,\"\"); print substr(FILENAME, length(FILENAME)-4, 5) \$1 \":\" \$2 }" > ../hibp_all_01.txt'

# run #1 (ssd)
real 835.48
user 773.24
sys 37.69

# run #2 (ramdisk)
real 824.02
user 760.14
sys 34.56

# This is your (@oschonrock) version.
# Removed the cutting of the file pathes since they already contain only the filenames, and moved the handling of carriage returns out of Awk and into a `tr` command.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" awk -F: "{ print FILENAME \$1 \":\" \$2 }" | tr -d "\\r" > ../hibp_all_02.txt'

# run #1 (ramdisk)
real 534.57
user 548.57
sys 54.11

# run #2 (ramdisk)
real 547.37
user 548.78
sys 54.11

# Removed the splitting of the lines (into fields) since we don't actually change the content. Originally I used fields so I could swap them if necessary.
# This resulted in a significant speed improvement.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" awk -F_ "{ print FILENAME \$0 }" | tr -d "\\r" > ../hibp_all_03.txt'

# run #1 (ramdisk)
real 144.48
user 148.94
sys 51.33

# run #2 (ramdisk)
real 176.83
user 152.15
sys 52.85

# Replaced the invocation of `tr` with Awk code.
# Note: the last line of the input files has no (carriage return) line ending which requires a conditional check.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" awk -F_ "{ n = length(\$0); c = substr(\$0, n, 1); if (c == \"\\r\") { print substr(FILENAME, length(FILENAME)-4, 5) substr(\$0, 1, n-1) } else { print substr(FILENAME, length(FILENAME)-4, 5) \$0 } }" > ../hibp_all_04.txt'

# run #1 (ramdisk)
real 360.04
user 325.25
sys 31.98

# run #2 (ramdisk)
real 372.78
user 333.47
sys 32.89

# Removed cutting of the file pathes.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" awk -F_ "{ n = length(\$0); c = substr(\$0, n, 1); if (c == \"\\r\") { print FILENAME substr(\$0, 1, n-1) } else { print FILENAME \$0 } }" > ../hibp_all_05.txt'

# run #1 (ramdisk)
real 310.47
user 277.24
sys 31.71

# run #2 (ramdisk)
real 322.22
user 281.16
sys 32.67

# `sed` performed an order of magnitude worse for me (than any other approach).
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -I{} sed -r "s#(.*[^\\r])\\r?\$#{}\\1#;\$a\\" "{}" > ../hibp_all_06.txt'

# run #1 (ramdisk)
real 5718.04
user 4959.86
sys 732.44

# Readded cutting of file pathes to the 3rd variant to see how much slower this makes the processing.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" awk -F_ "{ print substr(FILENAME, length(FILENAME)-4, 5) \$0 }" | tr -d "\\r" > ../hibp_all_07.txt'

# run #1 (ramdisk)
real 225.38
user 213.45
sys 53.76

# run #2 (ramdisk)
real 225.76
user 211.46
sys 53.21

# Using the GNU Parallel tool (see: https://www.gnu.org/software/parallel/) to split the input processing between multiple instances of the Awk script.
# I had to use a file for the Awk script, because I couldn't make it work with Parallel otherwise.
echo "{ print FILENAME \$0 }" > ../merge.awk
/usr/bin/time -p sh -c 'cat ../filenames.txt | parallel -k -m awk -F_ -f ../merge.awk | tr -d "\\r" > ../hibp_all_08.txt'

# run #1 (ramdisk)
real 257.27
user 259.53
sys 93.95

# run #2 (ramdisk)
real 188.13
user 260.53
sys 94.89

# Removed the "-k" option from `parallel` to see whether it makes a difference if we don't care about output order.
/usr/bin/time -p sh -c 'cat ../filenames.txt | parallel -m awk -F_ -f ../merge.awk | tr -d "\\r" > ../hibp_all_09.txt'

# run #1 (ramdisk)
real 171.47
user 257.66
sys 94.26

# A very simple Python implementation. I'm not a Python expert though, so it is possible that it can be significantly improved.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" ../merge.py > ../hibp_all_10.txt'

# run #1 (ramdisk)
real 352.35
user 321.56
sys 25.73

# Running multiple instances of the Python script in parallel.
/usr/bin/time -p sh -c 'cat ../filenames.txt | parallel -k -m ../merge.py > ../hibp_all_11.txt'

# run #1 (ramdisk)
real 204.11
user 785.95
sys 92.09

# A simple Perl implementation, but I'm not a Perl expert either.
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" ../merge.pl > ../hibp_all_12.txt'

# run #1 (ramdisk)
real 372.66
user 345.37
sys 25.23

# Multiple parallel instances of the Perl implementation.
/usr/bin/time -p sh -c 'cat ../filenames.txt | parallel -k -m ../merge.pl > ../hibp_all_13.txt'

# run #1 (ramdisk)
real 218.11
user 877.60
sys 86.22

# A simple Golang implementation.
# CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o merge_go merge.go
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" ../merge_go > ../hibp_all_14.txt'

# run #1 (ramdisk)
real 835.03
user 402.81
sys 513.66

# Multiple parallel instances of the Golang implementation.
/usr/bin/time -p sh -c 'cat ../filenames.txt | parallel -k -m ../merge_go > ../hibp_all_15.txt'

# run #1 (ramdisk)
real 404.63
user 811.26
sys 1594.51

Here's the code of merge.py:
#!/usr/bin/env python3

import sys
import os

# Check if any files were provided as arguments
if len(sys.argv) < 2:
    print("Usage: {} file1 file2 ...".format(sys.argv[0]))
    sys.exit(1)

# Iterate over each file provided in the command line arguments
for file_path in sys.argv[1:]:
    try:
        # Open the file for reading
        with open(file_path, 'r', encoding='ascii') as file:
            # Extract just the filename from the file path
            file_name = os.path.basename(file_path)

            # Process each line in the file
            for line in file:
                # Strip the CR+LF (or any new line character)
                line = line.rstrip('\r\n')

                # Print the filename and the line with only LF at the end
                print(f"{file_name}{line}")

    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
    except IOError as e:
        print(f"Error reading file '{file_path}': {e}")

Here's the code of merge.pl:
#!/usr/bin/env perl

use strict;
use warnings;

# Check if any files were provided as arguments
if (@ARGV == 0) {
    die "Usage: $0 file1 file2 ...\n";
}

# Iterate over each file provided in the command line arguments
foreach my $file_path (@ARGV) {
    # Open the file for reading
    open(my $fh, '<', $file_path) or die "Could not open file '$file_path': $!\n";

    # Extract just the filename from the file path
    my ($file_name) = $file_path =~ m|([^/\\]+)$|;

    # Process each line in the file
    while (my $line = <$fh>) {
        # Remove the CR+LF at the end of the line
        chomp $line;
        $line =~ s/\r$//;  # Removes the carriage return if present

        # Print the filename and the line with only LF at the end
        print "$file_name$line\n";
    }

    # Close the file handle
    close($fh);
}

And here's the code of merge.go:
package main

import (
	"bufio"
	"fmt"
	"os"
	"path/filepath"
	"strings"
)

func main() {
	// Check if any files were provided as arguments
	if len(os.Args) < 2 {
		fmt.Printf("Usage: %s file1 file2 ...\n", filepath.Base(os.Args[0]))
		os.Exit(1)
	}

	// Iterate over each file provided in the command line arguments
	for _, filePath := range os.Args[1:] {
		// Open the file
		file, err := os.Open(filePath)
		if err != nil {
			fmt.Printf("Error: Could not open file '%s': %v\n", filePath, err)
			continue
		}
		defer file.Close()

		// Extract the filename from the file path
		fileName := filepath.Base(filePath)

		// Use bufio.Scanner to read the file line by line
		scanner := bufio.NewScanner(file)
		for scanner.Scan() {
			// Get the line and remove any CR+LF (Windows-style newlines)
			line := scanner.Text()
			line = strings.TrimRight(line, "\r")

			// Print the filename and the line with only LF at the end
			fmt.Printf("%s%s\n", fileName, line)
		}

		// Check if there were any errors during file reading
		if err := scanner.Err(); err != nil {
			fmt.Printf("Error reading file '%s': %v\n", filePath, err)
		}
	}
}

It seems that the best performance came from this:
cat ../filenames.txt | xargs -r -d "\\n" awk -F_ "{ print FILENAME \$0 }" | tr -d "\\r" > ../hibp_all_03.txt

So the best commandline seems to be this (this one must be executed in the directory where the HIBP files were downloaded):
find . -type f -printf "%f\\n" | egrep -a "^[0-9A-F]{5}\$" | LC_ALL=C sort | xargs -r -d "\\n" awk -F_ "{ print FILENAME \$0 }" | tr -d "\\r" > hibp_all.txt

Great stuff... I particularly like the sort of filenames before merge, that's essential!
You may also be interested in a little project I have been working on which is all about working with this dataset.
https://github.com/oschonrock/hibp
The key is... it uses "binary storage", which means the on disk size is halved. And binary means that each "record" is the same size, and that means we can use "binary search". So it does that and provides a "local server" you can connect to for queries.
The hibp_downloader uses libcurl with curl_multi under the hood, which is what curl --parallel  uses, so it benefits from libcurl's great implementation of HTTP2 multiplexing, which makes this great speed possible. I get about a 13min download from a cheap Virtual Machine on Fasthosts. It buffers up to 300 (by default, but tunable) text file contents in memory, so that it can write straight to a continuous file on disk, while keeping the order correct! hibp_download uses a second thread to convert the txt into the binary format and write to disk, so it keeps the network busy and bloaty text versions never see the disk.
hibp_server is a very lean server with only ~5MB resident memory footprint and can serve ~1kreq/s on a single and old CPU assuming a common consumer grade SSD.
All the tools are very lean on memory and disk space. All you need is enough disk space for one copy of the binary DB (current ~ 21GB) and a tiny amount of memory (about ~50MB during download, and ~5MB during serve). My goal was to make this feasible on a "small" size Virtual Machine, which you can get for a few $ per month, say 50GB SSD, 2 cores, and 2GB memory.
I have build instructions for Linux, FreeBSD and Windows, and it is tested on those. If this is popular I am happy to maintain packages.
Feedback very welcome!

Great project! :)

My goal was to make this feasible on a "small" size Virtual Machine, which you can get for a few $ per month, say 50GB SSD, 2 cores, and 2GB memory.

You can get a VM with 4 vcpus, 24 GB RAM and 50 GB storage (don't remember whether it's HDD or SSD) for free from Oracle Cloud. :) I have been using this for over a year now. They don't even charge you for egress traffic (Google does even in its "always-free" plan) ... or at least not at the amount of traffic I'm generating.

I will check that out. Interesting that disk size is the limiting factor. This DB is so large that even just one (text) copy of it almost fills that VM. And your curl + find + awk combiner temporarily requires 2 copies on disk which blows the storage budget.
If you had time to build my project and try it out on your VM that I would really appreciate it, and the feedback will help improve it.
Thanks.

# A simple Golang implementation.
# CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o merge_go merge.go
/usr/bin/time -p sh -c 'cat ../filenames.txt | xargs -r -d "\\n" ../merge_go > ../hibp_all_14.txt'

# run #1 (ramdisk)
real 835.03
user 402.81
sys 513.66

I am amazed the go version didn't do better. Much slower than ptyhon, perl and find+awk. Something not right there...?
In any case, what I realised when coding my project is there are 2 slow things here. Network and Disk. We want to use disk as little as possible and write to disk while we are downloading, so that time is essentially for free. My code also does the convert to binary and that is free too.

I am amazed the go version didn't do better. Much slower than ptyhon, perl and find+awk. Something not right there...?

I've found this interesting too. No idea why it came out this way. Didn't spend any time on optimizing the Python/Perl/Go implementations for performance.

The key is... it uses "binary storage", which means the on disk size is halved. And binary means that each "record" is the same size, 24 bytes, and that means we can use "binary search".

Storing the data in a proprietary binary format is not necessarily an advantage. It consumes less space and can be more efficient to search, but requires custom code/tooling to process, while a text-based format can be more easily processed with a number of tools. There're pros and cons on both sides.

Storing the data in a proprietary binary format is not necessarily an advantage. It consumes less space and can be more efficient to search, but requires custom code/tooling to process, while a text-based format can be more easily processed with a number of tools. There're pros and cons on both sides.

That's true of course. I can easily add a switch to spit out the text version. It would still be faster and use half the disk space (because no need for temp copy). Let me know if that is of interest?
What code/tooling exists to do something useful with the text file?
That's what I never understood. I started using this dataset many years ago, and used to insert it into a mysql DB with with the SHA1 as the primary index, and I used to convert it into binary on the way into the db. That process of importing was super slow and almost started breaking as the data grew, and the queries were like avg 45ms, quite slow?   That's why I started the with this custom tool, for querying, and later (because of your curl idea) for the download as well.
What do you with 38GB of text?

If you need the hashes for continuous use (e.g. integrated into an authentication system), then I agree that storing it in a performance-optimized format makes sense.
If you need it just for a one-off examination, research, etc., then dealing with a custom storage format might be an unnecessary overhead.

What do you do with 38GB of text?

As demonstrated: you can grep it or write a shell/awk script to process it, etc.
Usually I can do some light processing (statistics, etc.) faster in shell scripting or Awk, then writing any program code (Python, Go, etc.) for the same task. Of course if I want something for a longer period of use, a performance optimized solution is better.
It depends on your use case.

As demonstrated: you can grep it or write a shell/awk script to process it, etc.

I agree, of course. It's the UNIX way. I just literally can't think of a use case for this data, other than "look up a password and get the count". The SHA1 is non-intelligible by design.
Your use case above, is literally "download the data and combine it", ie "just getting the data in the first place". I can't think of another one.
But I guess you can never imagine all use cases.. I will add a "text export" switch.. both during download, and also if you already have the binary version. That way you never have to "store the 38GB of text", unless you specifically want to, and can just pipe it to the next process.
Just working on "resuming downloads"... some users have reported that their internet is so flaky they can't get through the 38GB without a failure (even with retries), so "resume" does seem useful, and is not so hard.
Tory has made such a fabulous resource available here, but unless you are happy with his online api (security questions...), then using it is actually quite hard: The .NET downloader, the file size, what to query it with... That's what I was trying to solve.

@muzso
FYI..
hibp_download  --text-out
and
hibp_download --resume
now both work. Although not together... ie you can't resume a text download.

Hi there,
When the goal is to search for a specific hash, to see if it has been pawned, splitting the hashes by starting character is the best choice, so that we can speed up the search by only searching the file that starts with the first character of its hash.
The script below was written for ZSH, but I believe it will work in BASH. It was also written assuming that you are downloading multiple files instead of just one. However, it will work with just one file, as you can see by studying the code.
Apparently SHA1 only uses hexadecimal characters (A-F and 0-9), so this will create 16 files. In practice, it multiplies the search speed by 16, since the other 15 will not be analyzed in an intelligent search.
#!/bin/zsh
count=0
for i in pwnedpasswords/*.txt; do
    while read LINE; do
        arr=({{A..F},{0..9}}) # apparently sha1 is hexadecimal and never uses {G..Z}
        for l in "${arr[@]}"; do
            if [[ "${LINE:0:1}" == "$l" ]]; then
                echo "$LINE" >> "hashes/$l.txt"
            fi
        done
    done < $i
    count=$(($count + 1))
    echo "Total files analyzed: $count"
done

Apparently SHA1 only uses hexadecimal characters (A-F and 0-9)

SHA1 is a hash function. It maps an arbitrary amount of bytes (on its input) into a sequence of 20 bytes (on its output). Its usual printable representation is hexadecimal which uses only characters a-f (or A-F) and 0-9.

When the goal is to search for a specific hash, to see if it has been pawned, splitting the hashes by starting character is the best choice, so that we can speed up the search by only searching the file that starts with the first character of its hash.
The script below was written for ZSH, but I believe it will work in BASH. It was also written assuming that you are downloading multiple files instead of just one. However, it will work with just one file, as you can see by studying the code.
Apparently SHA1 only uses hexadecimal characters (A-F and 0-9), so this will create 16 files. In practice, it multiplies the search speed by 16, since the other 15 will not be analyzed in an intelligent search.

Selecting a file at runtime is slow, because fopen() or similar, is a system call. Alternatively, keeping 1million files open is not realistic. The files need to be combined into a single file.
You may want to consider trying my project on this topic which has moved on a lot since above posts:
https://github.com/oschonrock/hibp
There are:

an even more efficient downloader  (using curl_multi in http2 mode under the hood): 5min download on Gbit connection
multiple hash representations sha1, ntlm and sha1(truncated to 64bits)
optional custom binary format, enabling binary search with just 30 disk accesses  to find record: >1000queries/s
optional "table of contents" index, by the N most significant bits, reduced disk accesses by half by default and 3x on queries/s - kind of what you are suggesting but in one file
binary fuse filters 8 and 16bit... 5-10 less storage, and >100,000queries /s   (but with a small false +ve rate)

have a look, feedback wanted.

authors: dextercd muzso nerun oschonrock troyhunt

repo: HaveIBeenPwned/PwnedPasswordsDownloader


text:
Release of pretraining and fine tuning code

Hi, thanks for a very nice piece of work.  Do you also plan to release pretraining and fine tuning code for this model?  Also, is there a way to apply the model on long sequences, such as those from the LRA benchmark?  Thanks!

thanks!

Code here: https://github.com/HazyResearch/safari We don’t have a config for fine tuning, but will look to add it soon!
…
On Wed, Mar 8, 2023 at 2:10 PM ksrinivs64 @.> wrote: Hi, thanks for a very nice piece of work. Do you also plan to release pretraining and fine tuning code for this model? Also, is there a way to apply the model on long sequences, such as those from the LRA benchmark? Thanks! — Reply to this email directly, view it on GitHub <#20>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDDIIWGFGZAM2YVAJGZHLDW3DDSRANCNFSM6AAAAAAVUC3GUA . You are receiving this because you are subscribed to this thread.Message ID: @.>

Hello, I am also interested in the fine-tuning code. If you do not have the time to cobble it together can you provide some hints on how a fine-tuning harness can be constructed? I really want to push this to its limits.
Also a  few points.

Great work. I honestly believe this is the path to true coherent multi-modality
I tried out the models and I am impressed with the context length.
I did notice a higher propensity to hallucinate gibberish at extreme lengths but I assume that is due to model size.

Thanks for all the hard work. You're a rock star.

Thank you for the kind words!
If you do not have the time to cobble it together can you provide some hints on how a fine-tuning harness can be constructed?

In safari, we'll need to put in some hooks for loading a pre-trained model: https://github.com/HazyResearch/safari/blob/main/train.py#L170. The H3 model definition in that repo has slightly different parameter names than the one in this repo, so we may need to have some custom code to rename the model parameters upon loading the state dict. Then it should work.
We'll try to get this implemented soon!

Just came here to say I'm interested in the fine-tuning code as well! Great work! Listening to the podcast from Deep Papers and found out about you guys!
Been looking for a solution to the long context window for a little while now. So I'm excited to start training it on custom data!

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Multilabel classifier for images

Hey
I am currently working on a multilabel classifier for images and am curious if we can construct a multilabel classifier using reef. I have a dataset with more than 6 labels.
Looking forward to your response

I have same question @paroma .
How does apply_heuristics() in heuristic_generator.py works in case of multi-class scenario. ?
@shantanuaggarwal2695 are you able to resolve this issue ?

repo: HazyResearch/reef


text:
usuario crash

Heberon parece que hay un usuario que esta crasheando el servidor, podrías checarlo?

el nick es KoopaRed

license: MIT


text:
Skipping beyond just the intro

2020-04-10 10:51:15,076 :: bw_plex :: DEBUG :: plex.py:1191 :: Found One-Punch.Man.s02e06 theme start 00:25, theme end 06:35, ffmpeg_end 00:00 progress 12:20 fake_progress 12:30 best_time 06:35 credits_start 00:00 credits_end 00:00,
Happens randomly with different shows. This was the longest skip so far, its a little annoying and defeats the purpose of having this installed :(
Any ideas?

Nope, i need the entire log.
This program has serval issues but in this case there we fail to detect the correct end of the intro
This can be of different reasons but most of the times it because the show uses the theme song different places in the first 15 minutes and we try pick correct theme sequence and fail.
The best corse is action would be to manually set the correct time (it’s cli option)


text:
Listener update

Listeners now provide the old / new values for the object.

Very nice integration of this feature! 👍

source: gharchive/pull-request

authors: ByRicardoSimoes tiggerbiggo

repo: HephaistosCorp/Loki


text:
Replace govendor with dep

Hi @joehandzik,
You could move the vendoring management before v1.0.0 from govendor to dep, and a vendoring update would be great too.
Thanks!

Hmm, I think we'll need to save this for v2.0.0 or sneak it into a v1.1.0 or something like that. Apologies for tagging so quickly, but we're trying to move fast to track to a product release. We're happy to work with you to find a path that works for everyone!

Hi @joehandzik ,
I just realized that the log options are not present anymore.

authors: joehandzik mjtrangoni

license_source: bigquery


text:
Dev analyzer.py

PR Checklist
Please check if your PR fulfills the following requirements:

Issue : #18
[x] The commit message follows next guideline
(1) It must has 'issue' number
(2) over 50 lines or significant changes related to the main function, the commit message must have a 'title-content' structure.
(3) Title, capitalize the first letter of the first letter of the title
(4) At the end of the title '.' Ban
(5) The title is an imperative tone
(6) Change the line every 72 characters in English
(7) Write the text according to what and why rather than how
[x] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[x] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[x] Documentation content changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[x] Yes
[ ] No

Good job@!

repo: HibikeQuantum/Bear-Auto-Publisher


text:
Fake Vive Controller on the ground.

I have noticed that there is always a fake vive controller on the ground. Sometimes games decide to use it and need to get a quick reboot. But there are a select amount of games that always use that fake vive controller and never use any input from the psmove controllers. I want to use psmoveservice as a reliable source of PC VR and if I'm going to start buying games, I need to know that they'll work. Does anyone know how to override the Vive controller input and force all games to use the PSMoveService. Isn't it supposed to do that anyways?

It looks like the steamvr driver API provides a constant for defining a generic tracker rather than a controller:
https://github.com/ValveSoftware/openvr/blob/bcac1bfbe2be3c1ff37e4f62b8e412fcdf29248e/headers/openvr_driver.h#L164
I'm betting if we provided a config option to say that given psmove controller should label itself as a generic tracker that you would get the behavior you would expect. I'll have to add support for this though. It shouldn't be that complicated. I'll see if I can get to making a test build for you to try out this next week with that option.

Thank you for looking into this!

repo: HipsterSloth/PSMoveSteamVRBridge


text:
headTracking - can add a third psmove on the headset to Have HeadTracking ?

Hello there,
can add a third psmove on the headset to Have HeadTracking ?
thanks for this exelent works

Hello @ychoucha,
This sort of question should be asked on the google group.
The answer depends on what headset you have. If you're using VRidge it can be done using PSMoveFreepieBridge. If you're not then it's probably not currently possible but that's what issue #1 is about.

Hello,
thanks for u respond,  i m using Pimaxx 4k-UHD, the psmove controler works perfectly with 2 psEye, now i m just looking to make the last things missing on this headset the headTracking
thanks


text:
added file

I have added a useless file
Do not accept this pull request

Shut up! I want to watch Netflix!

authors: APaganini Hoeppke

repo: Hoeppke/OxPythonInduction


text:
1, HumanPoseのプログラムを作成する

HumanPoseのモデルを読み込んで実装するための関数を作成する
関数名: HumanPose
引数: frame
リターン: flag (bool)
構成:
1, モジュールのインポート
2．モデルのパスを指定する
3、モデルを読み込む
4、frameデータを読み込んだモデルを使って分析させる
模索中
5、現在の重心のジョイントを検出する
6、5で検出したジョイントの位置を保存する
7、新しいフレームで検出した重心のジョイントの座標と6のジョイントの座標との差文をとる
8、重心の座標の変化が激しかったら(6で分析した重心の座標+- 5)酔っているとみなしてTrueを返す。

各ジョイントの座標の割り出しは何とかできそう。
あるnフレーム目の時とn-1フレーム目の時の差分を出して変化の大きさを調べられるかも？
表示の仕方がかなりややこしいため、上記の処理の前に値の変換を行う必要がありそう

とりあえずは上記のやり方でうまくできそう


text:
setWifiCallbackAll

Hi
Thank you for adding "setWifiCallbackAll "
I'm trying to use it but I'm failed to do it
I have a function called  void startNTP(), I used  homeSpan.setWifiCallback(startNTP); to startNTP after wifi connected.
I want to use homeSpan.setWifiCallbackAll
I tried this
int count=0;
if( WiFi.isConnected()){
count++;
}
homeSpan.setWifiCallbackAll(startNTP(count>0));
Help me the best way I can use it
Thank you so much

I'm not sure you need to call startNTP more than once after WiFi is called.  I don't know exactly what the routine does, but it would likely recover automatically once WiFi is re-established without needing to be started again.  Also, note that the ESP32 has built-in NTP functionality which is called from homeSpan.enableWebLog() so you can set the clock this way without needing startNTP() at all.
Regardless, if you want to use setWifiCallbackAll(void (*func)(int count)), note that as per the documentation:

the function func must be of type void and accept a single int argument, count, into which HomeSpan passes the number of times WiFi has been established

If you've not used C/C++ a lot, this syntax can sometimes cause confusion.  In your example above, when you write startNTP(count>0), you are NOT passing a function, but rather you are passing the return value of startNTP (whatever that may be), which is incorrect.
To use setWifiCallbackAll(void (*func)(int count)) you would still simply write homeSpan.setWifiCallbackAll(startNTP).  The key is that startNTP() itself needs to be defined to accept a single int argument like this: void startNTP(int myCount).  This allows HomeSpan to PASS YOU the number of times it has re-established WiFi.  Your code regarding WiFi.isConnected() above should be deleted - you do NOT need to compute any of this yourself.
In sum, if you've used homeSpan.setWifiCallback(startNTP), you call homeSpan.setWifiCallbackAll(startNTP) in exactly the same way, except that you modify the definition of your function from void startNTP(){} to void startNTP(int myCount){} and inside that function myCount will automatically provide the number of times WiFi has been re-established. I purposely used myCount instead to count to emphasize that this is a variable internal to startNTP() and can be named anything you want.  All HomeSpan knows is that it is going to pass an int to startNTP() as the first (and only) argument - it does not care what you call it.
Note that many of the HomeSpan callbacks work in this same manner (it is a standard C/C++ construct, also extensively used by the ESP32 IDF code).

@HomeSpan Thank you so much for your support and good explanation.
With your guidance, now the code is running very well.
Thank you

btw, in “setup()”, you can always call “sntp_set_time_sync_notification_cb(your_callback);”
This gets called about every 3 hours and you can use that to get the ntp time sync.

I'm not sure you need to call startNTP more than once after WiFi is called. I don't know exactly what the routine does, but it would likely recover automatically once WiFi is re-established without needing to be started again. Also, note that the ESP32 has built-in NTP functionality which is called from homeSpan.enableWebLog() so you can set the clock this way without needing startNTP() at all.

I use NTP library in order to get this kind of statement in my code:
if ( ntp.hours()==17 && ntp.minutes()>=30){
flag_for_motion =true;
}
May I use  homeSpan.enableWebLog() to get the same statement as that? if it is possible I can use it to get some free space,
My sketch size is (94%) of program storage space

This is a good question - I just added a new FAQ that provides all the details in case others have the same question as well.
Note if you find your sketch is getting too big, you can compile using a larger partition scheme by selecting a new one directly from the Tools Menu on the Arduino IDE.  I tend to use the one labeled "Minimal SPIFFS" instead of the default.  The default reserves 1.5MB of flash for a SPIFFS partition, which is not used at all by HomeSpan.  Under "Minimal SPIFFS" the useable flash memory for your sketch increases by about 50%, and you should find the sketch now takes up only 63% of the program space.

Thank you

authors: HomeSpan Nezaemmy frankonski

repo: HomeSpan/HomeSpan


text:
Add PlatformIO compatibility

A minor adjustment that makes the library compatible with PlatformIO. Tested on both PlatformIO and Arduino, it seems to be working fine.

What does HomeSpan show as the Arduino-ESP32 version number in the Serial Monitor when run under PlatformIO?  I'm not sure it will show the correct value.

Yeah, it's showing Arduino-ESP Ver.: ARDUINO_ESP32_GIT_DESC

I don't have PlatformIO installed but my understanding from others is a problem with the GitHub deployment - it does not include core_version.h.  However, Arduino.h does contain alternate version information, even though it is not always updated correctly.    I can add in some logic to use this information anyway when compiling with PlatformIO to avoid any compile errors.

However, it includes esp_arduino_version.h and we can get the version number from there.
https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/esp_arduino_version.h
It worked out, now it shows correctly Arduino-ESP Ver.: 2.0.3

Yes, this should work, but only for versions 2.0.4 and 2.0.3.  For some reason Espressif forgot to update the version numbers in 2.0.2 and 2.0.1 - they both show 2.0.0 instead.  If not for that, I would not need to use core_version at all since arduino_version.h can be used with PlatformIO as well Arduino.  I'll add this into the next release.

I am not sure what has changed, but PlatformIO seems to compile with no errors and pick up the correct Arduino platform version now without these fixes applied. So I think this can be closed as irrelevant for now.
However, I am getting a bunch of errors compiling sketch with Arduino platform version 2.0.2. Seems like now HomeSpan requires at least version 2.0.3.
In file included from .pio/libdeps/esp32/HomeSpan/src/HomeSpan.h:44,
                 from include/DEV_Sensor.hpp:1,
                 from src/main.cpp:52:
.pio/libdeps/esp32/HomeSpan/src/Utils.h:84:10: error: 'touch_value_t' does not name a type; did you mean 'touch_pad_t'?
   static touch_value_t threshold;
          ^~~~~~~~~~~~~
          touch_pad_t
.pio/libdeps/esp32/HomeSpan/src/Utils.h:174:33: error: 'touch_value_t' has not been declared
   static void setTouchThreshold(touch_value_t thresh){threshold=thresh;}
                                 ^~~~~~~~~~~~~
.pio/libdeps/esp32/HomeSpan/src/Utils.h: In static member function 'static boolean PushButton::TRIGGER_ON_TOUCH(int)':
.pio/libdeps/esp32/HomeSpan/src/Utils.h:107:66: error: 'threshold' was not declared in this scope
   static boolean TRIGGER_ON_TOUCH(int pin){return(touchRead(pin)<threshold);}
                                                                  ^~~~~~~~~
.pio/libdeps/esp32/HomeSpan/src/Utils.h: In static member function 'static void PushButton::setTouchThreshold(int)':
.pio/libdeps/esp32/HomeSpan/src/Utils.h:174:55: error: 'threshold' was not declared in this scope
   static void setTouchThreshold(touch_value_t thresh){threshold=thresh;}
                                                       ^~~~~~~~~
.pio/libdeps/esp32/HomeSpan/src/Utils.h:174:55: note: suggested alternative: 'thresh'
   static void setTouchThreshold(touch_value_t thresh){threshold=thresh;}
                                                       ^~~~~~~~~
                                                       thresh
In file included from include/DEV_Sensor.hpp:1,
                 from src/main.cpp:52:
.pio/libdeps/esp32/HomeSpan/src/HomeSpan.h: At global scope:
.pio/libdeps/esp32/HomeSpan/src/HomeSpan.h:720:33: error: 'touch_value_t' has not been declared
   static void setTouchThreshold(touch_value_t thresh){PushButton::setTouchThreshold(thresh);}
                                 ^~~~~~~~~~~~~

Thanks for catching the issue with touch_value_t.  Looks like this was introduced in 2.0.3.  I incorrectly assumed it was part of the full 2.0 branch.  Will address in next release.

authors: HomeSpan oleksiikutuzov


text:
Brew Upgrade Error After Uninstalling SketchyBar

brew config output
HOMEBREW_VERSION: 3.4.11-118-g1d1474a
ORIGIN: https://github.com/Homebrew/brew
HEAD: 1d1474a03f713fee59619b8707e142c05771add4
Last commit: 2 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 371d3b49e9f40873263a59e18955e54973766922
Core tap last commit: 3 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_EDITOR: code
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.8 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 13.1.6 build 1316
Git: 2.36.1 => /opt/homebrew/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.4-arm64
CLT: 13.4.0.0.1.1651278267
Xcode: 13.4
Rosetta 2: false

brew doctor output
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew-provided
script of the same name. We found the following "config" scripts:
  /Users/behnam/opt/anaconda3/bin/icu-config
  /Users/behnam/opt/anaconda3/bin/krb5-config
  /Users/behnam/opt/anaconda3/bin/freetype-config
  /Users/behnam/opt/anaconda3/bin/xslt-config
  /Users/behnam/opt/anaconda3/bin/libpng16-config
  /Users/behnam/opt/anaconda3/bin/libpng-config
  /Users/behnam/opt/anaconda3/bin/xml2-config
  /Users/behnam/opt/anaconda3/bin/python3.9-config
  /Users/behnam/opt/anaconda3/bin/python3-config
  /Users/behnam/opt/anaconda3/bin/curl-config
  /Users/behnam/opt/anaconda3/bin/ncursesw6-config
  /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10-config
  /Library/Frameworks/Python.framework/Versions/3.10/bin/python3-config

Verification

[X] I ran brew update and am still able to reproduce my issue.
[X] I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?
I installed Sketchybar using its documentation. Then I uninstalled it (again, according to the documentation).
What happened (include all command output)?
(base) ➜  ~ brew update                                                                                16:33
Already up-to-date.
brew:4: command not found: sketchybar

And also:
(base) ➜  ~ brew upgrade                                                                                16:36
brew:4: command not found: sketchybar

What did you expect to happen?
To not get the brew:4: command not found: sketchybar error.
Step-by-step reproduction instructions (by running brew commands)
brew tap FelixKratz/formulae
brew install sketchybar

brew services start sketchybar
brew services stop sketchybar

brew uninstall sketchybar
brew untap FelixKratz/formulae

brew upgrade

Sounds like the sketchybar command is still being invoked by the brew shell script, and since your brew doctor output isn't warning about unexpected changes to Homebrew itself, I can only think of one other possibility right now.
What's the output of echo $BASH_ENV? If it's not empty, check the contents of the file pointed to by $BASH_ENV.

Sounds like the sketchybar command is still being invoked by the brew shell script, and since your brew doctor output isn't warning about unexpected changes to Homebrew itself, I can only think of one other possibility right now.
What's the output of echo $BASH_ENV? If it's not empty, check the contents of the file pointed to by $BASH_ENV.

I tried echo $BASH_ENV and the output was empty. I then restarted terminal and for some reason it fixed the problem.

authors: gromgit ibehnam

repo: Homebrew/brew


text:
audit: --new-formula fails for uses_from_macos [...], since: :monterey

brew doctor output
CI for homebrew/homebrew-core

Verification

[X] My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
[X] I ran brew update twice and am still able to reproduce my issue.
[X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

brew config output
CI for homebrew/homebrew-core

What were you trying to do (and why)?
As already stated in Homebrew/Discussions#4130, I am trying to build a formula for micromamba (homebrew/homebrew-core#120924). For this, I need curl >= 7.66.0 but macOS 11 ships an older version. To fix this issue, I added uses_from_macos "curl", since: monterey to the formula.
  [...]
  depends_on "yaml-cpp"
  depends_on "zstd"

  uses_from_macos "curl", since: :monterey # uses CURLINFO_RETRY_AFTER, available since curl 7.66.0
  uses_from_macos "krb5"
  uses_from_macos "libarchive", since: :monterey
  uses_from_macos "zlib"
  [...]

What happened (include all command output)?
On macOS 11, the following happens:
==> brew audit micromamba --online --new-formula
==> FAILED
Full audit micromamba --online --new-formula output
  micromamba:
    * Dependency 'curl' is provided by macOS; please replace 'depends_on' with 'uses_from_macos'.
    * Dependency 'libarchive' is provided by macOS; please replace 'depends_on' with 'uses_from_macos'.
  Error: 2 problems in 1 formula detected

What did you expect to happen?
brew audit micromamba --online --new-formula should pass since I already use uses_from_macos with a good reason for since: monterey.
The formula for davix also uses uses_from_macos "curl", since: monterey. But this probably went through since the check only happens for new formulae.
Step-by-step reproduction instructions (by running brew commands)
Checkout `pavelzw/homebrew-core micromamba` on macOS 11
`brew audit micromamba --online --new-formula`

This would be nice to address but:

brew audit micromamba --online --new-formula should pass

--new-formula is expected to fail in many circumstances in which we manually override it.
Ideally this would be a e.g. non-required CI check instead.

@MikeMcQuaid could you please take a look at homebrew-core#122551 and override the --new-formula fails?
(Sorry for creating a second PR for this; when updating micromamba to 1.3.0, I accidentally pushed outdated commits of homebrew-core which lead to the branch being behind master and the PR getting closed -.-)

override the --new-formula fails?

Just comment in the PR and the relevant maintainer will take it into account.

Passing on this, sorry!

authors: MikeMcQuaid pavelzw


text:
Brew audit fails on rubocop gem download

Please follow the general troubleshooting steps first:

[x] Ran brew update and retried your prior step?
[x] Ran brew doctor, fixed as many issues as possible and retried your prior step?

Bug reports:
When doing:
brew audit --strict --online formula
I get:
==> Installing or updating 'rubocop' gem
ERROR:  Could not find a valid gem 'rubocop' (= 0.45.0), here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.org/specs.4.8.gz)
Error: Failed to install/update the 'rubocop' gem.

Brew doctor reports no errors; everything is up to date. Brew audit was working some days ago (I pushed the paraview formula on homebrew science lately). So something has changed.
I also tried:
brew uninstall --force openssl
brew install openssl

And read:
https://github.com/Homebrew/brew/issues/1396
brew config returns:
HOMEBREW_VERSION: 1.1.0-136-g34f08a9
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: 34f08a9512f86a37af5e95b83f47f5d69c92998c
Last commit: 2 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 25e8d2fd9178a977f050c59e568d187aa3aab097
Core tap last commit: 4 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_REPOSITORY: /usr/local/Homebrew
HOMEBREW_CELLAR: /usr/local/Cellar
HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
CPU: quad-core 64-bit haswell
Homebrew Ruby: 2.0.0-p648
Clang: 8.0 build 800
Git: 2.10.2 => /usr/local/bin/git
Perl: /usr/bin/perl
Python: /usr/local/bin/python => /usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7
Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby
Java: 1.8.0_102
macOS: 10.12.1-x86_64
Xcode: 8.1
CLT: 8.1.0.0.1.1476494586
X11: 2.7.11 => /opt/X11

env | grep -i gem and env | grep -i ruby returns nothing.

brew audit requires rubocop for the style checks, and it's failing to download from rubygems.  Are you behind a proxy?

No, at home. And I changed nothing (at least, not I am aware of). It worked 2 days ago.
curl -I  https://rubygems.org/specs.4.8.gz
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Yesterday I went to the rubygems.org website with Google Chrome, and there it worked. So I guess Chrome has it's own list of certificates?
Just to let you know that I retried and it now works. I did nothing on my system since I opened the bug. The certificate seems to be valid again. I would have liked to understand what happened ... but the problem does not seem to be on my side but on the rubygems side.

authors: dunn iMichka


text:
update-report: use appropriate pronoun in update message

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?
[ ] Have you successfully run brew man locally and committed any changes?

Before:
You have 1 outdated formula installed.
You can update them with brew upgrade.

After
You have 1 outdated formula installed.
You can update it with brew upgrade.

Review period will end on 2021-02-22 at 16:37:12 UTC.

Makes sense, thanks @dtrodrigues!

Review period ended.

authors: BrewTestBot MikeMcQuaid dtrodrigues


text:
python: stop passing no-user flag on installation

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew style with your changes locally?
[ ] Have you successfully run brew typecheck with your changes locally?
[x] Have you successfully run brew tests with your changes locally?
[ ] Have you successfully run brew man locally and committed any changes?

--no-user is not a supported end-user flag in pip, and is explicitly non-documented: https://github.com/pypa/pip/issues/8977#issuecomment-706535089
https://github.com/pypa/pip/commit/87bb4f9c13d5a4f1c5b746670705ae044112f854
It was added as part of building python packages from source in case a user's configuration specified --user installs due to the fact that you cannot do a --user install within a venv:
$ pip install --user foo
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Recent versions of pip fall back to a user installation if site-packages is not writeable. Additionally, when building a formula, only the global and site pip configurations are looked at -- the user configuration is empty within ..../.brew_home/, so this change in Homebrew doesn't impact users who have user installs as a local configuration.

Review period will end on 2021-02-22 at 18:07:50 UTC.

Yes - a user's global pip configuration does affect builds, but not their local configuration (since $HOME is replaced by .brew_home) or their environment variables (filtered out).
Running on macOS, during a build from source (for example for the jc python formula), the pip configuration is looked for in
global:
  /Library/Application Support/pip/pip.conf, exists: False
site:
  /opt/homebrew/Cellar/jc/1.14.3/libexec/pip.conf, exists: False
user:
  /private/tmp/jc-20210219-35155-hhi3f/jc-1.14.3/.brew_home/.pip/pip.conf, exists: False
  /private/tmp/jc-20210219-35155-hhi3f/jc-1.14.3/.brew_home/.config/pip/pip.conf, exists: False

On Linux, it is
global:
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: False
site:
  /home/linuxbrew/.linuxbrew/Cellar/jc/1.14.3/libexec/pip.conf, exists: False
user:
  /tmp/jc-20210220-30481-ijb3gc/jc-1.14.3/.brew_home/.pip/pip.conf, exists: False
  /tmp/jc-20210220-30481-ijb3gc/jc-1.14.3/.brew_home/.config/pip/pip.conf, exists: False

There are some settings which can be beneficial when passed into the Homebrew python building of a formula from source (see https://github.com/Homebrew/discussions/discussions/863). The only way to manipulate those is by changing the global config file, which typically requires administrative privileges.
The cases in which we want to allow a user to influence their formula builds and how much is an open question and a tradeoff between reproducibility and customization.
FWIW, the --isolated flag for pip doesn't do anything for us since it only ignores user configurations, and those are already ignored -- it doesn't isolate from site or global configurations: https://github.com/pypa/pip/issues/3828

Gotcha, thanks

Makes sense 👍🏻

Review period ended.

authors: BrewTestBot MikeMcQuaid Rylan12 dtrodrigues


text:
rubocops: use Style/UnlessLogicalOperators

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[x] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

This replaces Style/UnlessMultipleConditions, which was added in https://github.com/Homebrew/brew/pull/10256
The autocorrect part of UnlessMultipleConditions has been adapted for UnlessLogicalOperators in a pending PR: https://github.com/rubocop/rubocop/pull/9553

Review period will end on 2021-03-04 at 09:51:25 UTC.

Review period ended.

Thanks @SeekingMeaning!

authors: BrewTestBot MikeMcQuaid SeekingMeaning


text:
update.sh: always use stable branch for tags.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew tests with your changes locally?

This avoids creating a new branch that’ll never be deleted for each tag and differentiates between the master and stable branches.

@zmwangx Thoughts?


text:
Fix "undefined local variable" error in update-report

[x] Have you followed the guidelines in our Contributing document?
[ ] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[ ] Have you successfully run brew style with your changes locally?
[x] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

Resolves an error introduced in https://github.com/Homebrew/brew/commit/2451d017f9ebbf6b3f5fd79f15e048d2c109145a#diff-05ec45190dab3db11df80b2c154648c8023837fa34917dce9d1bcde0acf9c005L282
If you have a few minutes and you're willing to untap core, you should be able to repro with:
$ brew untap homebrew/core; HOMEBREW_NO_INSTALL_FROM_API=1 brew update-report
Untapping homebrew/core...
Untapped 3 commands and 6552 formulae (15,331 files, 710.4MB).
==> Tapping homebrew/core
Cloning into '/opt/homebrew/Library/Taps/homebrew/homebrew-core'...
remote: Enumerating objects: 1429703, done.
remote: Counting objects: 100% (318/318), done.
remote: Compressing objects: 100% (157/157), done.
remote: Total 1429703 (delta 190), reused 268 (delta 161), pack-reused 1429385
Receiving objects: 100% (1429703/1429703), 549.69 MiB | 3.90 MiB/s, done.
Resolving deltas: 100% (997762/997762), done.
Tapped 3 commands and 6554 formulae (6,906 files, 604.5MB).
Error: undefined local variable or method `core_tap' for Homebrew:Module
Please report this issue:
  https://docs.brew.sh/Troubleshooting
/opt/homebrew/Library/Homebrew/cmd/update-report.rb:318:in `install_core_tap_if_necessary'
/opt/homebrew/Library/Homebrew/cmd/update-report.rb:115:in `output_update_report'
/opt/homebrew/Library/Homebrew/cmd/update-report.rb:42:in `update_report'
/opt/homebrew/Library/Homebrew/brew.rb:93:in `<main>'

I've also enabled typechecking in update-report, which would have caught this error

Review period skipped due to critical label.

I'm still seeing this in CI: https://gitlab.haskell.org/haskell/cabal/-/jobs/1398914
+ brew update
==> Downloading https://ghcr.io/v2/homebrew/portable-ruby/portable-ruby/blobs/sha256:cf9137b1da5568d4949f71161a69b101f60ddb765e94d2b423c9801b67a1cb43
######################################################################## 100.0%
==> Pouring portable-ruby-2.6.8_1.arm64_big_sur.bottle.tar.gz
==> Homebrew has enabled anonymous aggregate formula and cask analytics.
Read the analytics documentation (and how to opt-out) here:
  https://docs.brew.sh/Analytics
No analytics have been recorded yet (nor will be during this `brew` run).
==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
  https://github.com/Homebrew/brew#donations
==> Tapping homebrew/core
Cloning into '/private/var/lib/gitlab-runner/builds/7KQsmDei/0/haskell/cabal/.brew/Library/Taps/homebrew/homebrew-core'...
Tapped 3 commands and 6[55](https://gitlab.haskell.org/haskell/cabal/-/jobs/1398914#L55)9 formulae (6,911 files, 605.5MB).
Error: undefined local variable or method `core_tap' for Homebrew:Module

I'm still seeing this in CI: https://gitlab.haskell.org/haskell/cabal/-/jobs/1398914
+ brew update
==> Downloading https://ghcr.io/v2/homebrew/portable-ruby/portable-ruby/blobs/sha256:cf9137b1da5568d4949f71161a69b101f60ddb765e94d2b423c9801b67a1cb43
######################################################################## 100.0%
==> Pouring portable-ruby-2.6.8_1.arm64_big_sur.bottle.tar.gz
�==> Homebrew has enabled anonymous aggregate formula and cask analytics.
Read the analytics documentation (and how to opt-out) here:
  https://docs.brew.sh/Analytics
No analytics have been recorded yet (nor will be during this `brew` run).
==> Homebrew is run entirely by unpaid volunteers. Please consider donating:
  https://github.com/Homebrew/brew#donations
==> Tapping homebrew/core
Cloning into '/private/var/lib/gitlab-runner/builds/7KQsmDei/0/haskell/cabal/.brew/Library/Taps/homebrew/homebrew-core'...
Tapped 3 commands and 6[55](https://gitlab.haskell.org/haskell/cabal/-/jobs/1398914#L55)9 formulae (6,911 files, 605.5MB).
Error: undefined local variable or method `core_tap' for Homebrew:Module

Hmm, not sure why that would be the case, perhaps there's some sort of caching happening? It looks like you might be using the latest master in CI, which I suspect is in inherently more fragile than checking out an official release: https://gitlab.haskell.org/haskell/cabal/-/blob/master/.gitlab/brew.sh#L6
You could pin that to a specific release for improved stability. You could also try to check out the latest release, something like this might do the trick:
git checkout $(git describe --tags `git rev-list --tags --max-count=1`)

@gMan1990 I'm sorry you're having this issue. What does brew --version output?

@dduugg
Homebrew 4.0.5
Homebrew/homebrew-core (git revision 0f4afdec394; last commit 2023-03-10)

Darwin edydeMacBook-Pro.local 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64

Oh, this fix didn't make it into the latest release 🤦
Let me see if someone can cut a new release for you.

Homebrew version 4.0.6 was just released, which incorporates this patch.

authors: BrewTestBot dduugg gMan1990 hasufell


text:
Homebrew-on-Linux.md: switch to pacman -S

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] ~Have you written new tests for your changes? Here's an example.~
[x] Have you successfully run brew style with your changes locally?
[ ] ~Have you successfully run brew typecheck with your changes locally?~
[ ] ~Have you successfully run brew tests with your changes locally?~

I realized the convention in documentations is to recommend -S, not -Syu, as a system upgrade is usually an overkill. This change also makes Homebrew's documentation more consistent.

Makes sense, thanks!

authors: MikeMcQuaid osalbahr


text:
🔨  Refactor Cask exceptions.

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.
[x] Have you successfully run brew tests with your changes locally?

Also fixes https://github.com/caskroom/homebrew-cask/issues/25095.

Altered some more messages.

I'm also getting this error.

MacOS.release was renamed to MacOS.version some time ago, seems like you still have some old Casks installed which are using MacOS.release.

MacOS.release was renamed to MacOS.version some time ago, seems like you still have some old Casks installed which are using MacOS.release.

That was basically it. My Library/Taps/caskroom/homebrew-cask was sitting on an old branch instead of master. Thanks.

Okay. I'm seeing reasons in the error messages as expected now.
$ mkdir /tmp/foo.rb
[/usr/local/Homebrew/Library/Taps on ⇄ pr-2771]
$ brew cask install /tmp/foo.rb
Error: Cask 'foo' is unavailable: '/tmp/foo.rb' is not a file.
Error: nothing to install
[✘ /usr/local/Homebrew/Library/Taps on ⇄ pr-2771]
$ brew cask install bogus-cask
Error: Cask 'bogus-cask' is unavailable: No Cask with this name exists.
Error: nothing to install
[✘ /usr/local/Homebrew/Library/Taps on ⇄ pr-2771]
$ brew cask install bogus-cask /tmp/foo.rb http://www.example.com/bogus/nonexistent-cask.rb
Error: Cask 'bogus-cask' is unavailable: No Cask with this name exists.
Error: Cask 'foo' is unavailable: '/tmp/foo.rb' is not a file.
==> Downloading http://www.example.com/bogus/nonexistent-cask.rb.

curl: (22) The requested URL returned error: 404 Not Found
Error: Cask 'nonexistent-cask' is unavailable: Failed to download http://www.example.com/bogus/nonexistent-cask.rb.
Error: nothing to install

How about having info and uninstall (and maybe audit and fetch and zap) give warnings with suggestions the way install and reinstall do?
[/usr/local/Homebrew/Library/Homebrew/cask on ⇄ pr-2771]
$ brew cask install powermat
Error: Cask 'powermat' is unavailable: No Cask with this name exists. Did you mean one of:
powermate
Error: nothing to install
[✘ /usr/local/Homebrew/Library/Homebrew/cask on ⇄ pr-2771]
$ brew cask info powermat
Error: Cask 'powermat' is unavailable: No Cask with this name exists.
[✘ /usr/local/Homebrew/Library/Homebrew/cask on ⇄ pr-2771]
$ brew cask uninstall powermat
Error: Cask 'powermat' is unavailable: No Cask with this name exists.
[✘ /usr/local/Homebrew/Library/Homebrew/cask on ⇄ pr-2771]
$ brew cask reinstall powermat
Error: Cask 'powermat' is unavailable: No Cask with this name exists. Did you mean one of:
powermate
Error: nothing to install
[✘ /usr/local/Homebrew/Library/Homebrew/cask on ⇄ pr-2771]
$

That always happens around a cask = CaskLoader.load(cask_token) call with a rescue around it. Maybe Hbc::CLI could get a load_cask_with_suggestions method to wrap it for convenience?

@apjanke, added a casks method to iterate over Casks given as arguments.

authors: apjanke reitermarkus


text:
mame 0.178

version bump
build and install the latest man
remove inreplace for removing 64 suffix, just use rename install (no need for if block as 32-bit support was dropped earlier)

rom-tools should be bumped too

Exactly. I didn't have time to look at its changelog yet, but don't expect much change this time. Feel free to open a PR if interested.

Cool, I was just making sure you knew they get updated together since I saw the one PR and not the other.

authors: ilovezfs tomyun

repo: Homebrew/homebrew-games


text:
Claim a bounty

zen adress: znShA6Xhamp7PoQMyJ1r4Lr7uqiK59WP37H

Zen wallet znajmRPQt4YGGpoMNmjHDpKzTq6oDgzjpfJ

repo: HorizenOfficial/Sidechains-SDK


text:
ratatui Version issue

Because this crate returns ratatui::text::Text, as ratatui evolves the types here don't line-up with future version. For example at this point you can do something like
fn render_text(s: &str) -> Text {
    md_to_tui::MarkdownParsable::parse_markdown(&some, None).unwrap_or_else(|_| {
        Text::raw(s)
    })
}

will fail if the crate consuming this crate us using a different version of ratatui, right now this would provide an error like the following
error[E0308]: mismatched types
   --> src/work.rs:199:13
    |
3 | /             Text::raw(s)
    | |______________^ expected `ratatui::text::Text<'_>`, found `Text<'_>`
    |
    = note: `Text<'_>` and `ratatui::text::Text<'_>` have similar names, but are actually distinct types
note: `Text<'_>` is defined in crate `ratatui`

I am not sure what the right way to fix this but I wanted to report it as I ran into it today.

thanks for telling me about it. I kinda forgot about this crate. I don't know if i will be working on it anymore.  But for now if you want make any changes to fix it feel free and, then I can merge it.

authors: FreeMasen Horryportier

repo: Horryportier/md-to-tui


text:
ActiveAdmin Integration

Just wondering what would be the best way to hook this into activeadmin for an admin user?

Hi, I try your active admin branch
Unfortunately...
undefined local variable or method `has_one_time_password' for AdminUser (call 'AdminUser.connection' to establish a connection):Class

At first time I try to implement it a got diff errors. For now -- no error on loginning to AA (I got code genereted for user in db) but i still go to admin panel just with login and password -- no second step auth :( Any advice? (active admin from master branch)

@Houdini I'm looking forward into integrating this gem into my app (which is using ActiveAdmin 1.0.0pre2). As the branch you linked is a few years old and many commits behind master, would it be possible to add the explanation to the readme, or support ActiveAdmin in the master branch too?
When comparing branches it seems like I'd have to fork master and change a few lines to integrate it, but that feels cumbersome, and even then I'm unsure if it will also work with ActiveAdmin 1.x, and not the 'old' 0.6 versions.
Thanks a lot!

@maia It's possible, but I don't know to long does it take.
Problem is that it's very hard to make same code work for both general purpose and AA.
But I should check new version of AA

authors: Houdini SergRain clarakwan maia

license: mit


text:
Troubles importing modules (for example: Utilities)

Testing a new feature that I implemented, the following error appears when trying to run Wallet.py:
Traceback (most recent call last):
  File "Wallet.py", line 26, in <module>
    from Utilities.Utility import get_data_from_file, gen_encryption_key
ModuleNotFoundError: No module named 'Utilities'

I noticed that running the file in a separate subfolder leads to that problem. (Importing Utilities being in the same folder that the module can be done without problem)
Of course, the solution for this is running the script in the same directory where the module is. But this can only be done in two ways:

Moving Wallet.py to the parent folder, where the module is. The problem is that if I must move to the parent folder all the python scripts that use the modules, I'll end up breaking the initial and clean structure of the proyect.
"If Mohammed doesn't go to the mountain then the mountain must go to Mohammed"... The second option is moving the module to the folder where Wallet.py is. But obviously, it won't be still accesible by other scripts that require it, and if we moved them also to the same folder we would end up in the same mess that in case one.

So... What can I do?

Yes, Like i said in the other issue, this branch is currently in development and is NOT functional. Once I get to functional Ill branch it and dev on a different branch.

repo: HourGlss/HourCrypto


text:
isize/usize and range syntax

int/uint types and suffixes have been replaced with isize/usize
range and range_inclusive has been replaced with range syntax

Big thanks to @pfalabella for most of the isize/usize changes!

This is great! Thanks!

authors: aochagavia jamesrhurst

repo: Hoverbear/rust-rosetta

license: unlicense


text:
HTMLUnit loads wrong locale file for Javascript Datepicker

Executing a jquery datepicker loads the wrong script from i18n resource folder. Instead of executing the default script (en_US), HTMLUnit executes the last file of i18n resource folder.
Datepicker is implemented link in : https://jqueryui.com/datepicker/#localization

Any chance to get more details or a demo page/folder and a short main method that shows your problem?

After consulting with our developers, it seems that the problem is indeed on our side. The I18n files are faulty.
Sorry for that :)

Great and thanks for the feedback

authors: dennisduysak rbri

repo: HtmlUnit/htmlunit


text:
About find_scale_from_depth inputs

https://github.com/Huangying-Zhan/DF-VO/blob/e9d91bad0fe4db84030511715718b53d40fe593f/vo_modules.py#L766-L768
Why the depth map is taken from current sample (self.cur_data['depth']), and not taken from reference?
According to function definition and implementation in https://github.com/Huangying-Zhan/DF-VO/blob/e9d91bad0fe4db84030511715718b53d40fe593f/vo_modules.py#L582 the depth should correspond to kp2 (which is ref), and not to kp1 (cur).
What am I missing?
Thanks for this work!

Hi @DanRos22 , sorry for late reply.
Please refer to the previous issue #7 .
Simple explanation will be:
cur_data[self.cfg.translation_scale.kp_src] actually stores kp_ref
while
ref_data[self.cfg.translation_scale.kp_src][ref_id] stores kp_cur.
The reason is mentioned in #7

repo: Huangying-Zhan/DF-VO


text:
block js like window.alert()

Can vex.dialog.alert be made to block execution like the native alert box?

No, there is no browser API that would make this possible. Any custom dialog must be asynchronous.

repo: HubSpot/vex


text:
Server Side Rendering

Hi,
correct me if I'm wrong but actually the mere fact of requiring the lib breaks any server bundle since a11y-dialog uses directly document.
ReferenceError: document is not defined
at node_modules/a11y-dialog/a11y-dialog.js:19:37
[...]

Trying to find some workarounds right now.

Interesting. We don’t have the problem at all at N26, even though we have an isomorphic application.

It should be fixed with https://github.com/HugoGiraudel/react-a11y-dialog/releases/tag/4.0.1. My bad!

authors: HugoGiraudel pascalduez


text:
Fix typo (max-width to min-width)

Small typo fix.
Breakpoints are mobile first (min-width), and example is graceful degradation (max-width).

Thanks. :)

authors: HugoGiraudel goschevski

repo: HugoGiraudel/sass-guidelines


text:
Cosmetic fixes

This PR consists of two separate commits: the first one was just reordering stuff around (see edit below) to better respect the line length limits.
(EDIT: Well, now I re-checked and I remembered that in the first commit I also modified some stuff because Nim initializes its variables to their zero value: see for example the changes in proc trapz and proc cumtrapz)
The second one changes const gaussWeights to be an array and not a Table — see if you're ok with this change. Arrays are usually faster than tables, but in this case it probably won't matter.

Thanks 😄 I should probably be running nimpretty or something on this 😅
Wow, I totally looked passed arrays 🤣 I had Table's O(1) in mind and didn't look at something that obvious.
I will take a look at the code tomorrow and approve it. 😉
Have tried using NumericalNim? What is your general opinion on it thus far?

You are right that it shouldn't really make a big difference though 😉 it just gets loaded on time and is neglectable compared to the rest of the logic in the proc. But every optimization is welcome 😁

I've pushed cosmetic changes for other two files too.

Have tried using NumericalNim?

I haven't had the time yet.
But if only it existed about 1.5–2 years ago, I would have used it in some my PhD-related stuff :) I was doing some comparisons (in Python, IIRC) between various ODE methods, both explicit (including but not limited to RK4 and DOPRI you have here) and implicit (Lobatto, Radau, Gauss-Legendre, etc.). The most from this list — maybe this gives you some ideas what to implement next :)

What is your general opinion on it thus far?

I have to have a better look before I can judge and give my opinion :) The changes in this PR are because this is my way of quickly skimming through the code. I would like to dig deeper but I cannot promise when that might happen, due to my lack of time.

Awesome! 😁
Totally understandable 😄👍 you must be skimming through lots of code 😉 seems like I should try out making a implicit method up next. Thanks for the link and for taking your time 😊 it helps immensely to not develop in a complete vacuum just for my own curiosity.
I'm also coming from a Python background and Nim was really easy to get started with. It really is a wonderful mix between ease of use and performance.

All the tests were successful 👍 Thanks once again for your support 😄

authors: HugoGranstrom narimiran

repo: HugoGranstrom/numericalnim


text:
Fixed typo in german translation

was the wrong time

thanks for the PR

authors: HugoGresse mueller-ma

repo: HugoGresse/Anecdote


text:
Added More Commands (ping, serverinfo, reboot)

Added some commands to the robot

Lovely
@doctormagico thanks for "allowing" me to use ur bot for testing ❤

@Huguitis

The best Pull Requester in the world :D

repo: HuguitisNodes/HuguitisNodesBot


text:
Capture request duration in Cloudwatch Metrics

This work captures Visualizing results defined in this doc
Changes:

request CreationDate is stored in DynamoDB Cache table during POST request
request_id is propagated to all subtasks in order to provide access to CreationDate in Cache table
Duration Cloudwatch Metric is published at the ends of the Reducer and Converter subtasks

metric includes Number of Bundles and Output Format dimensions

Do we also want to track duration metrics for cache hits? This is not included in these changes as I think it would dilute the results that we actually care about (full request processing times).

Most of the changed files are from propagating request_id through

repo: HumanCellAtlas/matrix-service


text:
（未回答的问题请删除，减少多余信息干扰）

描述错误
影视推荐内容模块，更新推荐时，出现

环境：
-操作系统及版本：win10 64位
-软件安装包及版本：V2.7.1

正如提示所说,你的网络无法直接访问Github上的文件.你要么使用代理,要么等待内置的推荐更新.

正如提示所说,你的网络无法直接访问Github上的文件.你要么使用代理,要么等待内置的推荐更新.

repo: Hunlongyu/ZY-Player


text:
请教下CMS类的资源站是否只能1次获取最多20个对象？

我搜了几个资源站，好像基本都是：
列表接收参数：
ac=list
t=类别ID
pg=页码
wd=搜索关键字
h=几小时内的数据
...

返回的json中的limit都是20，那是否意味着没有办法一次性请求返回超过20个？
（我看了下 本项目中tools.js的代码，貌似也没有相应的处理。 若是我漏看了烦请指正）

我看了一下苹果 cms和版本有关，最新几个版本支持 pagesize 参数

我看了一下苹果 cms和版本有关，最新几个版本支持 pagesize 参数

是否很多cms站点都会遵行苹果cms的协议？ 如果是的话，那也许过一段时间这些cms站点就可以通过他们提供的api加上pagesize参数取数据了吧。

应该是的 毕竟这么多站要采集，但我测试了好几个资源站都没更新，估计都停留在很久之前了

authors: Hiram-Wong dbsxdbsx


text:
JGO is calling

Hey we moved to discord. We would love to have you back and catch up with your latest projects! (I am an OP along with NegativeZero, Ray, and ra4king) but mostly everyone is here.

Hey a few of us JGOers are gonna be in west virginia u should hop on so we can talk about meetups

Are you guys really still stalking me :)

want the discord link?

come on cody just stop in for an hour

repo: Hydroque/LuaConsole


text:
Hygieia Github scm collector :     ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.

Geeting the following error while running the github-scm-collector
c.c.d.collector.CollectorTask - Running Collector: GitHub
2020-10-09 13:25:00,003 INFO  c.c.d.collector.CollectorTask - ------------------------------
2020-10-09 13:25:00,003 INFO  c.c.d.collector.CollectorTask - Starting...
2020-10-09 13:25:00,003 INFO  c.c.d.collector.CollectorTask - ------------------------------
2020-10-09 13:25:00,005 ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
org.springframework.core.convert.ConversionFailedException: Failed to convert from type java.lang.String to type com.capitalone.dashboard.model.CollectorType for value 'AgileTool'; nested exception is java.lang.IllegalArgumentException: No enum constant com.capitalone.dashboard.model.CollectorType.AgileTool
at org.springframework.core.convert.support.ConversionUtils.invokeConverter(ConversionUtils.java:41) ~[spring-core-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.core.convert.support.GenericConversionService.convert(GenericConversionService.java:192) ~[spring-core-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.core.convert.support.GenericConversionService.convert(GenericConversionService.java:176) ~[spring-core-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readMap(MappingMongoConverter.java:948) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:217) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.readValue(MappingMongoConverter.java:1185) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.access$200(MappingMongoConverter.java:78) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$MongoDbPropertyValueProvider.getPropertyValue(MappingMongoConverter.java:1133) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.getValueInternal(MappingMongoConverter.java:869) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:282) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter$1.doWithPersistentProperty(MappingMongoConverter.java:270) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mapping.model.BasicPersistentEntity.doWithProperties(BasicPersistentEntity.java:309) ~[spring-data-commons-1.11.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:270) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:231) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:191) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:187) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:78) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:2206) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1884) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1707) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1690) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:602) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.repository.support.SimpleMongoRepository.findAll(SimpleMongoRepository.java:268) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.repository.support.SimpleMongoRepository.findAll(SimpleMongoRepository.java:193) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at org.springframework.data.mongodb.repository.support.SimpleMongoRepository.findAll(SimpleMongoRepository.java:47) ~[spring-data-mongodb-1.8.0.RELEASE.jar!/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_72-internal]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_72-internal]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_72-internal]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_72-internal]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(RepositoryFactorySupport.java:475) ~[spring-data-commons-1.11.0.RELEASE.jar!/:na]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:460) ~[spring-data-commons-1.11.0.RELEASE.jar!/:na]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:432) ~[spring-data-commons-1.11.0.RELEASE.jar!/:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:61) ~[spring-data-commons-1.11.0.RELEASE.jar!/:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at com.sun.proxy.$Proxy45.findAll(Unknown Source) ~[na:na]
at com.capitalone.dashboard.collector.GitHubCollectorTask.clean(GitHubCollectorTask.java:89) ~[github-scm-collector-2.0.0-SNAPSHOT.jar!/:2.0.0-SNAPSHOT]
at com.capitalone.dashboard.collector.GitHubCollectorTask.collect(GitHubCollectorTask.java:127) ~[github-scm-collector-2.0.0-SNAPSHOT.jar!/:2.0.0-SNAPSHOT]
at com.capitalone.dashboard.collector.CollectorTask.run(CollectorTask.java:63) ~[core-2.0.0-SNAPSHOT.jar!/:2.0.0-SNAPSHOT]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81) [spring-context-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72-internal]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72-internal]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72-internal]
Caused by: java.lang.IllegalArgumentException: No enum constant com.capitalone.dashboard.model.CollectorType.AgileTool
at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_72-internal]
at org.springframework.core.convert.support.StringToEnumConverterFactory$StringToEnum.convert(StringToEnumConverterFactory.java:59) ~[spring-core-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.core.convert.support.StringToEnumConverterFactory$StringToEnum.convert(StringToEnumConverterFactory.java:45) ~[spring-core-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.core.convert.support.GenericConversionService$ConverterFactoryAdapter.convert(GenericConversionService.java:425) ~[spring-core-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
at org.springframework.core.convert.support.ConversionUtils.invokeConverter(ConversionUtils.java:35) ~[spring-core-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]
... 50 common frames omitted

What version of collector are you running? You can download the latest snapshot or release versions from Maven. The information is available in Readme section of hygieia-scm-github-collector

What version of collector are you running? You can download the latest snapshot or release versions from Maven. The information is available in Readme section of hygieia-scm-github-collector

Hi Vema    I have cloned latest repo  and used maven 3.6.2 and build it

repo: Hygieia/hygieia-scm-github-collector


text:
BUG(?): link to non-existent commit

18 silly fetchPackageMetaData error for ytdl-core@github:fent/node-ytdl-core#a2d8b7e41e2514d77ae58af3c8a4d5343f2f4465 Command failed: /usr/bin/git checkout a2d8b7e41e2514d77ae58af3c8a4d5343f2f4465
18 silly fetchPackageMetaData fatal: в дереве нет такой ссылки: a2d8b7e41e2514d77ae58af3c8a4d5343f2f4465
26 error code 128

package-lock.json refers to a non-existing commit node-ytdl-core and command npm i gives error code 128.
The change of commit hash to the last fixes this.

I send pull request with fix the problem.


text:
Compare function URL construction error.

Describe the bug
Compare function is not correctly set. Loading a run list shows the following pop-up windown message.
Compare function failed
TypeError: URL constructor: http://(ids, token) => 'http://acme.com:3030/report/foobar?q=' + ids.join('&q=') + "&token="+token is not a valid URL.

The above has been obfuscated slightly but demonstrates the point.
To Reproduce
Open this run list /run/list/20 to see the UI reporting the construction error.
Version
0.12.1

Java
What is the version of Java ?
17

The compare function changed with release 0.12. It is now and Horreum will append id=<runId> for each selected run

repo: Hyperfoil/Horreum


text:
Export project bundle

Is your feature request related to a problem? Please describe.
Auditors, Reviewers and some other individuals/roles at our funders, because of their existing systems and processes require to have a Dossier of each completed application.
These individuals will not have access to the hypha system and while the application is approved the information still has to go through some processes further upstream in the organisation.
Describe the solution you'd like
We'd like to be able to extract/export all information relating to a Project in one zip archive that contains

generated PDFs of the concept note and proposal
all attached documents
communication/messages
reviews
determinations

Describe alternatives you've considered
Manually extract this information by printing  or screen-shooting webpages, downloading documents.
Additional context
NA
Priority

Medium.

We will need to have the dossier for each project. We are still in early stages of using hypha and will probably need it in 6 - 8 weeks.
Affected roles

Staff (Assigned lead)

We understand that there are different visibility levels set for reviews and determinations and assume that the Lead will redact wherever necessary.
Ideal deadline
Mid - End September 2021

Related to #2327

authors: fourthletter killapop

repo: HyphaApp/hypha


text:
tts problem

I converted it to tts. There was no problem with the conversion. It didn't play, and when I tried to download it, it said failed and prohibited.

you must follow the issue template, check your download configuration

authors: blaisewf kbuwel

repo: IAHispano/Applio


text:
[Error  :MaxWasUnavailable.LethalModDataLib] Failed to save data to file LCSaveFile2.moddata with key LethalLevelLoader.LLLSaveFile.LLLSaveFile2.customItemDictionary! Exception: System.NullReferenceException: Object reference not set to an instance of an object

[00:03:07.9974744] [Error  :MaxWasUnavailable.LethalModDataLib] Failed to save data to file LCSaveFile2.moddata with key LethalLevelLoader.LLLSaveFile.LLLSaveFile2.customItemDictionary! Exception: System.NullReferenceException: Object reference not set to an instance of an object
at ES3Writer.Write[T] (System.String key, System.Object value) [0x0001b] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001B
at ES3.Save[T] (System.String key, T value, ES3Settings settings) [0x0001e] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001E
at ES3.Save[T] (System.String key, T value, System.String filePath) [0x00009] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_0009
at LethalModDataLib.Features.SaveLoadHandler.SaveData[T] (T data, System.String key, System.String fileName) [0x0004f] in <2e27e651b8b1406f89ea9ae6d53c5d1e>:IL_004F
[00:03:08.0059776] [Error  :MaxWasUnavailable.LethalModDataLib] Failed to save data to file LCSaveFile2.moddata with key LethalLevelLoader.LLLSaveFile.LLLSaveFile2.allItemsList! Exception: System.NullReferenceException: Object reference not set to an instance of an object
at ES3Writer.Write[T] (System.String key, System.Object value) [0x0001b] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001B
at ES3.Save[T] (System.String key, T value, ES3Settings settings) [0x0001e] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001E
at ES3.Save[T] (System.String key, T value, System.String filePath) [0x00009] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_0009
at LethalModDataLib.Features.SaveLoadHandler.SaveData[T] (T data, System.String key, System.String fileName) [0x0004f] in <2e27e651b8b1406f89ea9ae6d53c5d1e>:IL_004F
[00:03:08.0085087] [Error  :MaxWasUnavailable.LethalModDataLib] Failed to save data to file LCSaveFile2.moddata with key LethalLevelLoader.LLLSaveFile.LLLSaveFile2.itemSaveDataList! Exception: System.NullReferenceException: Object reference not set to an instance of an object
at ES3Writer.Write[T] (System.String key, System.Object value) [0x0001b] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001B
at ES3.Save[T] (System.String key, T value, ES3Settings settings) [0x0001e] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001E
at ES3.Save[T] (System.String key, T value, System.String filePath) [0x00009] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_0009
at LethalModDataLib.Features.SaveLoadHandler.SaveData[T] (T data, System.String key, System.String fileName) [0x0004f] in <2e27e651b8b1406f89ea9ae6d53c5d1e>:IL_004F
[00:03:08.0115653] [Error  :MaxWasUnavailable.LethalModDataLib] Failed to save data to file LCSaveFile2.moddata with key LethalLevelLoader.LLLSaveFile.LLLSaveFile2.itemSaveData! Exception: System.NullReferenceException: Object reference not set to an instance of an object
at ES3Writer.Write[T] (System.String key, System.Object value) [0x0001b] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001B
at ES3.Save[T] (System.String key, T value, ES3Settings settings) [0x0001e] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_001E
at ES3.Save[T] (System.String key, T value, System.String filePath) [0x00009] in <9a53cee95b8c4093a7e8ca243c325c34>:IL_0009
at LethalModDataLib.Features.SaveLoadHandler.SaveData[T] (T data, System.String key, System.String fileName) [0x0004f] in <2e27e651b8b1406f89ea9ae6d53c5d1e>:IL_004F

fixed

repo: IAmBatby/LethalLevelLoader


text:
Track VCAP_APPLICATION.application_id

This information can be used to de-duplicate tracking records.

Delivered by #227


text:
Task name reference to AAC

Not the bigger issue on earth but the set of "xxx_reverseproxy_conf roles (add/delete/update) and also the role execute_isamcfg have "AAC" reference in their play name as can be reported in the following search result:
isam-ansible-roles\add_reverseproxy_conf\tasks\main.yml(2): - name: Configure Reverse Proxy for AAC (Updates)
isam-ansible-roles\delete_reverseproxy_conf\tasks\main.yml(1): - name: Configure Reverse Proxy for AAC (Deletes)
isam-ansible-roles\execute_isamcfg\tasks\rp_config_adds.yml(3): - name: Configure Reverse Proxy for AAC (Adds)
isam-ansible-roles\execute_isamcfg\tasks\rp_config_updates.yml(3): - name: Configure Reverse Proxy for AAC (Updates)
isam-ansible-roles\update_reverseproxy_conf\tasks\main.yml(2): - name: Configure Reverse Proxy for AAC (Updates)
Of course, these roles are very useful but not only for "AAC" reverse proxies but also for any reverse proxies that one can think off. When these roles run in playbooks their output can cause confusion if you know you have not integrated any AAC functionality yet.
I would volunteer to change those names to something more generic. I propose replacing “Configure Reverse Proxy for AAC (xxx)” with “Configure Reverse Proxy stanza entries (xxx)”
Any thought?

@sygilber - please make the changes and submit a pull request. That was an oversight on my part.
FYI: I have logic to execute isamcfg using a CLI call - need to find time to clean it up and upload. You will need v9.0.3 for it though.

Thanks

authors: ram-ibm sygilber

repo: IBM-Security/isam-ansible-roles