text:
invalid byte sequence in UTF-8 due to special character in step definition

Summary
Special characters in step definition like french accuentated characters prevent the HTML report from displaying the snippet:
invalid byte sequence in UTF-8 (ArgumentError)

-1# Couldn't get snippet for

Expected Behavior
If a language is supported in Gherkin .feature file, it should also be supported in step definition (.rb file).
Current Behavior
The character 'é' is not handled properly in HTML report code snippet and breaks the report (no embedded screenshot, broken report on jenkins)

Steps to Reproduce (for bugs)

cucumber --init

Create a test.feature file in the features folder
# language: fr
Fonctionnalité: Test

  Scénario: Issue with special character
    * é

Create a test.rb file in the features/step_definitions folder
Given /e/ do
	raise 'error'
end

Run cucumber
cucumber --format html --out report.html

Open report.html
Context & Motivation
I'm using fr language so any of my member of the team is able to understand the steps. A workaround for this would be really appreciated while a fix is not released.
Your Environment

Operating System and version: Debian 9.4.0 AMD64 gnome

cucumber --version
3.1.0

ruby --version
ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]

gem list
*** LOCAL GEMS ***

activemodel (5.1.5)
activerecord (5.1.5)
activesupport (5.1.5)
addressable (2.5.2)
anemone (0.7.2)
archive-zip (0.11.0)
arel (8.0.0)
ast (2.4.0)
autoparse (0.3.3)
backports (3.11.1)
bigdecimal (default: 1.3.4)
builder (3.2.3)
bundler (1.16.2)
bundler-unload (1.0.2)
childprocess (0.8.0)
chromedriver-helper (1.2.0)
cmath (default: 1.0.0)
concurrent-ruby (1.0.5)
csv (default: 1.0.0)
cucumber (3.1.0)
cucumber-core (3.1.0)
cucumber-expressions (5.0.13)
cucumber-tag_expressions (1.1.1)
cucumber-wire (0.0.1)
data_magic (1.2)
date (default: 1.0.0)
dbm (default: 1.0.0)
debase (0.2.2)
debase-ruby_core_source (0.10.3)
declarative (0.0.10)
declarative-option (0.1.0)
did_you_mean (1.2.0)
diff-lcs (1.3)
etc (default: 1.0.0)
executable-hooks (1.4.2)
extlib (0.9.16)
faker (1.8.7)
faraday (0.15.2)
fcntl (default: 1.0.0)
ffi (1.9.23)
fiddle (default: 1.0.0)
fileutils (default: 1.0.2)
gdbm (default: 2.0.0)
gem-wrappers (1.3.2)
gherkin (5.0.0)
google-api-client (0.21.2)
googleauth (0.6.2)
headless (2.3.1)
httpclient (2.8.3)
i18n (0.9.5)
image_size (2.0.0, 1.5.0)
io-console (default: 0.4.6)
io-like (0.3.0)
ipaddr (default: 1.2.0)
json (default: 2.1.0)
jwt (2.1.0)
launchy (2.4.3)
little-plugger (1.1.4)
log4r (1.1.10)
logging (2.2.2)
memoist (0.16.0)
mime-types (3.1)
mime-types-data (3.2016.0521)
mini_magick (4.8.0)
mini_portile2 (2.3.0)
minitest (5.11.3, 5.10.3)
multi_json (1.13.1)
multi_test (0.1.2)
multipart-post (2.0.0)
mysql2 (0.4.10)
net-telnet (0.1.1)
nokogiri (1.8.2)
openssl (default: 2.1.0)
os (0.9.6)
page-object (2.2.4)
page_navigation (0.10)
parallel (1.12.1)
parser (2.5.1.0)
power_assert (1.1.1)
powerpack (0.1.1)
psych (default: 3.0.2)
public_suffix (3.0.2)
rainbow (3.0.0)
rake (12.3.0)
rdoc (default: 6.0.1)
representable (3.0.4)
require_all (1.5.0)
retriable (3.1.1, 1.4.1)
robotex (1.0.0)
rspec (3.7.0)
rspec-core (3.7.1)
rspec-expectations (3.7.0)
rspec-mocks (3.7.0)
rspec-support (3.7.1)
rubocop (0.56.0)
ruby-debug-ide (0.6.1)
ruby-progressbar (1.9.0)
rubygems-bundler (1.4.4)
rubyzip (1.2.1)
rvm (1.11.3.9)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
selenium-webdriver (3.10.0)
signet (0.8.1)
stringio (default: 0.0.1)
strscan (default: 1.0.0)
syntax (1.2.2)
test-unit (3.2.7)
thor (0.20.0)
thread_safe (0.3.6)
tzinfo (1.2.5)
uber (0.1.0)
unicode-display_width (1.3.2)
watir (6.10.3)
watir-scroll (0.3.0)
webrick (default: 1.4.2)
wraith (4.2.1)
xmlrpc (0.3.0)
yml_reader (0.7)
zlib (default: 1.0.0)

I don't think so, Rubymine show UTF-8. Also checked with command line
file -i features/step_definitions/misc_steps.rb
features/step_definitions/misc_steps.rb: text/plain; charset=utf-8

Mea culpa, I see the same error. Something must be not specifying encoding when writing the HTML report...

Or maybe it's the code that tries to read the source code snippet and embed it in the HTML - that's more likely.

Hi again, did you get some time to look at this issue ?

No, did you? :-)

I won’t prioritise this, but feel free to submit a pull request with a fix.

Sure I could try.
I never looked at cucumber sources, any idea where I should start investigating ? Which files ?

I'd like to take care of this during this Hacktoberfest!

have same error message on MAC sporadically... one fail out of 3 same errors

when I run tests without html report following is the console output:

macOS 10.13.6 High Sierra
cucumber --version
3.1.2
ruby --version
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-darwin17]
gem list
*** LOCAL GEMS ***
activesupport (4.2.11.1)
addressable (2.6.0, 2.3.8)
atomos (0.1.3)
awesome_print (1.8.0)
backports (3.14.0, 3.13.0)
bigdecimal (default: 1.4.1)
builder (3.2.3)
bundler (2.0.1, default: 1.17.2)
bundler-unload (1.0.2)
calabash-android (0.9.9, 0.9.8)
calabash-cucumber (0.21.10)
CFPropertyList (3.0.0)
claide (1.0.2)
clipboard (1.3.3)
cmath (default: 1.0.0)
cocoapods (1.6.1)
cocoapods-core (1.6.1)
cocoapods-deintegrate (1.0.4)
cocoapods-downloader (1.2.2)
cocoapods-plugins (1.0.0)
cocoapods-search (1.0.0)
cocoapods-stats (1.1.0)
cocoapods-trunk (1.3.1)
cocoapods-try (1.1.0)
colored2 (3.1.2)
command_runner_ng (0.1.4)
concurrent-ruby (1.1.5)
csv (default: 3.0.2)
cucumber (3.1.2)
cucumber-core (3.2.1)
cucumber-expressions (6.0.1)
cucumber-tag_expressions (1.1.1)
cucumber-wire (0.0.1)
date (default: 1.0.0)
dbm (default: 1.0.0)
did_you_mean (1.3.0)
diff-lcs (1.3)
domain_name (0.5.20180417)
e2mmap (default: 0.1.0)
edn (1.1.1)
escape (0.0.4)
etc (default: 1.0.1)
executable-hooks (1.6.0)
fcntl (default: 1.0.0)
fiddle (default: 1.0.0)
fileutils (default: 1.1.0)
forwardable (default: 1.2.0)
fourflusher (2.2.0)
fuzzy_match (2.0.4)
gem-wrappers (1.4.0)
geocoder (1.5.1)
gh_inspector (1.1.3)
gherkin (5.1.0)
http-cookie (1.0.3)
httpclient (2.8.3)
i18n (0.9.5)
io-console (default: 0.4.7)
ipaddr (default: 1.2.2)
irb (default: 1.0.0)
json (2.2.0, default: 2.1.0, 1.8.6)
logger (default: 1.3.0)
luffa (2.1.0)
matrix (default: 0.1.0)
mime-types (2.99.3)
minitest (5.11.3)
molinillo (0.6.6)
multi_json (1.13.1)
multi_test (0.1.2)
mutex_m (default: 0.1.0)
nanaimo (0.2.6)
nap (1.1.0)
net-telnet (0.2.0)
netrc (0.11.0)
openssl (default: 2.1.2)
ostruct (default: 0.1.0)
power_assert (1.1.3)
prime (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (3.0.3)
rake (12.3.2)
rdoc (default: 6.1.0)
rest-client (2.0.2, 1.6.7)
retriable (2.0.2)
rexml (default: 3.1.9)
rouge (2.0.7)
rss (default: 0.2.7)
ruby-macho (1.4.0)
rubygems-bundler (1.4.5)
rubyzip (1.2.2)
run_loop (4.2.2)
rvm (1.11.3.9)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
shell (default: 0.7)
slowhandcuke (0.0.3)
stringio (default: 0.0.2)
strscan (default: 1.0.0)
sync (default: 0.5.0)
syntax (1.2.2)
test-unit (3.2.9)
thor (0.20.3)
thread_safe (0.3.6)
thwait (default: 0.1.0)
tracer (default: 0.1.0)
tzinfo (1.2.5)
unf (0.1.4)
unf_ext (0.0.7.6)
unirest (1.1.2, 1.0.8)
webrick (default: 1.4.2)
xamarin-test-cloud (2.2.0)
xcodeproj (1.9.0, 1.8.2)
xcpretty (0.3.0)
xmlrpc (0.3.0)
zlib (default: 1.0.0)

Ping @Oopla / @sarzamas - Have either of you got a git repo we can pull this down and run it on? Are you still getting the issue if you try the 4.0.0.rc1 version of cucumber.
@sjakobi - You mentioned hacking on this last year, did you get anywhere with it?

I am facing the same issue on my framework. I realised that a specific method called within the step file is causing this trouble. When I copied the code to the 'login_step.rb' file it worked fine. But I need to maintain my code refactored and this issue is not allowing me that.

Guys. I found the problem in my code. My features are written in Portuguese and the keyword "Then" is "Então". Because of this "ã" the problem happened. Maybe this can be useful to someone.

As mentioned before, if you can create a MVCE ideally in a small git repo we could pull down and run directly then that will be of greatest help.
Trying to piece together code in another language isn't something I'm willing to invest time into as it's likely I'll get it wrong.

Ping @luke-hill
I found the way how to reproduce error on any of your projects
....basically its written in bug title...
due to special character in step definition
These are necessary steps to be done for reproduction:

particular step description in <.feature> file and in <_steps.rb> file should be written in non-latin characters.

this step should encounter error of class Calabash::Android::WaitHelpers::WaitError
(this happens every time when trait method is returning error)

launch cucumber with flags:  -f html -o report.html

check error in report.html

replace all non-latin characters in step with latin characters - and no problem occurs

So you can trigger error easily on your environment
for example:

step in .feature file should look like the following:

Feature: Error with UTF-8.
Scenario: Generate error with UTF-8
  * Я вижу ошибку UTF-8

step in <_steps.rb> file should look like the following:

Given ("Я вижу ошибку UTF-8") do
  @page = page(SomePage).await(timeout:10)
end

this instuction will trigger trait method described in SomePage.rb file

SomePage.rb file should look like the following:

class SomePage < Calabash::ABase
  def trait
       "* id:'any'"
  end
end

then you start test as you do it regular
> bundle exec cucumber -t @smoke -p android_dev_usb -f pretty --expand -f html -o report.html

you check the console output and  report.html file

replacing  text 'Я вижу ошибку UTF-8' into 'I see error UTF-8' and backwards will trigger reproduction of error

Could you comment on what and where I should trace to figure out whether I am wrong or its a bug in thml formatter?
P.S.
if I don't use option -f html - all is fine with same steps in russian
thanks

Well - its definitely a bug in html formatter!
In continuation of my investigation I suddenly found the workaround:
The magic idea is to avoid making a screenshot of non-latin characters in code snippet!!!
as you may see a screenshot of code snippet is done -2 +2 lines of code prior and after the line causing an error

so I get the error if Russian characters are to be present in screenshot (are to be visible in any of these 5 lines)
once I added some placeholder line to avoid line with russian letters to appear in screenshot - error dissappeared
So in my example above you will get the UTF-8 error if you have
step in Error_steps.rb file looks like the following:
Given ("Я вижу ошибку UTF-8") do
  @page = page(SomePage).await(timeout:10)
end

But if you add two spare lines in between  - there will not be error occurrence
Given ("Я вижу ошибку UTF-8") do

  @page = page(SomePage).await(timeout:10)
end

Wow - that magic trick helps to continue write steps in russian and still receive html report in proper way
Never the less its a bug!!!

I'll re-open if you can quickly provide a reproducible scenario. As has been mentioned several times we're not going to piece together comments. There is way too much confusion.
For the avoidance of doubt this is what I would expect.

A repo I can clone
a single command to be ran cucumber or rspec for instance.
An observable failure

Also bear in mind that your PC's own config may affect this.
Now to put some perspective on things, the html formatter in ruby is essentially in maintenance mode, and only small bug fixes are going to come in, because we're rewriting (or looking to), all the formatters using protocol buffers. There is a reasonable amount of info on github about this, so if you want to continue looking into this, just know what the current "status-quo" is.

HI, @luke-hill
Thanks for your attention and beg pardon for the delay with my answer
I managed to make bug reproducible within small cucumber ruby project on GitHub
instructions to reproduce and workaround patch to steps_definitions and my environment are in readme.md
Check it for details here:
https://github.com/sarzamas/UTF8_issue_in_cucumber_report.html
Hope this is possible to be fixed soon
Thanх,
sarzamas

HI, @luke-hill
Thanks for your attention and beg pardon for the delay with my answer
I managed to make bug reproducible within small cucumber ruby project on GitHub
instructions to reproduce and workaround patch to steps_definitions and my environment are in readme.md
Check it for details here:
https://github.com/sarzamas/UTF8_issue_in_cucumber_report.html
Hope this is possible to be fixed soon
Thanх,
sarzamas

Having a quick look over the next couple of days.
Whilst I do, can you try and diet down that repo as much as possible. The fewer LOC there is, the easier it is for someone to narrow down the failure.
Some of those page objects have 50-60 lines (Do they need them?)
There are 5 pages? Surely we only need 1?
Ideally there should be between 20 and 40 lines of ruby code in total. There is close to 1000 at the moment, can you try remove absolutely everything that isn't needed.

I deleted the .pry_history file from C:/Users//.local/share/pry/

To anyone watching. This I said I'd triage but I haven't yet. Is there a git repo I can pull down to look at this
(Looking up, @sarzamas you mentioned you have one), could you re-link me to it.

authors: Oopla aslakhellesoy lflucasferreira luke-hill sarzamas sjakobi

repo: cucumber/cucumber-ruby


text:
Panic in FillPath about not the same runtime

All values in the example come from the same cue.Context. I'm assuming something like this should work and is likely related to the transition from Runtime to Context.
What version of CUE are you using (cue version)?
v0.4.0 Go API
What did you do?
go run fillpath.go
package main

import (
	"fmt"

	"cuelang.org/go/cue"
	"cuelang.org/go/cue/cuecontext"
)

const orig = `
a: b: "c"
`

const replace = `
a: b: "C"
`

func main() {
	ctx := cuecontext.New()

	o := ctx.CompileString(orig)
	r := ctx.CompileString(replace)

	f := replacer(r, o, ctx)
	fmt.Println(f)
}

func replacer(replace, orig cue.Value, ctx *cue.Context) cue.Value {
	var result cue.Value

	switch orig.IncompleteKind() {
	case cue.StructKind:
		result = ctx.CompileString("")
		s, _ := orig.Fields()
		for s.Next() {
			sel := s.Selector()
			path := cue.MakePath(sel)
			r := replace.LookupPath(path)
			if r.Exists() {
				tmp := replacer(s.Value(), r, ctx)
				result = result.FillPath(path, tmp)
			}
		}

	case cue.ListKind:
		vals := []cue.Value{}
		l, _ := orig.List()
		for l.Next() {
		}
		result = ctx.NewList(vals...)

	default:
		p := orig.Path()
		r := replace.LookupPath(p)
		if r.Exists() {
			result = r
		}
	}

	return result
}

What did you expect to see?
Not a panic
What did you see instead?
panic: values are not from the same runtime

goroutine 1 [running]:
cuelang.org/go/cue.Value.FillPath({0xc000021ea0, 0xc00022d8c0, 0x0}, {{0xc000207be8, 0xc00022d3b0, 0x0}}, {0x9006a0, 0xc00000ea08})
        /home/tony/go/pkg/mod/cuelang.org/go@v0.4.0/cue/types.go:1652 +0xdfe
main.replacer({0xc000021ea0, 0xc00022d290, 0x0}, {0xc000021ea0, 0xc00022d5f0, 0x0}, 0xc000266270)
        /home/tony/cue/repro/gocode/fillpath.go:46 +0x425
main.replacer({0xc000021ea0, 0xc00022d440, 0x0}, {0xc000021ea0, 0xc00022cfc0, 0x0}, 0x200000003)
        /home/tony/cue/repro/gocode/fillpath.go:45 +0x3c5
main.main()
        /home/tony/cue/repro/gocode/fillpath.go:29 +0xec
exit status 2

I'm not sure I entirely follow the code, but I think the innermost invocation of replacer() returns the zero value of cue.Value:
	tmp := replacer(s.Value(), r, ctx)  // returns zero Value
	result = result.FillPath(path, tmp) // result has context, but the context of tmp (zero Value) is different

This can happen in the default case of the switch statement in replacer():
	default:
		p := orig.Path()
		r := replace.LookupPath(p)
		if r.Exists() {  // false
			result = r
		}
	}

	return result // zero -> tmp above

So I'd say the panic is expected as not all values are from the same context.

@antong good catch, I did not expect that case to be false given the example data. What is happening is that Path is returning the full path from the root during recursion, rather than being scoped to the value passed in.
The following does remove the panic. There are a few ways actually.
	default:
		p := orig.Path()
		r := replace.LookupPath(p)
		if r.Exists() {  // false
			result = r
		} else {
			result = orig
		}
	}

	return result // zero -> tmp above

But does not give correct output for what I am trying to do. Need to process the orig.Path() to get the last component because there is not an easy way to get the current Value's label at present.

@verdverm - I think from your analysis above, this can therefore be closed because there is no bug. But please say if otherwise.

source: gharchive/issue

authors: antong myitcv verdverm

repo: cue-lang/cue

license_type: permissive

license_source: github-api


text:
cue: builds are slow

validate and build takes 4m24s on a tiny data set with 2 schemas.
Investigate why, fix if possible, parallelize if not.

Closed by e8b4be97eb71aff1ce529f4a69df0fdbd36172ce

authors: bketelsen rawkode

repo: cueblox/blox

license: MIT


text:
Add strings.NumRunes function

This PR provides an ExactRunes function that prevents you from needing to do things like this:
username: strings.MaxRunes(8) & strings.MinRunes(8)

@mpvl Okay, tests generated. If this isn't helpful, definitely feel free to close 😄 Totally up to you.

Note that this is still a viable option. Looking at various outstanding proposals, though, we may want to hold off on implementing this for now.
Firstly, it is somewhat tricky, as it interacts with various OpenAPI.
Also, we should consider the impact of allowing the number argument to be of any type, e.g. 1, <3, >=4, etc. This would make the Min and Max variants redundant, actually.

Hi @lucperkins - thanks for creating this PR.
CUE recently moved to a new organisation on GitHub: https://github.com/cue-lang/cue/issues/1078. Unfortunately we were unable to transfer this repository to the new organisation, and so pull requests will need to be re-created at https://github.com/cue-lang/cuelang.org.
Apologies for the inconvenience - we look forward to seeing you in CUE's new home.
Thank you again for contributing to the CUE project!

source: gharchive/pull-request

authors: lucperkins mpvl myitcv

repo: cuelang/cue


text:
视频源消失了。

版本、安装方式、系统

你在使用什么版本的
2.73

你通过什么方式安装
本站下载

你所使用的操作系统
WIN1O

描述问题： 所有的视频源消失了。如何找回呢？ 重新安装也没有用。

Linux ，同样的问题。
解决方案： $ rm -rf ~/.config/zy

Win10 同样的问题，求高手解答

Mac也是，根本上不了网

可以先在2.7.2保存Json，然后更新完再导入进去。

下一版本会支持从github网站读取源站文件,而且可以自行设置在线的json文件.

搜索框无效，搜什么剧，怎么搜，什么都搜不出来

authors: Lynn-7 YangtseSu cuiocean itnewgay jiehua1995 lovexiaov mumuyiren

repo: cuiocean/ZY-Player


text:
Add barcode to Inventory Item when 976$p is present

Shelf ready orders from vendors include the barcode in the MARC. Would need to be added to the Item created by the FOLIO Orders app. There is no MARC to Item mapping outside of Data Import field mapping profiles, so need to determine the best way to handle this:

A – ~set up a Data Import profile that this app executes after PO is created?~
B – or directly add the barcode with a followup PUT to /inventory/items?

We'll need a test file with the 976$p  field populated for this.

@fereira I have a test file. You want me to open a new PR with it and you can work from there or do you prefer I send it to you directly?
Also, please note that we decided to go with option B from above:

directly add the barcode with a followup PUT to /inventory/items

Can it be attached to this issue?
I don't quite understand the relationship between inventory/instance and inventory/items.  When executing a PUT to /inventory/items/{itemId} what should be used for the itemId.   Is the content basically the instanceResponse with the barcode added?

We can't attach .mrc files here. I sent it to you via Slack.
For Inventory, the hierarchy is:

Instances

Holdings

Items

The itemId is the UUID for the Item in question.
https://s3.amazonaws.com/foliodocs/api/mod-inventory/p/inventory.html#inventory_items__itemid__put
You definitely do not want to POST the response from a GET to /inventory/instances/{instanceID}.

I've implemented some code which follows this hierarchy am getting some odd results.  Using the instance ID I am calling holdings-storage/holdings?query=(instanceId==" + instanceId + ")" and that returns an array of "holdingsRecords" with one element.  Then using the id from the holdings record I'm calling "inventory/items?query=(holdingsRecordId=" + holdingsId + ")".  That returns an array of items, which, in the case for every record in the mrc file (there are eight records), it returns 9 items.   Iterating over those items shows that each of the items has a "title" field with the same contents.  That doesn't seem right, and I would suspect that putting the one barcode per record into all of the items isn't correct either.

It appears that every time that I run the app to process the shelfreadyAUX.mrc file, it is adding an inventory item.  Reading the items to get an item for a barcode to be added is probably correct but I'm not sure how to detect how many items there should be for a holdings record.

@fereira At the moment, FOLIO has very limited deduping capabilities at the SRS and Inventory level. So yes, when we use are the same test .mrc files for this app, it results in the creation of duplicate items.
For now, humans in Acquisitions will be doing their best to avoid submitting the same .mrc file to this app multiple times...

That kind of makes it hard to test.  I tried deleting purchase orders but that didn't have any impact on the holdings inventory.

Yeah, the FOLIO Orders app only creates Inventory pieces. @natalyapik should be able to help by providing a second shelf ready test file if you're interested.

authors: cappadona fereira

repo: cul-it/order-import-poc


text:
Prep for deployment

@georev All yours. Let me know what I missed.

[x] Push image to 092831676293.dkr.ecr.us-east-1.amazonaws.com/folio/order-import-poc:latest
[x] Copy .env to s3://ecs-env-files/folio/order-import-poc/.env

Sorry about that @georev. I think I have it sorted. Image remains unchanged, but I did push a new version of the .env to S3.

Sorry about that @georev. I think I have it sorted. Image remains unchanged, but I did push a new version of the .env to S3.

@fereira Robbie has redeployed if you want to take another look:
http://cul-folio-contrib.library.cornell.edu/order-import-poc/import

@fereira Robbie has redeployed if you want to take another look:
http://cul-folio-contrib.library.cornell.edu/order-import-poc/import

Thanks @fereira. We have an early draft of the deployment documentation available in Confluence.


text:
Understanding reduced_view

Saw some code in CuPy was leveraging reduced_view, but am still trying to wrap my head around it. The documentation unfortunately didn't help me much.

Returns a view of the array with minimum number of dimensions.

In particular the phrase "minimum number of dimensions" does not make much sense to me. Minimum in what sense? Subject to what?
Any help clarifying this would be much appreciated. 🙂

If I'm correct, "minimum" means "0 or 1". Usually, reduced view is 1-dimensional if there are not less than 1 element in the array. However, there could be 0-dimensional array which holds 0 element and only meta-information about shape is held on.

>>> import cupy
>>> a = cupy.empty((3, 3))
>>> a.reduced_view()
array([0., 0., 0., 0., 0., 0., 0., 0., 0.])
>>> b = cupy.empty((3, 0))
>>> b.reduced_view()
array([], dtype=float64)

Closing as the question of the author has been answered.

authors: fiarabbit jakirkham kmaehashi

repo: cupy/cupy


text:
Change the name from scatter_add to add.at for ease of migration with numpy

If scatter_at behaves identically to add.at, shouldn't we name both the same … for connivance in migration from numpy

Implemented in #7077.

authors: MohamedAliRashad kmaehashi


text:
Extend CuDNNError to have more debugging information

cuDNN API very frequently causes unexpected errors depending on specific conditions, which is often very difficult to debug.
This PR extends CuDNNError to include the low-level input configurations to CuPy cuDNN wrapper API, to make it possibly a bit easier to debug.
Before
CUDNN_STATUS_NOT_SUPPORTED

This PR
cuDNN Error: CUDNN_STATUS_NOT_SUPPORTED
  func: cudnnConvolutionForward
  x: shape=(4, 32, 16, 16, 16), dtype=float32, strides=(524288, 16384, 1024, 64, 4)
  W: shape=(32, 4, 1, 1, 1), dtype=float32, strides=(16, 4, 4, 4, 4)
  b: None
  y: shape=(4, 32, 10, 10, 10), dtype=float32, strides=(128000, 4000, 400, 40, 4)
  pad=(2, 2, 2), stride=(2, 2, 2), dilation=(5, 5, 5), groups=8
  auto_tune=False, tensor_core='auto'
  d_layout=0, w_layout=0

ImosCI, test this please.

Jenkins CI test (for commit 4d3b3af133e8713a4a11927e10200642dd8f98c0, target branch master) succeeded!

amd error appears unrelated.

authors: Crissman chainer-ci niboshi


text:
Rename cupy.padding submodule to cupy._padding

Rel #3420.
This PR renames cupy.padding submodule to cupy._padding to indicate it is CuPy's private submodule and not intended to be used by users.

@takagi Could you fix for Travis failure?

I've fixed it!

Jenkins, test this please.

Jenkins CI test (for commit f1ab4c2a2e5f14425859fc4fa9c5cecd8f068531, target branch master) succeeded!

LGTM.

authors: asi1024 chainer-ci takagi


text:
Rename submodules under cupy.linalg package

Rel #3420.
This PR renames the submodules under cupy.linalg to indicate that they are CuPy's private submodules and not intended to be used by users.

Probably blocked by #3234?

pfnCI, test this please.

Jenkins CI test (for commit 99b94c723aaf16e6551852b590ac8eb437b117e5, target branch master) failed with status FAILURE.

pfnCI, test this please.

pfnCI, test this please.

Jenkins CI test (for commit 51dd35fc56be3285bb393d3a6feac15550c3918b, target branch master) failed with status FAILURE.

pfnCI, test this please.

Jenkins CI test (for commit 95e63993d1b75b07122a54912210ac51b15a6e2b, target branch master) succeeded!

authors: chainer-ci emcastillo takagi


text:
Add path and readonly options to cupyx.optimizing.optimize

Implements #3574.

pfnCI, test this please.

Jenkins CI test (for commit 5bf15fc43811df1819d5ce06cdf850df265df57d, target branch master) succeeded!

Thanks!
pfnCI, test this please.

Jenkins CI test (for commit e253f3fe5f4526c41a750249987ca5a87c7830a6, target branch master) succeeded!

authors: chainer-ci kmaehashi


text:
Add parallel build feature

Use the CUPY_NUM_BUILD_JOBS to control the number of modules being built in parallel.
This PR enables the parallel compilation of thrust and cub, which are the modules that takes most of the time.
When building for all the compute archs, the master branch buiold time is 18 minutes, with this PR it becomes 9 minutes.
Moreover, if we limit to a single compute capability with something like CUPY_NVCC_GENERATE_CODE=arch=compute_62,code=sm_62 , we get a build time of 3 minutes, which is enough for the majority of the users.
Currently, the default is 4 processes because as long as we ensure that there are two threads, cub and thrust will be built independently.
Cub and Thrust are a single .cu file that is built for all the compute archs, and this is what takes the majority of the compile time. Unfortunately this process can't be parallelized unless we run nvcc ... --dryrun and run these commands in a different thread by ourselves.
Credit goes to this SO post:
https://stackoverflow.com/questions/11013851/speeding-up-build-process-with-distutils

BTW, pip install -e . --global-option="build_ext" --global-option="-j4" achieves the same 😄

Setting self.parallel = True in custom_build_ext instance seems enough?
https://github.com/pypa/setuptools/blob/400b71c9b932893ab97dd6368ae7bfb822064a87/setuptools/_distutils/command/build_ext.py#L454

This closes #1961

fixed to use self.parallel

Setting self.parallel = True in custom_build_ext instance seems enough?

So neat! Why isn't this undocumented?!

Jenkins, test this please

Jenkins CI test (for commit e72de6024284cdd1ee826846d126c0f2cb8dd704, target branch master) succeeded!

LGTM!

@emcastillo Could you add CUPY_NUM_BUILD_JOBS to docs?

Currently, instability is discovered in some environments. See #4344 for details.

authors: chainer-ci emcastillo kmaehashi leofang


text:
ROCm: disable PTDS

Follow-up of #4322.
PTDS does not work on HIP and leads to segfault when going out of scope (https://github.com/cupy/cupy/issues/4132#issuecomment-775734406). This PR disables it:

Raise if ptds=True is set in the constructor
Avoid creating the cupy.cuda.Stream.ptds singeleton
Skip PTDS tests

Jenkins, test this please

Jenkins CI test (for commit a47e624d12a03564e2bdf837d209c5a64fc35631, target branch master) succeeded!

authors: chainer-ci leofang


text:
Consider parsing Cursorless commands extension side

I wonder if we should just send a sequence of terms to cursorless and then have a full-blown parser there (tree-sitter?). Then we could define precedence rules, etc. Ie "head inside every funk air past bat" would be split into a sequence of seven tokens: [head, inside, every, funk, air, past, bat], and then cursorless VSCode would figure out how to parse it.

See also #193
See also #789

I'm worried that this means that we need to implement the same grammar logic both talon and extension side.

Def worth a conversation.  We'd basically make Talon side as simple as possible.  I'm just not sure it's possible to do what we want with Talon's simple grammar model, ie precedence rules etc

authors: AndreasArvidsson pokey

repo: cursorless-dev/cursorless


text:
Add language overrides for delimiter maps

This is me breaking out some functionality that was part of PR #1911 since that PR has lots of work to be done and it may still benefit other languages that get in sooner. I've applied the suggested changes from pokey in that PR's code review.
I've also added a placeholder map for lua which uses [[ ]] for multiline support as noted in PR #1962, however I realized as I went to add it that it's not as simple as the way I did it for nix because I can't just reuse the existing singleQuote entry since lua actually uses single quotes. So this may actually be a bit harder, and reminds me of this discussion. So before I go randomly hacking stuff I'm curious what you think the best approach here is.
These are the comments I had left in the Nix PR about these changes:

I called returning delimiterToText as getSimpleDelimiterMap to kind of mirror complexDelimiterMap.

I'm not sure is if you still want a delimiterMap.ts standalone file, and then a getSimpleDelimiterMap.ts only for that function. Now delimiterToText isn't referenced anywhere else, so seemed maybe okay to keep them together.

I also thought about adding getComplexDelimiterMap(), but atm because complexDelimiterMap only ever uses the keys from the delimiterToText it won't change even if the language has different values, so seemed unnecessary. These are all things I could guess at your preferences, but may as well ask instead.

I also noticed leftToRightMap in that file isn't actually used anywhere so can be deleted I think, but also not sure about doing that as part of a totally unrelated code change, to keep commits clean.

Checklist

[_] I have added tests
[_] I have updated the docs and cheatsheet
[x] I have not broken the cheatsheet

I would be tempted to introduce a new delimiter type for [[, and then make complex delimiter list customisable so you can include it
WDYT @AndreasArvidsson

See https://github.com/cursorless-dev/cursorless/pull/1911#discussion_r1386549043 for a comment about some code from this PR.

I would be tempted to introduce a new delimiter type for [[, and then make complex delimiter list customisable so you can include it
WDYT @AndreasArvidsson

I think I need a bit more info on your suggestion

I would be tempted to introduce a new delimiter type for [[, and then make complex delimiter list customisable so you can include it
WDYT @AndreasArvidsson

I think I need a bit more info on your suggestion

the problem is that in lua, [[ and ]] can be used as delimiters for a quote. So I'm proposing to add those as a new delimiter type doubleSquareBrackets. Then we'd need "string" to include those. "string" is defined as a complex delimiter string: ["singleQuotes", "doubleQuotes", "backtickQuotes"],, so we'd add doubleSquareBrackets to that list, but only for Lua.
One thing we'd need to think about is that that would cause [[foo]] to be viewed as a pair for our text-based matchers, and it wouldn't be considered a "square", which is probably not great. So we'd prob need a way for that pair not to be active outside of Lua

I would be tempted to introduce a new delimiter type for [[, and then make complex delimiter list customisable so you can include it
WDYT @AndreasArvidsson

I think I need a bit more info on your suggestion

the problem is that in lua, [[ and ]] can be used as delimiters for a quote. So I'm proposing to add those as a new delimiter type doubleSquareBrackets. Then we'd need "string" to include those. "string" is defined as a complex delimiter string: ["singleQuotes", "doubleQuotes", "backtickQuotes"],, so we'd add doubleSquareBrackets to that list, but only for Lua.
One thing we'd need to think about is that that would cause [[foo]] to be viewed as a pair for our text-based matchers, and it wouldn't be considered a "square", which is probably not great. So we'd prob need a way for that pair not to be active outside of Lua

I agree, but we still need to make square actually work of course.

Ok @fidgetingbits I think we're in agreement. Lmk if you need help figuring out how to put the above to practice

Ok @fidgetingbits I think we're in agreement. Lmk if you need help figuring out how to put the above to practice

Ya some guidance would be good, as much as I'd love to try to figure it out on my own. My typescript experience consists of what's in this PR. I suspect it would take me quote long time work out what to do.

Ok I'll take this over when I get a minute

Looks reasonable to me

Ok @fidgetingbits I made some tweaks. Given some more recent discussions (#1812 (comment)), I don't think this is the long-term solution, but:
* It's not too much code,

* It's not too bad an abstraction, and

* I need it for a tree-sitter upgrade I'm doing to handle strings in Python

Let me know if you're happy with my changes (see e82832f)

Added lua test looks good to me. Your changes look good too, reads cleaner.

authors: AndreasArvidsson fidgetingbits pokey


text:
Scroll to adjust opacity?

Hey, thanks for the wonderful app, it's really useful. Let me know if there's a better place than Github Sponsor to send you coffee money.
I use an app called Floating, and it has a cool feature where I can scroll up or down to adjust opacity. If there was something similar in Pipper I think it would be really cool. Still a great app nonetheless. As well, when I adjust opacity on Pipper it looks a bit strange compared to Floating. I'll add some images so you get what I mean.
Thanks again for the work.

Left is Pipper, right is Floating. It seems it has something to do with the white background of YouTube? Not sure.

Yeah, seems to be due to the white background, here's another image after I changed the YouTube setting to dark.

Yes, can confirm this.
I'm thinking about allowing users to inject js and css, we can probably use that to remove backgrounds on "known websites" (such as YouTube).
Food for thought!

authors: curzel-it javentous

repo: curzel-it/pipper


text:
Feature Discussion: automation switches

One nice feature about native Home Assistant automations is they all come with an "automation.whatever" entity that can be turned off or triggered manually (usually for testing). Pyscript doesn't have this so turning off an automation either means commenting it out or adding @state_active('input_boolean.my_automation == "on"') to the methods and then creating the input_boolean.
Is there any interest in letting pyscript do this for us, perhaps even on an "as requested" basis?
Example code:
@state_trigger('binary_sensor.living_motion == "on"')
@automation_switch('common_area_lights')
def living_on():
  light.living.turn_on()

@state_trigger('binary_sensor.kitchen_motion == "on"')
@automation_switch('common_area_lights')
def kitchen_on():
  light.kitchen.turn_on()

In this example, two methods use the same "automation_switch". Pyscript would create the switch on the first invocation, and realize it was already created on the second invocation and do nothing. If these methods are deleted, pyscript should realize there are no more methods that use this automation_switch and delete it from Home Assistant.
Additionally, each of these functions would have the equivalent of @state_active('switch.common_area_lights == "on"') attached to them. If the switch is off, they don't trigger. Even better would be to give the function the same treatment as a "filename comment out" so that the "shutdown" functions happen when turned off, and the "startup" functions happen when turned on.
Creating the switch in Home Assistant is the hardest part of the puzzle, likely, only because pyscript does nothing like it so far. We can't just state.set the switch entity because 1) a switch with that entity id might already exist and 2) that doesn't create all the home assistant hooks to listen for turn_off and turn_on service calls. So we have to use the switch platform to create the switch.
If we don't want to get into all the platform entity creation stuff, another option would be to provide (pyscript.automation_turn_on and pyscript.automation_turn_off services that take automation: common_area_lights as data (with possibly better names than I've come up with). This would make it really easy to turn on/off a pyscript automation from code/native automations, but a little more difficult to do it from the UI since users would have to use a button card (or something like it) to handle this instead of just adding the switch entity to a row. It's less nice, but MUCH easier since platform entities can be a bit cumbersome.
Thoughts?
This feature discussion brought to you by me playing "flashlight tag" in the dark with my kids the other day and realizing this was impossible since pyscript kept turning all my lights on. So I had to comment everything out, enjoy the game, and then turn everything back on. I would have still had to manually turn something off and then on again with the above in place, but I could have written a single pyscript to handle the on/off functionality (to be used then and any time again in the future) instead of having to find and edit the "state_active" bits of every applicable automation.
I really like the idea of each function classifying itself with the decorator. This gets really fancy when I can supply multiple names:
@automation_switch('common_area_lights', 'living_lights', 'lights')

Thus providing me with three switches, one to turn off all common_area light automations, one to turn off all living light automations, and one to turn off all light automations in the whole house. They end up working a bit like "tags" for the functions that the function authors can use as they see fit.

Is there any interest in letting pyscript do this for us, perhaps even on an "as requested" basis?

I believe very much there is.
I will work around that in the HA frontend by setting an extra state that I set or unset in the UI (in a specific card), and test in my automation its state before going forward. Having that built-in would be awesome.

@wsw70 teens seem to enjoy it too. very young kids will need a buddy. basically.... turn off ALL the lights. Then play "hide and go seek" (except the hiding is different since you can hide in plain sight since it's dark). The "seeker" gets the only flashlight. If you get lit up by the flashlight, you're out.

@dlashua oh that's great! I will actually use it as a game for my kids and their friends (probably somewhere outside) after we are free from COVID (in France we have a curfew starting at 20:00 and nobody meets noone anyway). Thanks for the idea!

Is there a place where the current, loaded automations from pyscript are registered? A place that could be queried from within pyscript?
I just found the list of the files the automations are ran from (pyscript.list_global_ctx())

It depends on exactly what pieces you're looking for. These bits aren't designed for user interaction (i.e. you can break things if you mess around with it) but... you can get a list of all the names of all registered triggers, for instance, like this:
import custom_components.pyscript.global_ctx as global_ctx

@time_trigger('startup')
def test_startup():
    for gctx_name in global_ctx.GlobalContextMgr.contexts:
        gctx_obj = global_ctx.GlobalContextMgr.contexts[gctx_name]
        for one_trigger in gctx_obj.triggers:
            trig_obj = one_trigger.trigger
            if trig_obj is not None:
                log.info(trig_obj.name)

I will work around that in the HA frontend by setting an extra state that I set or unset in the UI (in a specific card),

This ended up being more difficult than expected (for now :)). While I can manage states for automations (mostly thanks to @dlashua code) and display them nicely with https://github.com/thomasloven/lovelace-auto-entities in a dynamic way, this is just a display of their status.

Adding a switch at the place where on is apparently requires https://www.home-assistant.io/integrations/switch.template/, which in turn probably means that the templates need to be hardcoded in a file (which is directly in conflict with the dynamic nature).
I will think about that a bit more.

Instead of creating a switch, I would suggest we create something similar to either

the AutomationEntity: https://github.com/home-assistant/core/blob/dev/homeassistant/components/automation/init.py#L212
the ScriptEntity: https://github.com/home-assistant/core/blob/dev/homeassistant/components/script/init.py#L247

This will give us the flexibility to define services that make sense for pyscript and we can add states and attributes that can provide extra information, such as what is requested in https://github.com/custom-components/pyscript/issues/130

@raman325 not a bad idea at all. Thanks!

I'd be happy to take a crack at this unless you already have thoughts on how to do it. I stumbled on this because of a recent need for such an entity 🙂

@raman325 Take a look at PR #129 because it's part of the way there. Then, add comments/codes to that?

@raman325 Take a look at PR #129 because it's part of the way there. Then, add comments/codes to that?

#129 is for user defined entities whereas I understand this to be about providing entities automatically for each pyscript service/automation. If you agree, then where do you see the intersection? I imagine that implement this feature, we'd have to create entities while loading contexts from files.

#129 is for user defined entities whereas I understand this to be about providing entities automatically for each pyscript service/automation. If you agree, then where do you see the intersection? I imagine that implement this feature, we'd have to create entities while loading contexts from files.

This is about handling them somewhat automatically, though there would still be a decorator to request the behavior, I imagine.
However, they both require the entity platforms to be handled on the Home Assistant side. So, while they are not the same thing, the code behind them is similar and related.

This is about handling them somewhat automatically, though there would still be a decorator to request the behavior, I imagine.
However, they both require the entity platforms to be handled on the Home Assistant side. So, while they are not the same thing, the code behind them is similar and related.

The way I envisioned this, there would be an entity for every automation, service, or app that pyscript is aware of (I see this use case as very similar to the automation and script platforms). I don't see the harm in providing this by default, so no decorator would be required. Is there a problem you are trying to solve in using a decorator that I am missing?
The work you are doing in #129 is related in that in both cases, we are trying to create entities, but the difference is that you are using existing platforms in that PR (switches, sensors, etc.), whereas here we'd be defining our own EntityComponent. I don't think the code will end up being all that related, but happy to be proven wrong. Perhaps since you have made progress on #129 already, we should wait to see what comes out of that and then we can decide how to address this one.

The way I envisioned this, there would be an entity for every automation, service, or app that pyscript is aware of (I see this use case as very similar to the automation and script platforms). I don't see the harm in providing this by default, so no decorator would be required. Is there a problem you are trying to solve in using a decorator that I am missing?
The work you are doing in #129 is related in that in both cases, we are trying to create entities, but the difference is that you are using existing platforms in that PR (switches, sensors, etc.), whereas here we'd be defining our own EntityComponent. I don't think the code will end up being all that related, but happy to be proven wrong. Perhaps since you have made progress on #129 already, we should wait to see what comes out of that and then we can decide how to address this one.

I don't have a problem with having every Pyscript Trigger create an "automation" automatically by default. Using the decorator allowed it to have more features. For example, with Native Home Assistant Automations, many people have 2 automations for a motion activated light: one to turn it on when there is motion, and another to turn it off when the motion stops. Yes, this can be done in one automation and that is true in Pyscript as well, but the point is, there are two "automations" that are working toward the same goal. The advantage to using the decorator is that if I use the same "id" on the decorator for both triggers, then the two triggers/automations can be linked together and controlled with the same Toggle. So, if I want to turn off the motion activated lights in my living room, I only hav to turn off ONE thing... instead of two.
So that's an advantage. But, as you mentioned, the disadvantage is now the user has to ask for the functionality instead of it being automatiic. I'm sure there's a way to do both.
However, more to the point, I tried, just now, to create a proof of concept for this, but it seems Home Assistant doesn't allow the "Automation" platform to be extended the way "Switch" can be. It seems this is also true for "Script". So, if we make it a "Switch", I can still show this proof of concept. But, If we want it to be an Automation (which, I agree, is better) then it is indeed different than #129 and you should take a crack at it. I'd like to see how that's done.

I don't have a problem with having every Pyscript Trigger create an "automation" automatically by default. Using the decorator allowed it to have more features. For example, with Native Home Assistant Automations, many people have 2 automations for a motion activated light: one to turn it on when there is motion, and another to turn it off when the motion stops. Yes, this can be done in one automation and that is true in Pyscript as well, but the point is, there are two "automations" that are working toward the same goal. The advantage to using the decorator is that if I use the same "id" on the decorator for both triggers, then the two triggers/automations can be linked together and controlled with the same Toggle. So, if I want to turn off the motion activated lights in my living room, I only hav to turn off ONE thing... instead of two.
So that's an advantage. But, as you mentioned, the disadvantage is now the user has to ask for the functionality instead of it being automatiic. I'm sure there's a way to do both.
However, more to the point, I tried, just now, to create a proof of concept for this, but it seems Home Assistant doesn't allow the "Automation" platform to be extended the way "Switch" can be. It seems this is also true for "Script". So, if we make it a "Switch", I can still show this proof of concept. But, If we want it to be an Automation (which, I agree, is better) then it is indeed different than #129 and you should take a crack at it. I'd like to see how that's done.

But, as you mentioned, the disadvantage is now the user has to ask for the functionality instead of it being automatic. I'm sure there's a way to do both.

As a side comment: pyscript means code anyway. It is not like the HA automations which you build through a click-next procedure.
So having a tighter control over what is exposed is in my opinion a plus. This is only one extra line that works fine with the stack of @decorators one will have anyway.

But, as you mentioned, the disadvantage is now the user has to ask for the functionality instead of it being automatic. I'm sure there's a way to do both.

As a side comment: pyscript means code anyway. It is not like the HA automations which you build through a click-next procedure.
So having a tighter control over what is exposed is in my opinion a plus. This is only one extra line that works fine with the stack of @decorators one will have anyway.

Two things:

I would not recommend we extend the automation platform, because among other things, that will be confusing, and pyscript has different functionality than the automation platform does. Instead, we'd create the pyscript platform and define entities that represent the various pyscript automations, services, and apps there.
Your use case of having two automations that are more or less tied together is a good one. Instead of solving this with decorators, I would propose that we solve this by creating entities at the file/app level and entities at the individual service/automation level. So, in my case, I have a file called climate.py which holds automations related to the climate in my system (let's call them auto1 and auto2). We could essentially create three entities, one called pyscript.climate, and the other two called pyscript.climate.auto1 and pyscript.climate.auto2 (I have to check whether we can namespace this way or if we have to use underscores). This would allow users to turn on and off auto1 and auto2 individually, but they can also turn off the whole file. The file becomes the ID and we no longer need a decorator.

Two things:

I would not recommend we extend the automation platform, because among other things, that will be confusing, and pyscript has different functionality than the automation platform does. Instead, we'd create the pyscript platform and define entities that represent the various pyscript automations, services, and apps there.
Your use case of having two automations that are more or less tied together is a good one. Instead of solving this with decorators, I would propose that we solve this by creating entities at the file/app level and entities at the individual service/automation level. So, in my case, I have a file called climate.py which holds automations related to the climate in my system (let's call them auto1 and auto2). We could essentially create three entities, one called pyscript.climate, and the other two called pyscript.climate.auto1 and pyscript.climate.auto2 (I have to check whether we can namespace this way or if we have to use underscores). This would allow users to turn on and off auto1 and auto2 individually, but they can also turn off the whole file. The file becomes the ID and we no longer need a decorator.

Two things:

I would not recommend we extend the automation platform, because among other things, that will be confusing, and pyscript has different functionality than the automation platform does. Instead, we'd create the pyscript platform and define entities that represent the various pyscript automations, services, and apps there.
Your use case of having two automations that are more or less tied together is a good one. Instead of solving this with decorators, I would propose that we solve this by creating entities at the file/app level and entities at the individual service/automation level. So, in my case, I have a file called climate.py which holds automations related to the climate in my system (let's call them auto1 and auto2). We could essentially create three entities, one called pyscript.climate, and the other two called pyscript.climate.auto1 and pyscript.climate.auto2 (I have to check whether we can namespace this way or if we have to use underscores). This would allow users to turn on and off auto1 and auto2 individually, but they can also turn off the whole file. The file becomes the ID and we no longer need a decorator.

Two things:

I would not recommend we extend the automation platform, because among other things, that will be confusing, and pyscript has different functionality than the automation platform does. Instead, we'd create the pyscript platform and define entities that represent the various pyscript automations, services, and apps there.
Your use case of having two automations that are more or less tied together is a good one. Instead of solving this with decorators, I would propose that we solve this by creating entities at the file/app level and entities at the individual service/automation level. So, in my case, I have a file called climate.py which holds automations related to the climate in my system (let's call them auto1 and auto2). We could essentially create three entities, one called pyscript.climate, and the other two called pyscript.climate.auto1 and pyscript.climate.auto2 (I have to check whether we can namespace this way or if we have to use underscores). This would allow users to turn on and off auto1 and auto2 individually, but they can also turn off the whole file. The file becomes the ID and we no longer need a decorator.

We already create and use the pyscript platform. Unfortunately, I don't believe it's possible for a custom_component to indicate UI display for entities. So, I don't think we can get a toggle switch to show up on a pyscript.whatever entity. If this isn't true, I'd love to see any documentation you can link to to show how it's done. But, even then, I don't think it's possible to display a different UI for different entities in the same domain, and since pyscript is already being used for persistent storage needs and simple sensor needs.
Some "automations" are used more than once. Like this:
rt = []
def light_on_30_secs(light_entity):
  @state_trigger('{light_entity} == "on"')
  def turn_off():
    task.sleep(30)
    light.turn_off(entity_id=light_entity)

  rt.append(turn_off)

light_on_30_seconds('light.living')
light_on_30_seconds('light.kitchen')

In the above case, turning off the "filename.light_on_30_secs.turn_off` automation would turn off both the living and the kitchen, when that may not be what the user wants.
On top of that, with switches at the file, and function level, we have to deal with things like... what happens if I turn off the file level automation, but then turn on a function automation in that file.
And, though it doesn't effect this since we can just underscores, but, no, Home Assistant entity_ids can only have one "." between the domain and entity. No other "." are allowed.
I like the idea of being able to turn off an entire file/app. And we already have this with the "commented out filename" feature craig added, so the means to do so is already there. But I still think manually selected automation names that can be attached to more than one trigger function will provide the most flexibility and the least amount of "well, that's not what I expected to happen".

We already create and use the pyscript platform. Unfortunately, I don't believe it's possible for a custom_component to indicate UI display for entities. So, I don't think we can get a toggle switch to show up on a pyscript.whatever entity. If this isn't true, I'd love to see any documentation you can link to to show how it's done. But, even then, I don't think it's possible to display a different UI for different entities in the same domain, and since pyscript is already being used for persistent storage needs and simple sensor needs.
Some "automations" are used more than once. Like this:
rt = []
def light_on_30_secs(light_entity):
  @state_trigger('{light_entity} == "on"')
  def turn_off():
    task.sleep(30)
    light.turn_off(entity_id=light_entity)

  rt.append(turn_off)

light_on_30_seconds('light.living')
light_on_30_seconds('light.kitchen')

In the above case, turning off the "filename.light_on_30_secs.turn_off` automation would turn off both the living and the kitchen, when that may not be what the user wants.
On top of that, with switches at the file, and function level, we have to deal with things like... what happens if I turn off the file level automation, but then turn on a function automation in that file.
And, though it doesn't effect this since we can just underscores, but, no, Home Assistant entity_ids can only have one "." between the domain and entity. No other "." are allowed.
I like the idea of being able to turn off an entire file/app. And we already have this with the "commented out filename" feature craig added, so the means to do so is already there. But I still think manually selected automation names that can be attached to more than one trigger function will provide the most flexibility and the least amount of "well, that's not what I expected to happen".

What entities already exist in the pyscript domain? I haven't seen them personally.
Assuming ^ is true I don't see any breaking change here. Existing services can still exist within the pyscript domain but we'd introduce new services for the entities that represent the function and its action (turn on, turn off, run, etc). Your example can be resolved by providing two entities, one for each entity ID which we should be able to detect within pyscript

What entities already exist in the pyscript domain? I haven't seen them personally.
Assuming ^ is true I don't see any breaking change here. Existing services can still exist within the pyscript domain but we'd introduce new services for the entities that represent the function and its action (turn on, turn off, run, etc). Your example can be resolved by providing two entities, one for each entity ID which we should be able to detect within pyscript

The entities that exist are user created. We have state.persist for persisting the data across a restart.
Adding services isn't the problem. The problem is, (and I'd love to be wrong about this), is that making an entity called pyscript.my_automation and making a service called pyscript.turn_on that takes an entity_id of pyscript.my_automation doesn't give that entity a Toggle Switch in the UI. To my knowledge, there is no way to give an entity of pyscript.my_automation a Toggle Switch in the UI without a custom card. Installing a companion card to go along with pyscript isn't a big deal except, that card would have to be indicated manually every time it needs to be used. It wouldn't show up in "more info" or in "dev tools" or on the default dashboard if you've not customized lovelace in anyway.
I understand what you're saying. And I agree, if it is possible to do it that way, that would be better. I just don't think that it's possible (without modifications to Home Assistant Core). So, if possible, send a PR or link to another custom_component that shows a toggle UI on a custom domain entity.

The entities that exist are user created. We have state.persist for persisting the data across a restart.
Adding services isn't the problem. The problem is, (and I'd love to be wrong about this), is that making an entity called pyscript.my_automation and making a service called pyscript.turn_on that takes an entity_id of pyscript.my_automation doesn't give that entity a Toggle Switch in the UI. To my knowledge, there is no way to give an entity of pyscript.my_automation a Toggle Switch in the UI without a custom card. Installing a companion card to go along with pyscript isn't a big deal except, that card would have to be indicated manually every time it needs to be used. It wouldn't show up in "more info" or in "dev tools" or on the default dashboard if you've not customized lovelace in anyway.
I understand what you're saying. And I agree, if it is possible to do it that way, that would be better. I just don't think that it's possible (without modifications to Home Assistant Core). So, if possible, send a PR or link to another custom_component that shows a toggle UI on a custom domain entity.

you are right - I consulted with the FE experts in #devs_frontend and from a UI perspective we'd want to use existing domains. So I retract my previous statement about not using existing domains. Maybe we should try using automation for automations/files/apps, and script for services. I have to dig in further to these entity types to see if they make sense for our use cases.

you are right - I consulted with the FE experts in #devs_frontend and from a UI perspective we'd want to use existing domains. So I retract my previous statement about not using existing domains. Maybe we should try using automation for automations/files/apps, and script for services. I have to dig in further to these entity types to see if they make sense for our use cases.

Please do. From what I can see, those platforms cannot be extended by custom components. But, it might just be that it needs to be done in a different way that I'm not familiar with.

Please do. From what I can see, those platforms cannot be extended by custom components. But, it might just be that it needs to be done in a different way that I'm not familiar with.

Alright so I've made progress on this. Thus far I have created automation entities for automations/files/apps - I don't know that we need entities for services unless we have a way of tracking when an automation/service is running which I haven't tackled yet. I need to do a little more work to make sure the turning on and off actions of the files vs automations work logically with each other.
https://github.com/custom-components/pyscript/compare/master...raman325:automation_entities
This is all pretty messy, it would be easier if it was easier to track the internals of each automation/file as it gets parsed in to pyscript. But it's progress!

Alright so I've made progress on this. Thus far I have created automation entities for automations/files/apps - I don't know that we need entities for services unless we have a way of tracking when an automation/service is running which I haven't tackled yet. I need to do a little more work to make sure the turning on and off actions of the files vs automations work logically with each other.
https://github.com/custom-components/pyscript/compare/master...raman325:automation_entities
This is all pretty messy, it would be easier if it was easier to track the internals of each automation/file as it gets parsed in to pyscript. But it's progress!

So THAT'S how you make Entities in the automation Domain. Very interesting.

So THAT'S how you make Entities in the automation Domain. Very interesting.

A slightly different take would be to use MQTT autodiscovery to create a switch for the automation.

A slightly different take would be to use MQTT autodiscovery to create a switch for the automation.

@cazador481 this would work. However, it would also add the requirement that the user run MQTT, add a bit of delay to the process, and be, in general, more complex. Since we have access to native Home Assistant entities inside a custom_component, we may as well use them. This is a nice work around for apps like AppDaemon and NodeRed, though, since they are not embedded in Home Assistant like Pyscript is.

@cazador481 this would work. However, it would also add the requirement that the user run MQTT, add a bit of delay to the process, and be, in general, more complex. Since we have access to native Home Assistant entities inside a custom_component, we may as well use them. This is a nice work around for apps like AppDaemon and NodeRed, though, since they are not embedded in Home Assistant like Pyscript is.

repo: custom-components/pyscript


text:
Scalar parameter/variable support

Code generation seems to fail for scalar parameters and variables, as opposed to array parameters/variables. For example, suppose we have something like
w_a = cp.Parameter(1,name='w_a')
a = cp.Variable(1,name='a')
objective = cp.Minimize(w*a)

This will generate code with errors in both the example and cpg_solve, since the code seems to assume each parameter and variable is an array. The cpg_example will call cpg_update_w(0, w_value), but the cpg_update_w function only accepts one argument. Similarly, cpg_solve->cpg_retrieve_prim() will attempt to assign CPG_Prim.a = ecos_workspace[k], which will throw an error: incompatible types when assigning to type ‘c_float *’ {aka ‘double *’} from type ‘pfloat’ {aka ‘double’}
Is there a simple way to deal with this (e.g. some scalar=True flag)? A workaround might be to make all scalars arrays of length 2 and only use the first index, but obviously that's not desirable.
Edit:
A better reproduction is to set n=1 in the network.ipynb example. This will give the same kind of error.

@maxschaller

Good catch! To make it work immediately, simply omit the first argument when constructing your variable and write a = cp.Variable(name='a').
cvxpygen has been tested for variables with squeezed representation. When constructing a cvxpy variable by passing 1 as first positional argument, it will be of shape (1,), a vector of length 1 and not a scalar (the squeezed representation).
I will make sure that non-squeezed representations of variables work as well. Non-squeezed representations of parameters should already work.

Awesome, easy fix, confirmed it works on my application.
Thank you!

authors: SteveDiamond ldstrohl maxschaller

repo: cvxgrp/cvxpygen


text:
Build with clang resolves missing libasan link error

My first build was not using clang but my system's default of gcc, and linking failed on missing libasan. I get the impression that the DuckDB developers default is clang. Switching to clang resolved the linking error, and also gives fewer (only one remaining actually) warnings about type mismatches when building the codebase. (I only had to add sqlite-devel to my Fedora system.)
I think it would be good to update the README to instruct people to use clang (unless they know what they are doing). Perhaps the cmake configuration should check for libasan and sqlite-devel to be installed.

Welcome @arjenpdevries , can you give the console output. Also might be worth implementing #94 as a response.

Both GCC and Clang should work fine. Indeed, defaulting to an optimized build should solve this. Neither libasan nor sqlite-devel are required to build the core of DuckDB. Libasan is used for debugging purposes (address sanitizer) and SQLite is used only for the tests. Then if people want to debug and run the tests they can install these libraries.

Console output:
build$ cmake -DCMAKE_BUILD_TYPE=Debug --clean-first -j 4 ..
build$ make > make.log 2> make.err
[..]
/usr/bin/ld: cannot find /usr/lib64/libasan.so.5.0.0 inside /
collect2: error: ld returned 1 exit status
make[2]: *** [src/CMakeFiles/duckdb.dir/build.make:697: src/libduckdb.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:173: src/CMakeFiles/duckdb.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

Attached complete output: make.log (uninteresting) and make.err (various warnings given by gcc that could be addressed).

PS: I just reported it because it is a "first build experience" and I assumed you want that to be as accessible as possible. I understand that maybe I should have chosen Release instead of Debug and then it might not have happened :smile:. Perhaps suggesting Clang as default is not a bad idea though, especially not if that would be your own default (?).

Indeed we absolutely want it to work out of the box. However, it should succeed with any C++11 compatible compiler (and we do also run tests on both clang, gcc and msvc).
In this case it didn't fail because of the compiler, but rather because of the missing dependency (address sanitizer). I think the proper solution therefore would be to disable the address sanitizer by default (i.e. DEBUG compiles only with -O0 -g), we can then make a separate build configuration DebugWithSanitizer that also compiles with the sanitizer.
And indeed those warnings should be fixed, thanks for the list!

We now use release build by default

authors: Mytherin arjenpdevries hannesmuehleisen

repo: cwida/duckdb


text:
[Python] Parser error when query starts with spaces

The following query from a python environment crashes due to the leading spaces not being truncated:
make python_release
import duckdb

def main()
  con = duckdb.connect()
  con.execute(f"""

  -CREATE PROPERTY GRAPH snb VERTEX TABLES (person LABEL person) EDGE TABLES (person_knows_person SOURCE KEY
  (personid) REFERENCES person (personid) DESTINATION KEY (friendid) REFERENCES person (personid) LABEL knows)
  """)
if __name__ == "__main__":
  main()

Unable to reproduce. Seems fixed

repo: cwida/duckpgq-extension


text:
Display support

Hey :)
I love your project, and I'm amazed by what you've done. Is there any change you will be able to add support for displaying things on screen? I bought this thing, and after a lot of trouble with drivers for Windows, I also don't get it to work on Linux. I could test things if you would like me to.

Thanks for your interest.  With my current workload it's unlikely I'll have anything more written sooner than a few months.  I'll add another comment here when there's something you might find useful.

license_source: bigquery


text:
4.x

Had to rebase so we can merge into master

@job13er self-merging

repo: cyaninc/beaker

license: mit


text:
exposed apikey

hi, your apikey is exposed in one of your file

how

like

solved


text:
Secretless publishes images on tag-triggered builds

At current, Secretless master branch Jenkins pipelines run whenever there is a change to the master branch, and we determine whether to publish based on whether the most recent tag matches the current commit.
https://github.com/cyberark/secretless-broker/blob/8d7f757f7bcfb2cb26283e685da06e0cc8121d84/bin/publish#L36
We now have support in Jenkins for tag-triggered builds (docs link TBA), so we'd like to update the project to publish when it's a tag build rather than publishing based on a logical question that could fail if another commit is pushed to the master branch before the build completes.

Docs on Jenkins triggers
Example conditional stage for tag jobs only

authors: hughsaunders izgeri

repo: cyberark/secretless-broker


text:
Challenge the need of CONJUR_APPLIANCE_URL in the secrets-provider container

We say in the docs that this variable is needed but it doesn't seem to be used in the secrets-provider or in the conjur-authn-client.
We should challenge if this can be removed from the docs and the tests manifests (of both secrets-provider & kubernetes-conjur-demo).
Doc bug - https://app.zenhub.com/workspaces/moti-5d99d900491c060001c85cba/issues/cyberark/conjur-docs/591

This variable is used in the conjur ruby API which is used for retrieving secrets from Conjur. Closing this issue along with the doc issue.


text:
Find middle Element of LinkedList

Implemented a function to when given a head, that returns the middle Element of a LinkedList. Please assign this beatiful issue to me @lovishprabhakar

/assign


text:
added my company challenge project

added my small challenge project from codecademy for a fictional company

Good work @MarcHartmannDev!

authors: MarcHartmannDev lovishprabhakar


text:
main == preset-uncommon

Uncommon

Use Data .toString() so I can get the time according to my current TZ.

Thanks for the PR; I will look into this soon.

You can review the changes now.

sk

authors: cybercoder-naj suryavirkapur

repo: cybercoder-naj/logestic


text:
Integer out of range

Nice project! My id's have bigint as datatype, I get this error when I run the populate_facets function:
ERROR:  integer out of range
CONTEXT:  SQL statement "INSERT INTO explore.gbif_enriched_facets
SELECT facet_id, (gbifid >> 20) chunk_id, facet_value collate "POSIX", rb_build_agg(gbifid::int4 ORDER BY gbifid)
FROM explore.gbif_enriched d,
    LATERAL (
        VALUES (1, date_trunc('decade', eventdate)::text),
               (2, recordedby::text),
               (3, preparations::text),
               (4, stateprovince::text),
               (5, lifestage::text)
    ) t(facet_id, facet_value)
GROUP BY facet_id, facet_value collate "POSIX", chunk_id
    "
PL/pgSQL function faceting.populate_facets(oid,boolean,boolean) line 23 at EXECUTE
SQL statement "SELECT faceting.populate_facets(v_table_id, false)"
PL/pgSQL function faceting.add_faceting_to_table(regclass,name,faceting.facet_definition[],text,boolean,boolean) line 60 at PERFORM
SQL state: 22003

Added support for bigint typed columns in 90c9a83b8f3d3c16c39e7a1dd9ec9242ca3e7717. It will still cause out of range though for values > 2^51. However the roaringbitmap datatype is quite inefficient for sparse datatypes anyway so for globally unique id's youd want to maintain a denser sequentially assigned id column anyway. I will add a documentation example for that.

authors: ants rudibroekhuizen

repo: cybertec-postgresql/pgfaceting


text:
Issue 41: add local fromEvent module

Remove rx-dom dependency.
Add fromEvent local module.
References #41, #42

Any comments, @staltz ?

Ah great, I'll merge this. Sorry for not commenting earlier, I sometimes don't get email notification from github (it seems like a bug), I don't know why.

I cannot express enough thumbs up for this :+1: :+1: :+1: :+1: :+1: :dancer:

I'll go test it and release a new minor version

authors: Frikki staltz

repo: cyclejs/cycle-dom


text:
Not working with python3

When I run on python3, including the tests, I get a lot of errors - https://travis-ci.org/hanxue/infoqscraper/jobs/14867284
Pardon my limited Python knowledge - is infoqscraper written for Python 2.x and no foreseeable plans to support Python 3.x?

Bump

Hi Francisco,
IIRC I added Python 3 support several years ago but forgot to close this issue. Are you facing an issue ?

I tried using it today with python 2, but had to fall back to python 2.
OSX default instalation of Anaconda, and brew's version of Python 2.7.
I may be able to fetch the bug tomorrow.

I am closing this one.
You are welcome to open a new ticket if you face any issue with python 3 or OS X.

authors: FranciscoE-Hudl cykl hanxue

repo: cykl/infoqscraper


text:
Automate making secp256k1 lib from source

I'm proceeding with using precompiled secp256k1 libraries for the time being, but we could automate making it as in https://github.com/cypherstack/flutter_liblelantus/blob/main/scripts/linux/CMakeLists/secp256k1/template_CMakeLists.txt

per Julian, see also https://stackoverflow.com/questions/5971921/building-a-library-using-autotools-from-cmake for building sparkmobile's own secp256k1 impl instead of using the flutter_liblelantus method

plugin-branch's approach (to which staging has now been fast-forwarded) obviates the need for this issue because the old (flutter_liblelantus-style) approach builds secp256k1 from source.


text:
Error: 'Browser.setWindowBounds' wasn't found

Using chrome latest (unstable). What is even more strange is when I call console.log(Browser) I see it defined but cannot use it.

Full Stack:
{ Error: 'Browser.setWindowBounds' wasn't found
    at ProtocolError (/Users/anton/personal/node-selenium-starter/node_modules/chrome-remote-interface/lib/chrome.js:14:9)
    at /Users/anton/personal/node-selenium-starter/node_modules/chrome-remote-interface/lib/chrome.js:69:28
    at Chrome.handleMessage (/Users/anton/personal/node-selenium-starter/node_modules/chrome-remote-interface/lib/chrome.js:247:13)
    at WebSocket.<anonymous> (/Users/anton/personal/node-selenium-starter/node_modules/chrome-remote-interface/lib/chrome.js:224:27)
    at emitTwo (events.js:106:13)
    at WebSocket.emit (events.js:191:7)
    at Receiver._receiver.onmessage (/Users/anton/personal/node-selenium-starter/node_modules/ws/lib/WebSocket.js:143:54)
    at Receiver.dataMessage (/Users/anton/personal/node-selenium-starter/node_modules/ws/lib/Receiver.js:385:14)
    at extension.decompress (/Users/anton/personal/node-selenium-starter/node_modules/ws/lib/Receiver.js:354:40)
    at _inflate.flush (/Users/anton/personal/node-selenium-starter/node_modules/ws/lib/PerMessageDeflate.js:279:12) code: -32601 }

Yeah, that's strange actually, it seems that the method is present in the protocol descriptor (even with the remote option) but it's not actually implemented.
You should file an issue to the Chromium project or to the devtools-protocol. Feel free to link to this issue if you do so.

when I call console.log(Browser) I see it defined but cannot use it

This is just FYI, please note that the version you're seeing there (unless you're passing the remote option) is the local version of the protocol.json which may or may not be in sync with the version of Chrome you're using. More info here and here.

I'm closing this as there's nothing I can do about it, but if you decide to open an upstream issue please post the link here. Thanks!

authors: abacaj cyrus-and

repo: cyrus-and/chrome-remote-interface


text:
Explicitly offloading nogil code to device/GPU

Allow marking nogil-code to be run on a device/GPU.
There is also a discussion on the mailing-list: https://mail.python.org/pipermail/cython-devel/2020-January/005262.html.
Older CEPs suggest doing similar things by

either automatically determining regions for offload using OpenCL  (https://github.com/cython/cython/wiki/enhancements-opencl) which is relatively tricky and intriduces a new tool
or pretty extensively extend the Cython language (https://github.com/cython/cython/wiki/enchancements-metadefintions)

A first and already very powerful step would be to explicitly mark code that should be offloaded, minimize language extensions and not require extra tools.
As a start, we could consider only parallel devices, such as GPUs and use OpenMP target since Cython already uses OpenMP for parallelism
Let's consider a simple example for computing pairwise distances between vectors in parallel:
def pairwise_distance_host(double[:, ::1] X):
    cdef int M = X.shape[0]
    cdef int N = X.shape[1]
    cdef double tmp, d
    cdef double[:, ::1] D = np.empty((M, M), dtype=np.float64)
    cdef int i,j,k
    with nogil:
        for i in prange(M):
            for j in prange(M):
                d = 0.0
                for k in range(N):
                    tmp = X[i, k] - X[j, k]
                    d = d + tmp * tmp  # cython would interpret '+=' as a reduction variable!
                    D[i, j] = sqrt(d)
    return np.asarray(D)

The parallel region is implicitly defined by the first, outermost prange. For offloading we could demand that the parallel region needs to be defined explicitly:
...
    with nogil, parallel():
        for i in prange(M):
...

Now all we need is a marker that the parallel region should be offloaded to a device:
...
    with nogil, parallel(device={}):
        for i in prange(M):
...

Cython should take care of a safe way to define data mappings: transferring the necessary data to the device and from device to the host:

by default arrays are sent from host to device when entering the parallel region and from the device to host when exiting
read-only data is only sent from host to device but not from device to host
ideally, write-only data will not be sent from host to device but only from device to host
ideally, we can also detect that a variable is not used outside the parallel region so that we do not need to transfer any data (only allocate and deallocate)

Because complex indexing can make it impossible to correctly determine the best mapping and because data-movement is often the biggest performance bottleneck, we also need a way for experts to optimize the data movement. For that, device should accept a dictionary mapping variable names to a map-type (as borrowed from OpenMP target map clauses)

"to" means host to device
"from" means device to host
"tofrom" means host to device and device to host
"alloc" means not data-transfer at all

In the above example, the input array/memview is read-only on the device, so we could indicate it like this:
...
    with nogil, parallel(device={X:'to'}):
        for i in prange(M):
...

Map-values provided in device overwrite whatever Cython would automatically infer.
Another common challenge in offloading is that computation might go back and forth between host and GPU. In such cases it is often required to keep data on the GPU between different GPU regions even if a host-section is in between. As an example, let's look at the above code and block the computation by only computing a single row of the output array at once. Note that this will be needed anyway if the input array becomes large since the output vector size increases with quadratically and might simply not fit on the GPU.
def pairwise_distance_row(double[:, ::1] X):
    cdef int M = X.shape[0]
    cdef int N = X.shape[1]
    cdef double tmp, d
    cdef double[:, ::1] D = np.empty((M, M), dtype=np.float64)
    cdef double[::1] Dslice
    cdef int i,j,k
    with nogil:
        for i in range(M):
            Dslice = D[i,:]
            with parallel(device={Dslice:'from', X:'to'}):
                for j in prange(M):
                    d = 0.0
                    for k in range(N):
                        tmp = X[i, k] - X[j, k]
                        d = d + tmp * tmp  # cython would interpret '+=' as a reduction variable!
                        Dslice[j] = sqrt(d)
    return np.asarray(D)

Even though we only transfer slices of D in Dslice from device to host, the entire input array X will be send from host to device in every iteration of the outermost loop. The suggested solution adds a data-context (to be used with with) defining the lifetime of variables on the device. Let's simply use the same keyword device and let it accept the same mappings:
...
    with nogil, device({X:'to'}):
        for i in range(M):
...

Since this is an expert tool we might not want or need to infer any map-type and leave it to the programmer.
Calling functions in a device block
The OpenMP compiler will try to inline a function that appears in a target/device section/block and will usually
complain if that's not possible. For such cases Cython provides the decorate @cython.device to explicitly make
functions available on the device::
    @cython.device
    cdef double _dist(double[:] v1, double[:] v2) nogil:
        cdef double d = 0.0
        cdef double tmp
        for k in range(min(v1.shape[0], v1.shape[0])):
            tmp = v1[k] - v2[k]
            d += tmp * tmp  # cython would interpret '+=' as a reduction variable!
        return sqrt(d)

    def pairwise_distance_target_row_context_annotated_func(double[:, ::1] X):
        cdef int M = X.shape[0]
        cdef double tmp, d
        cdef double[:, ::1] D = np.empty((M, M), dtype=np.float64)
        cdef double[::1] Dslice
        cdef int i,j,k
        with device({X:'to'}):
            with nogil:
                for i in range(M):
                    Dslice = D[i,:]
                    with parallel(device={Dslice:'from'}):
                        for j in prange(M):
                            Dslice[j] = _dist(X[i], X[j])
        return np.asarray(D)

A first prototype is implemented here: https://github.com/fschlimb/cython/tree/offload
Limitations, open questions etc

In most cases the generated code does not work if @boundscheck and @wraparound are not set to False.
OpenMP does not allow mapping overlapping memory regions. We need to at least check that 2 memviews do not overlap
C-pointers are not properly checked
Only C-contiguous memviews are supported
need support for setup.py/distutils
need tests
need docu (syntax, semantics, and how to setup offload compiler)
error reporting back to host is disabled, properly mapping related variables could allow a useful error reporting
string support has not been looked at yet

@ogrisel @GaelVaroquaux @oleksandr-pavlyk @DrTodd13

This is exciting!

A first prototype is implemented here: https://github.com/fschlimb/cython/tree/offload

@fschlimb the link is broken.

@jeremiedbb Sorry. The branch is now back online.

I also added the keyword simd for prange which will add it to the openmp pragma (without any further check)

I managed to build llvm/clang with openmp offloading capability. However, using your branch of cython the compilation of the toy program you wrote in the beginning of this thread (pairwise_distance_host) fails with this error message:
 error: unexpected OpenMP clause 'shared' in directive '#pragma omp for'
         #pragma omp for private(__pyx_t_10, __pyx_t_11, __pyx_t_12, __pyx_t_13, __pyx_t_14, __pyx_t_15, __pyx_t_16, __pyx_t_17, __pyx_t_18, __pyx_t_19, __pyx_t_20, __pyx_t_21, __pyx_t_22) private(__pyx_filename, __pyx_lineno, __pyx_clineno) shared(__pyx_parallel_why, __pyx_parallel_exc_type, __pyx_parallel_exc_value, __pyx_parallel_exc_tb) lastprivate(__pyx_v_d) firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) lastprivate(__pyx_v_j) lastprivate(__pyx_v_k) lastprivate(__pyx_v_tmp)

Here I'm not even trying to use offloading. I tried with cython master and it compiles.

@jeremiedbb I forgot to mention that you need to switch off some cython checks.
# cython: boundscheck=False
# cython: wraparound=False

Sometimes you also need # cython: initializedcheck=False .
Of course you can also use @decorators.
This can probably auto-enabled in with-device blocks, but for now I chose the easier path...

thanks, it works

now I have this error when trying to use  offloading:
/home/parietal/jduboisb/R/tst/tst.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __tgt_register_requires

Here's what it does for me:
clang -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fopenmp -fopenmp-targets=nvptx64 -fPIC -I.../miniconda3/envs/cython/include/python3.8 -c test_pairwise_distance.c -o build/temp.linux-x86_64-3.8/test_pairwise_distance.o

clang -shared -fopenmp -fopenmp-targets=nvptx64 build/temp.linux-x86_64-3.8/test_pairwise_distance.o -o build/lib.linux-x86_64-3.8/test_pairwise_distance.cpython-38-x86_64-linux-gnu.so

I just forgot to put the -fopenmp-targets=nvptx64 in the link flags as well... It works now \o/
Thanks for your help. I will do some tests with real code from scikit-learn and I'll let you know how it goes.

Great!

I looked at kmeans in scikit-learn and identified a limitation. scikit-learn currently implements an explicit parallel loop, which lets each thread call gemm on its own block (I take this is to avoid oversubscription due to MKL's internal threading). While calling functions within a device block is ok, MKL does not provide device functions for BLAS (to call BLSA from the device). Porting the code as-is to OpenMP-offload is quite a challenge (we'd need to transform the prange into a series of OpenMP tasks with appropriate dependences).
Question (@ogrisel @GaelVaroquaux @jeremiedbb): is this a common pattern (for example in scikit-learn) we'd want to see supported? Providing a specialization one algorithm or two for GPU would probably be ok- the question is if this is a common or rare case.
From the cython perspective I tend to believe this is a corner case.
BTW: Another option for sklearn would be to provide a task-based backend for cython and joblib based on dpc++/SYCL and then by default avoid the explicit outer prange...

We could make some experiments in a branch of scikit-learn where the euclidean distances are computed in a more naive way without calling into BLAS to see whether or not GPU offloading can bring us good performances. And we could compare to an implementation of KMeans that uses vendor specific BLAS implementations such as cuML / cuBLAS.

authors: GaelVaroquaux fschlimb jeremiedbb ogrisel

repo: cython/cython


text:
Enable remote_autostart as default on Xdebug configuration

Fix #733
Goal
Activate the Xdebug Step Debugging automatically after start debugging (on Visual Studio Code installed on any Linux OS) and refresh the browser page.
DESCRIPTION
According to the XDebug documentation

boolean xdebug.remote_autostart = false
Normally you need to a HTTP GET/POST variable to start remote debugging (see Step Debugging). When this setting is set to 1, Xdebug will always attempt to start a remote debugging session and try to connect to a client, even if the GET/POST/COOKIE variable was not present.

See my issue Xdebug doesn't work when configured for Visual Studio Code (on Linux)
More information on
https://xdebug.org/docs/all_settings
https://xdebug.org/docs/remote#browser_session

How can I proceed to resolve this PHP8 checks?

repo: cytopia/devilbox