text:
Reading cloudflare:index:Ruleset Causes Panic

What happened?
On the latest version of the provider (v5.7.0 at time of writing) we're attempting to import a cloudflare:index:Ruleset resource into our stack and running into the following panic:
Diagnostics:
  cloudflare:index:Ruleset (CDN Optimization Rules):
    error: Preview failed: error reading from server: EOF

  pulumi:pulumi:Stack (infrastructure-main):
    panic: DefaultValue() should not be called during schema generation
    goroutine 106 [running]:
    github.com/pulumi/pulumi-terraform-bridge/pf/internal/schemashim.(*blockSchema).DefaultValue(0xc000730870)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/pf@v0.9.2/internal/schemashim/block_schema.go:123 +0x27
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.getDefaultValue(...)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.51.0/pkg/tfbridge/schema.go:1350
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.isDefaultOrZeroValue({0x18c2780, 0xc00116c840}, 0x18c2780?, {{0x13819e0?, 0xc0001a5278?}})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.51.0/pkg/tfbridge/schema.go:1361 +0x44
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.extractSchemaInputs({{0x158f780?, 0xc001211920?}}, {0x18c29c0, 0xc0011115c0}, 0xc000d712c0?, 0x0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.51.0/pkg/tfbridge/schema.go:1438 +0x3e6
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.ExtractInputsFromOutputs(0x18b9810?, 0xc0005fa820?, {0x18b7448?, 0xc001179ed0}, 0xc000730870, 0x0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.51.0/pkg/tfbridge/schema.go:1468 +0x15f
    github.com/pulumi/pulumi-terraform-bridge/pf/tfbridge.(*provider).ReadWithContext(0xc00025f600, {0x18b36c8?, 0xc000614b70?}, {0xc0011c6a00, 0x9d}, {0xc000aa89b0, 0x46}, 0x0?, 0xc000614bd0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/pf@v0.9.2/tfbridge/provider_read.go:69 +0x2f9
    github.com/pulumi/pulumi-terraform-bridge/pf/internal/plugin.(*providerServer).Read(0xc000436870, {0x18b36c8, 0xc000614b70}, 0xc0007bf9e0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/pf@v0.9.2/internal/plugin/provider_server.go:370 +0x188
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Read.func1({0x18c1770?, 0xc000436870?})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.4/muxer.go:350 +0x39
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.resourceMethod[...](0xc0005fad20?, 0x40, 0xc00010f780?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.4/muxer.go:303 +0xbd
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Read(0x0?, {0x18b36c8?, 0xc000614b70?}, 0x40?)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.4/muxer.go:349 +0x68
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Read_Handler.func1({0x18b36c8, 0xc000614b70}, {0x154f420?, 0xc0007bf9e0})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.72.2/proto/go/provider_grpc.pb.go:591 +0x7b
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x18b36c8, 0xc0006148a0}, {0x154f420, 0xc0007bf9e0}, 0xc00003ff00, 0xc0005e5ab8)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3e8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Read_Handler({0x15d1b60?, 0xc0005fad20}, {0x18b36c8, 0xc0006148a0}, 0xc0006af810, 0xc000434080)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.72.2/proto/go/provider_grpc.pb.go:593 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc000630000, {0x18bb640, 0xc000aaa000}, 0xc00030fe60, 0xc000436900, 0x23a1ed8, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.0/server.go:1337 +0xdf3
    google.golang.org/grpc.(*Server).handleStream(0xc000630000, {0x18bb640, 0xc000aaa000}, 0xc00030fe60, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.0/server.go:1714 +0xa36
    google.golang.org/grpc.(*Server).serveStreams.func1.1()
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.0/server.go:959 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
    	/home/runner/go/pkg/mod/google.golang.org/grpc@v1.56.0/server.go:957 +0x18c
    error: preview failed

We thought that it was originally how we were importing the resource (something like maybe one of the parameters we were passing) but upon trying to create a minimal reproduction of the issue by just reading the RulesSet via a getter, we see the same panic.
See the "Steps to Reproduce" section for how we were able to reproduce it.
When we downgraded back to v5.0.0, we were able to read the resource successfully.
The issue may have been introduced in v5.1.0, but we are unable to run the provider (there was a bug fixed in #406). v5.1.1 was the first actual version where we could see the bug when we were attempting to "walk" versions.
Expected Behavior
We initially expected to be able to import the resource successfully.
In our minimum reproduction, we should be able to pulumi.export() the attributes of the resource without a panic.
Steps to reproduce
I created our minimal reproduction via Docker.
Start a container with:
$ docker run --rm -it ubuntu:22.04

Inside the container, run a few things to bootstrap the environment:
apt update && apt install -y curl python3 python3.10-venv
curl -fsSL https://get.pulumi.com/ | sh
export PATH=~/.pulumi/bin:$PATH
pulumi login --local
mkdir /tmp/testing
cd /tmp/testing
python3 -m venv .
source bin/activate
pip install pulumi pulumi-cloudflare
export CLOUDFLARE_API_TOKEN="OUR_TOKEN_HERE"
export PULUMI_CONFIG_PASSPHRASE="foobar"

Create a requirements.txt with the following:
Arpeggio==2.0.2
attrs==23.1.0
dill==0.3.7
grpcio==1.56.0
parver==0.4
protobuf==4.23.4
pulumi==3.76.1
pulumi-cloudflare==5.7.0
PyYAML==6.0.1
semver==2.13.0
six==1.16.0

Install the requirements with:
pip install -r requirements.txt

Using the automation API, create a script like the following. Be sure to replace the values of ZONE_ID and RULESET_ID.
import json
import sys

import pulumi
import pulumi_cloudflare as cloudflare
from pulumi import automation as auto

ZONE_ID = "09deadbeef"
RULESET_ID="09badcaffe"

def prog():
    ruleset = cloudflare.Ruleset.get(
        "ruleset",
        f"zone/{ZONE_ID}/{RULESET_ID}",
    )

    pulumi.export("ruleset", ruleset)

project_name = "testing"
stack_name = "testing"
stack = auto.create_or_select_stack(
    stack_name=stack_name, project_name=project_name, program=prog
)

stack.workspace.install_plugin("cloudflare", "v5.7.9")
print("plugins installed")

stack.refresh(on_output=print)

stack.preview(on_output=print)

The refresh is successful, but when you start doing the preview, the panic occurs.
Output of pulumi about
Other notes:

Arch is aarch64 as I am on an M1 Mac currently. Our runners where the initial panic occurred during CI run x86_64.
The warnings are just because I'm using the automation API, so there's no project or stack file.
root because I'm lazy and in a Docker container.

Version      3.76.1
Go Version   go1.20.6
Go Compiler  gc

Host
OS       ubuntu
Version  22.04
Arch     aarch64

Backend
Name           c131e0286c4d
URL            file://~
User           root
Organizations

Pulumi locates its logs in /tmp by default
warning: Failed to read project: no Pulumi.yaml project file found (searching upwards from /tmp/testing). If you have not created a project yet, use `pulumi new` to do so: no project file found
warning: Failed to get information about the current stack: no Pulumi.yaml project file found (searching upwards from /tmp/testing). If you have not created a project yet, use `pulumi new` to do so: no project file found

Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@guineveresaenger Could you please take a look?

I got context here. It's a plugin framework bug in the bridge. What happened is that
tfbridge.ExtractInputsFromOutputs
got reused from the sdk/v2 bridge which is no longer safe as it passes schema-only provider shim into sdk/v2 bridge machinery. The schema-only provider is not equipped for runtime use, hence we get DefaultValue() should not be called during schema generation while not being in the schema generation context anymore. A possible fix here is to wrap the schema-only provider when passing to tfbridge.ExtractInputsFromOutputs or use a weaker interface in tfbridge.ExtractInputsFromOutputs. In the Plugin Framework case there is no capability currently to detect default values for properties; but proceeding as if no default value is defined is preferable to having a panic.

Hi @dmizelle - this should be fixed on the next release of this provider!

authors: dmizelle guineveresaenger mikhailshilkov t0yv0

repo: pulumi/pulumi-cloudflare


text:
Shared Services Platforms page "How Pulumi Helps" section should be updated

the how pulumi helps section on shared services looks really strange followed by the section above it; id consider moving this to the bottom or changing to 3 cards and doing a row

This is about https://www.pulumi.com/solutions/shared-services-platforms/.

source: gharchive/issue

authors: lukehoban zchase

repo: pulumi/pulumi-hugo

license_type: permissive

license_source: github-api


text:
follow up bug fixes to docs nav reorg

Description
these arent blocking, only bugs i found
Checklist:

[x] I have reviewed the style guide.
[x] If blogging, I have reviewed the blogging guide.
[x] I have manually confirmed that all new links work.
[x] I added aliases (i.e., redirects) for all filename changes.
[x] If making css changes, I rebuilt the bundle.

this will go green once docs stuff lands

source: gharchive/pull-request


text:
Learn Pulumi with a module and some topics

This example takes the Introduction to Pulumi on AWS tutorial and ports it to a Learn Pulumi "module" (which we will soon rename to "tutorial") to give us a sense of what we're currently working with.

also just in case people are looking for it
https://github.com/pulumi/pulumi-hugo/blob/master/LEARN.md

authors: cnunciato mattstratton


text:
connection error:  socket closed

Hello,
since the update i'm always getting the error: connection error: socket closed.
I think the server closes the socket..i don't need an solution i only want to knnow what happened.
Did the agar.io server rejected the client ? (i changed my init key)
The proxies i use aren't bad ones, and before the update they worked perfect. My other thougt was that  agar.io banned my ip's.... do you know what happened ?
Thanks,
David

Do you use agario-client version 1.3.0?
examples/basic.js works fine for me, is it broken for you?

yes I do....

authors: d-bots-client pulviscriptor

repo: pulviscriptor/agario-client

license: mit

license_source: bigquery


text:
NOT WORKING

plz tell how to install in detailed instructions and doesnt it have a code to install andn tampermonkey
if not just give how to make it work

This is not bot, this is library for agar.io protocol. If you know Node.JS/JavaScript language then you install it via npm install agario-client, read documentation https://github.com/pulviscriptor/agario-client/blob/master/README.md and write your own code.
If you don't then you're in the wrong place.

i did make one but only normal are coming for facebook it says failed to get token

Dude.. how often does i have to say it?? ALL LEAKED ACCOUNTS ARE BANNED FROM $MONEY$CLIP$

can you help me make it work at least trying with my one account

Set the cookies of YOUR facebook account in and then log off ur Account from agario and then run the feeder

not working do you have your script that is working

or can you say how to get  token
for agario fb

@kurama129 you dont have to acces the token, just get COOKIES!!!


text:
Searching through items in a piece while on a page other than 1 will not show results

When I am on the second page of items in a piece (in this example the second page of People) and try to search, I get back a page that says it has results but displays none. If I choose at the bottom to go to Page 1 they will display, so it seems like the Search feature does not bring you to the page 1 of search results.

This got fixed.

authors: boutell runyx

repo: punkave/apostrophe


text:
[Bug]: Cannot use the browser immediately after installing it

Steps to reproduce
With an empty cache (~/.cache/puppeteer does not exist), after installing the browser using browsers.install, we cannot use the browser with puppeteer.launch.
You can reproduce the problem with the code at https://github.com/rgl/try-puppeteer-in-bun:
git clone https://github.com/rgl/try-puppeteer-in-bun.git
cd try-puppeteer-in-bun
bun install --frozen-lockfile
rm -rf ~/.cache/puppeteer
bun run main.js

Expected results
Expected it to launch the just installed browser.
Actual results
It fails with:
2023-09-09T14:17:43.077Z Downloading the browser Chrome/116.0.5845.96...
2023-09-09T14:17:49.732Z Launching the browser...
610 |   }
611 |   get stdio() {
612 |     return this.#stdioObject ??= this.#createStdioObject();
613 |   }
614 |
615 |   spawn(options) {
                 ^
ESRCH: No such process
 syscall: "open"
   errno: -3

      at spawn (node:child_process:615:14)
      at node:child_process:2:40
      at new Process (/home/vagrant/Projects/try-puppeteer-in-bun/node_modules/@puppeteer/browsers/lib/esm/launch.js:104:31)
      at launch (/home/vagrant/Projects/try-puppeteer-in-bun/node_modules/@puppeteer/browsers/lib/esm/launch.js:58:11)
      at /home/vagrant/Projects/try-puppeteer-in-bun/node_modules/puppeteer-core/lib/esm/puppeteer/node/ProductLauncher.js:60:31
      at processTicksAndRejections (:1:2602)

@rgl does it reproduce in node?

@OrKoN, it does work fine with node 18.17.1, but not with bun 1.0.0.

@rgl is it the problem with bun perhaps?

I forgot to mention, the second time its executed in bun, it works fine. It only fails when starting with a clean cache.

I see, it would sound like some of the buns API are not completely compatible. Note that we don't support bun so if you can figure out what's the difference between bun and node, I would recommend filing an issue against bun.

authors: OrKoN rgl

repo: puppeteer/puppeteer


text:
Request.continue(overrides) does not set cookie

Steps to reproduce
Tell us about your environment:

Puppeteer version: 2.0.0
Platform / OS version: Ubuntu
Node.js version: 10.15.3

What steps will reproduce the problem?
Start a puppeteer browser with the request interception code:
await page.setRequestInterception(true);
page.on('request', request => {
  // Override headers
  const headers = {cookie: 'hello=world;'};
  request.continue({headers});
});

Browse to a page which already has a cookie set by the browser, then check the request received by the server.
What is the expected result?
The server should receive the cookie header: "hello=world".
What happens instead?
The server receives receives the original cookie header set by the page, but it does not equal "hello=world" as it should. Setting the cookie header only works if the original request does not have a cookie set at all.
What is the use case?
I'm developing an app for pen-testing web apps, I need to let the user tamper with the request data i.e. changing header values (including cookies!), change POST data etc.

I have same problem, i can't solve it

have a look at fetchURLResponse() function located in file puppeteer-bot-2a.js in my project: https://puppeteer.theater/

use page.setcookie

the same problem

This is still an issue for me on puppeteer v22.11.2. However, I find that I can override cookies if I use the stealth plugin:
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

authors: entrity evanrolfe nicoandmee zhaojunlike ztob


text:
Chore: expose quality variable

Sometimes users do not want very high quality video that may take much disk space or they may want clearer video. This PR is to expose quality variable so that users can adjust it themselves to meet the needs of testing.

Hi @Lightning00Blade I saw this PR failed. Can you help to check the failure?


text:
(BKR-536) skip ssh installation for images that already provide it

Add check to see if a host tells us it already has ssh installed, if so, don't attempt to install it again.  To skip installation, add the key  to your node definition, eg:
HOSTS:
  pe-puppet.localdomain:
    roles:
      - "agent"
      - "master"
    platform: "el-7-x86_64"
    image: "pe2015-2-0_aio-master"
    ssh_installed: true

Refer to this link for build results (access rights to CI server needed):
http://jenkins-beaker.delivery.puppetlabs.net//job/qe_beaker_btc-intn/1522/
Test PASSed.

I'm -1 on this. We should either do auto detection which is not possible , or just not care about it.
it brings no extra overhead anyway.

@electrical how would we autodetect?  I need something along these lines as I've got a docker image that isn't bootable with stock beaker. What do you think about my suggest to have static Dockerfiles (hipchat channel)?  That would solve this for me.  As for not caring - how can we bypass the above failure? thanks

{"stream":"Step 3 : RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key\n"}

{"stream":" ---\u003e Running in 6c122e50aff0\n"}

{"stream":"Generating public/private rsa key pair.\n/etc/ssh/ssh_host_rsa_key already exists.\nOverwrite (y/n)? "}

{"errorDetail":{"code":1,"message":"The command '/bin/sh -c ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key' returned a non-zero code: 1"},"error":"The command '/bin/sh -c ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key' returned a non-zero code: 1"}

Your problem is not as much that ssh is installed but that we always do the keygen part.
There are 2 ways we can solve this;

The main image you use, remove the generated keys out of it and let beaker generate it. This ensures its random at all times.

Do a test if the file exist and only if it doesn't run ssh-keygen.
For example: test -f /etc/ssh/ssh_host_rsa_key|| ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
This should only run the ssh-keygen part if the file doesn't exist.

@electrical I had a think about this and came up with a different approach - closed in favour of https://github.com/puppetlabs/beaker/pull/961

repo: puppetlabs/beaker


text:
(GH-2210) Configure 'modules' with 'bolt project init'

This updates the bolt project init command to configure modules in
bolt-project.yaml, allowing users with new projects to immediately
start using the module workflow.

If bolt project init is run, modules will be an empty array []
If bolt project init --modules is run, modules will be an array of
declarations for the specified modules

This also updates the command so that it can only be run once per
project. Previously, if a project had a configuration file, but no
Puppetfile, the bolt project init --modules command would still
install modules. Now, Bolt will error in this case and display a helpful
error that the user should instead use the bolt module add command.
Closes #2210
!feature

Configure modules with bolt project init
(#2110)
The bolt project init command will now configure the modules key
in the bolt-project.yaml file, enabling the bolt module command.

CLA signed by all contributors.

authors: beechtom puppetcla

repo: puppetlabs/bolt


text:
(BOLT-1195) Convert a YAML plan to a Puppet plan

This creates a new command bolt plan convert <path-to-plan> which accepts either a relative or absolute path a YAML plan and prints to stdout the equivalent Puppet plan.

This relies on https://github.com/puppetlabs/puppet/pull/7485

Would be helpful to (temporarily) commit your Gemfile pointing to the change in puppet.

Travis failures are just rubocop on the Gemfile :P

CLA signed by all contributors.

it looks like puppet/pal is not being set up properly
bundle exec bolt plan convert yaml/plans/foo.yaml
bundler: failed to load command: bolt (/Users/adreyer/src/bolt/.bundle/ruby/2.5.0/bin/bolt)
NameError: uninitialized constant Bolt::PAL::YamlPlan::Puppet
Did you mean?  Bolt::PuppetDB
  /Users/adreyer/src/bolt/lib/bolt/pal/yaml_plan.rb:77:in `block in initialize'
  /Users/adreyer/src/bolt/lib/bolt/pal/yaml_plan.rb:68:in `each'
  /Users/adreyer/src/bolt/lib/bolt/pal/yaml_plan.rb:68:in `map'
  /Users/adreyer/src/bolt/lib/bolt/pal/yaml_plan.rb:68:in `initialize'
  /Users/adreyer/src/bolt/lib/bolt/transpiler.rb:59:in `new'
  /Users/adreyer/src/bolt/lib/bolt/transpiler.rb:59:in `parse_plan'
  /Users/adreyer/src/bolt/lib/bolt/transpiler.rb:22:in `transpile'
  /Users/adreyer/src/bolt/lib/bolt/cli.rb:275:in `execute'
  /Users/adreyer/src/bolt/exe/bolt:10:in `<top (required)>'
  /Users/adreyer/src/bolt/.bundle/ruby/2.5.0/bin/bolt:23:in `load'
  /Users/adreyer/src/bolt/.bundle/ruby/2.5.0/bin/bolt:23:in `<top (required)>'

Re: Templating: I see a few negatives to templating without any gain over building a string. 1. It's less readable, especially if like me you're not familiar with it 2. It's trickier to support inline logic, like "only add a parenthesis here if there are no parameters otherwise add a newline" all on one line is very long, 3. It adds a new tool to learn (granted it's ruby-ish, but it's still adding complexity). I didn't think it was worth exploring further, but if there are advantages I'm not considering I'm happy to listen.

It looks like strings are always being emitted in single quotes. We should be checking whether it's an EvaluableString and switching based on that. Strings which were double-quoted in the YAML should be double-quoted in the output.

WIthout having looked at code and just playing with this I see a couple things.

No convert description in bolt help

cas@cas-ThinkPad-T460p:~/working_dir/bolt$ bolt
Usage: bolt <subcommand> <action> [options]

Available subcommands:
  bolt command run <command>       Run a command remotely
  bolt file upload <src> <dest>    Upload a local file
  bolt script run <script>         Upload a local script and run it remotely
  bolt task show                   Show list of available tasks
  bolt task show <task>            Show documentation for task
  bolt task run <task> [params]    Run a Puppet task
  bolt plan show                   Show list of available plans
  bolt plan show <plan>            Show details for plan
  bolt plan run <plan> [params]    Run a Puppet task plan
  bolt apply <manifest>            Apply Puppet manifest code
  bolt puppetfile install          Install modules from a Puppetfile into a Boltdir
  bolt puppetfile show-modules     List modules available to Bolt

Having a step description can result in double quoting in arg list

parameters:
  nodes:
    type: TargetSpec
  foo:
    type: String[1]
    description: foo
    default: bar
steps:
  - name: run_task
    task: sample
    target: $nodes
    description: hi
    parameters:
      message: hello world
return: $run_task

cas@cas-ThinkPad-T460p:~/working_dir/bolt$ bolt plan convert Boltdir/site/yaml_plans/plans/test_convert.yaml
# WARNING: This is an autogenerated plan. It may not behave as expected.
plan yaml_plans::test_convert(
  TargetSpec $nodes,
  String[1] $foo = 'bar'
) {
$run_task = run_task('sample', $nodes, ''hi'', {'message' => 'hello world'})

return $run_task
}
The converted puppet plan contains invalid puppet code: Syntax error at 'hi' (line: 6, column: 42)

Eval can be used to have duplicate variable assignment that is not caught.

parameters:
  nodes:
    type: TargetSpec
  foo:
    type: String[1]
    description: foo
    default: bar
steps:
  - name: run_task
    task: sample
    target: $nodes
    parameters:
      message: hello world
  - eval: |
      $run_task = $foo
    name: eval_step

return: $run_task

cas@cas-ThinkPad-T460p:~/working_dir/bolt$ bolt plan convert Boltdir/site/yaml_plans/plans/test_convert.yaml
# WARNING: This is an autogenerated plan. It may not behave as expected.
plan yaml_plans::test_convert(
  TargetSpec $nodes,
  String[1] $foo = 'bar'
) {
$run_task = run_task('sample', $nodes, {'message' => 'hello world'})
$eval_step = $run_task = $foo

return $run_task
}

Also I think that the warnings/errors about invalid puppet are getting printed to both stdout and stderr.

@donoghuc:

I made adding the help text part of #977 for docs review
Woops! Removed the extra quotes.
I'm not sure what to do about that...I run the plan through the evaluating parser once it's converted, perhaps there's another puppet evaluator I need to use?
Investinating...

Conflicting variable assignments are explicitly one of the things we can't detect, and are indeed a
key reason we emit the warning about how the plan may not be correct.

@donoghuc I think I've addressed all your comments - using with() || seems to work for me for the eval code block use case. @nicklewis I think I'm going to head out for the night, but will make those changes first thing tomorrow!

I thought I remembered talking about printing the error/warning when we detect invalid puppet code to stderr only (thought i dont see that in the ticket)? I see that it is exiting with exit code 1 but it seems like the error is still being printed to stdout.

Do we need to give parameters the with() treatment like for eval?
parameters:
  nodes:
    type: TargetSpec
  x:
    type: Integer
  y:
    type: Integer
steps:
  - name: run_task
    task: sample::echo
    description: "foo"
    target: localhost
    parameters:
      message: |
        $z = $x + $y
        $z + 1
return: $run_task

cas@cas-ThinkPad-T460p:~/working_dir/bolt$ bolt plan run yaml_plans::test_convert x=1 y=1 -t localhost
Starting: plan yaml_plans::test_convert
Starting: foo on localhost
Finished: foo with 0 failures in 0.01 sec
Finished: plan yaml_plans::test_convert in 0.01 sec
Finished on localhost:
  cas-ThinkPad-T460p got passed the message: 3
  {
  }
Successful on 1 node: localhost
Ran on 1 node

cas@cas-ThinkPad-T460p:~/working_dir/bolt$ bolt plan convert Boltdir/site/yaml_plans/plans/test_convert.yaml
# WARNING: This is an autogenerated plan. It may not behave as expected.
plan yaml_plans::test_convert(
  TargetSpec $nodes,
  Integer $x,
  Integer $y
) {
  $run_task = run_task('sample::echo', 'localhost', "foo", {'message' => $z = $x + $y
$z + 1
})

  return $run_task
}
The converted puppet plan contains invalid puppet code: Syntax error at 'z' (line: 8, column: 1)

Re: Code blocks, I made https://tickets.puppetlabs.com/browse/BOLT-1285 - I'm going to resolve to comment just since it's kind of long.

authors: adreyer donoghuc lucywyman nicklewis puppetcla


text:
1.1.1 release notes

This includes all the changes from Jeff's original PR, plus some minor wording changes. We can merge it once we have the go/no-go meeting, since merging it will post the release notes to the docs site.

:+1: LGTM.  Will hold on merge until we do the go/no-go decision - hopefully later today.

repo: puppetlabs/puppet-server


text:
(PUP-9927) Exclude PKey.read monkey patch on JRuby 9.1

JRuby 9.1 (openssl-jruby) doesn't implement the OpenSSL::PKey.read
method despite being "compatible" with MRI 2.3. We don't need that
generic PKey.read method on JRuby, so don't try to monkey patch it.

We don't test master against jruby 9.1, but a quick test shows it loads successfully:
$ ruby --version
jruby 9.1.17.0 (2.3.3) 2018-04-20 d8b1ff9 Java HotSpot(TM) 64-Bit Server VM 25.162-b12 on 1.8.0_162-b12 +jit [darwin-x86_64]
$ bx rspec spec/unit/x509/cert_provider_spec.rb
Run options: exclude {:broken=>true, :benchmark=>true}
...........*........................................................

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Puppet::X509::CertProvider when loading crls and input is invalid raises when invalid input is inside BEGIN-END block
     # jruby bug: https://github.com/jruby/jruby/issues/5619
     Failure/Error:
       expect {
         create_provider(crlpath: crl_path).load_crls
       }.to raise_error(OpenSSL::X509::CRLError, 'nested asn1 error')

       expected OpenSSL::X509::CRLError with "nested asn1 error", got #<OpenSSL::X509::CRLError: java.io.IOException: malformed PEM data encountered> with backtrace:
         # ./lib/puppet/x509/cert_provider.rb:109:in `block in load_crls_from_pem'
         # ./lib/puppet/x509/cert_provider.rb:108:in `load_crls_from_pem'
         # ./lib/puppet/x509/cert_provider.rb:93:in `load_crls'
         # ./spec/unit/x509/cert_provider_spec.rb:137:in `block in (root)'
         # ./spec/unit/x509/cert_provider_spec.rb:136:in `block in (root)'
     # ./spec/unit/x509/cert_provider_spec.rb:136:in `block in (root)'

Finished in 2.34 seconds (files took 2.25 seconds to load)
68 examples, 0 failures, 1 pending

CLA signed by all contributors.

I should just remove our calls to OpenSSL::PKey.read and the related monkey patches. Let me fix that.

I haven't functionally tested this yet, but it seems clear and correct.

authors: joshcooper justinstoller puppetcla

repo: puppetlabs/puppet


text:
(maint) Update EZBake to 1.5.0

This version of EZBake stores the build metadata (including the deployed
snapshot version and fully-resolved versions of all snapshot
dependencies) in a persistent location, rather than one that gets
deleted after two weeks.

I tested this by creating a build job with lein with-profile ezbake ezbake build and observing that it completed successfully. According to the EZBake changelog the only difference between these two versions is the generation of this metadata.

Test PASSed

CLA signed by all contributors.

authors: aperiodic puppetcla puppetlabs-jenkins

repo: puppetlabs/puppetdb


text:
(PDB-2700) Migrate log test utils to trapperkeeper

Prior to this commit, testutil functions related to logging
were duplicated in trapperkeeper and puppetdb.
This commit removes the duplicated functions and switches to
the ones provided by trapperkeeper.

Test PASSed

CLA signed by all contributors.

Test PASSed

Test PASSed

Test PASSed

Test PASSed

Test PASSed

authors: austb puppetcla puppetlabs-jenkins


text:
(PDOC-75) Work with both versions of 'interpret_any'

Prior to this commit, strings would fail with puppet 4.4.0 and
newer. This was because strings was making use of a method that was
marked API private and thus subject to change. The method was changed
in 4.4.0 release of puppet to take two variables, meaning that strings
much adjust how it calls the method based on the version of puppet
it's running with.

@hlindberg @kylog Updated and tests are passing!

repo: puppetlabs/puppetlabs-strings


text:
Add initial version of wash find

This commit adds the initial support for wash find. wash find is
very similar to GNU find and BSD find in that it supports an expression
based language composed of primaries and operators. Primaries are
predicates constructed on a specific entry. This commit supports the -true,
-false, and -name primaries, where -true specifies a predicate that always
returns true; -false specifies a predicate that always returns false; and
-name  specifies a predicate that returns true if the entry’s cname
matches the specified shell glob.
The supported operators are “()”, “!/-not”, “-a/-and”, and “-o/-or” (in order of
decreasing precedence). “-not” takes a single predicate p and returns a
new predicate q s.t. q(e) = ! p(e), where ! is the logical not operator. -and
takes predicates p1 and p2 and returns a new predicate q s.t.
q(e) = p1(e) && p2(e), where && is the short-circuited logical and operator.
Similarly, -or takes predicates p1 and p2 and returns q(e) = p1(e) || p2(e)
where || is the logical or operator. “()” is useful to control precedence. For
example, wash find <path> -true -o -true -a -false will evaluate to true
since -a has greater precedence than -o. However,
wash find <path> ( -true -o - true ) -a -false will evaluate to false.
Like the existing find command, wash find also supports concatenating predicates.
For example, something like wash find <path> -true -false will be
parsed as wash find <path> -true -a -false.
Subsequent commits will add more primaries, a better usage, and will also
properly recurse down into all the listed subdirectories.
Signed-off-by: Enis Inan enis.inan@puppet.com
Contributions to this project require sign-off consistent with the Developers Certificate of Origin. This can be as simple as using git commit -s on each commit.

Pushed an update addressing the feedback.

repo: puppetlabs/wash


text:
Any plan for a dashboard?

So we can control and see all my stuff within Emacs.

I've thought about it and that was initially the plan when I started this project. It's a feature I'd love to add one day. The ground work is there now so maybe I'll work on that soon

I've been able to make a lot of progress on this already! I'd be happy if you'd be willing to try it out so far. It's in the dashboard branch. I still have a lot of testing to do, code cleanup, and documentation writing, but I think it's in a decent state so far.
If you use straight, you can use this recipe :straight (:type git :host github :repo "purplg/hass" :branch "dashboard")
I'd like to be able to add "rows" of widgets in the future, but that'd add a decent amount of additional complexity and I haven't decided on a good way to do it, so I'll hold off on that for now. Also I haven't done anything, except color the icons blue and make the group names larger, on the appearance of the dashboard. Eventually I wanna make it prettier and customizable.
If you have any other suggestions or ideas, I'd be happy to hear them.
The basic required setup is to configure hass-dash-layout according to the example in the docstring for that variable. Then you can call (hass-dash-open) and it'll open the dashboard. I can help you with setup if you decide to try it out. I realize the documentation is a bit lacking right now.

Awesome :tada: !!
With the dashboard branch I got this when I run (hass-setup). I'm not using straight btw, since I'm using a special nixos emacs config.
#s(request-response nil nil nil nil nil "https://my_url:443/api/" nil (:sync nil :type "GET" :headers (("User-Agent" . hass--user-agent) ("Authorization" . "Bearer my_token") ("Content-Type" . "application/json")) :data nil :parser #[0 "ÀÁ !" [hass--deserialize buffer-string] 2] :error hass--request-error ...) #<buffer  *request curl*-43543> nil nil ...)

No trace if I enable toggle-debug-on-error.

Thanks for testing!
I don't think that's an error but rather just a request object being returned that doesn't need to be and it outputs to the minibuffer. I've commit a change to fix that, something I should have done a long time ago haha.
Do hass--available-entities and hass--available-services populate after hass-setup is called?

Ah that's an important thing to handle! Good catch, I'll be sure to fix that soon

Should work now.
I also added the default service for a scene as scene.turn_on so you no longer need to specify that :service if that's what you're calling. I'll be adding more before release.

Just tested. it works. thanks!
Btw, the first request takes a while.

Btw, the first request takes a while.

It might be only on my side. I think I saw something similar with the normal HA dashboard.

Any recommended key binding for (hass-dash-open)?

As an Evil and former Doom user, I'm using SPC-a-d for hass-dash-open. SPC-a-c for hass-call-service. Works well for me.

@bbigras Thanks for your suggestion and help! Sorry it took so long to complete this. Been very busy with life lately. Let me know if you find any issues or have any other suggestions. :)

authors: bbigras purplg

repo: purplg/hass

license: MIT


text:
Intentionally empty repo?

Hi, dotenv's Cargo.toml refers to this repo, but it's empty. Is that intentional?

https://github.com/apiraino/rust-dotenv is the most up to date, based on the network graph. Its Cargo.toml points to apiraino/rust-dotenv, so it should be fixed on the next publish.

I've been meaning to write this for a couple of days now... sorry for not having got to it.
Sean Griffin was so kind to point me to the crates.io API to get the latest available version as source code, which currently is 0.13.0. It would be lovely if someone could spawn or update a fork of this repository with it to take over the head of the network.
curl -L crates.io/api/v1/crates/dotenv/0.13.0/download --output dotenv_0.13.0.tar.gz

We could fork it, but we'd also need to get added as owners on crates.io. Otherwise, we wouldn't be able to publish new versions of the crate.
@sgrif, are you looking for new maintainers?

Yes, the repo is intentionally blank.
Since the original author of dotenv (the real user named purplinimal) deleted the repo, it left a chance for a malicious user to create a github user with the same name and then create a repo named dotenv. Since crates.io will point to purplinimal/dotenv (which is how you got here) you won't realise the difference.
This user and repo was created just as a placeholder so that nobody misuses the repo link. It will be deleted once we restore the contents of dotenv in another repo (or it is moved to this repo). I should have left a note here regarding this)

Just curious, what's holding up restoring the contents in another repo? If you do that, update the link in Cargo.toml, and publish a new version, then crates.io will point to the correct repo, won't it?
I know finding time for open source work isn't always easy. It just seems like the time it takes to create a new user and blank repo is about the same time as it takes to restore the repo under an active user's account and update the crates.io metadata.
Thank you for the explanation.

I'm also wondering where the source code lives now. I'm not able to build dotenv 0.10.1. I get this error:
error[E0659]: `error_chain` is ambiguous (derive helper attribute vs any other name)
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/dotenv-0.10.1/src/lib.rs:23:40
   |
23 | #[cfg_attr(not(feature = "backtrace"), error_chain(backtrace = "false"))]
   |                                        ^^^^^^^^^^^ ambiguous name
   |
note: `error_chain` could refer to the derive helper attribute defined here
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/dotenv-0.10.1/src/lib.rs:22:17
   |
22 | #[derive(Debug, error_chain)]
   |                 ^^^^^^^^^^^
note: `error_chain` could also refer to the derive macro imported here
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/dotenv-0.10.1/src/lib.rs:10:1
   |
10 | #[macro_use]
   | ^^^^^^^^^^^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0659`.

But I don't know where to report it since this repo is empty and it hasn't been restored somewhere else. It's been two months since the last activity on this issue. Anyone know what's going on?

@jimmycuadra I've started taking over maintainership of dotenv. You can find the repo at https://github.com/dotenv-rs/dotenv.

@dgriffen great. Would be worth adding in the readme that this (your repo) isn't the official dotenv but a maintained fork from it. Thanks

@Dylan-DPC it will be published as the official dotenv. The dotenv-rs org has permissions to publish crates.

@dgriffen okay that's great. Let me know once there is a new release on crates.io and it reflects your repo, then i'll delete this repo.

@Dylan-DPC the new release is up. The only repo we couldnt get ahold of was dotenv_codegen_impl since there was only one listed owner. We renamed that crate since it wasnt a public dependency.

One thing I've noticed, this account will have to remain around since it is still listed as an owner on the crates. @Dylan-DPC would it be possible for you to add @sgrif as an owner of dotenv_codegen_impl?

They wouldn't be able to.

Oh, for some reason I thought that it was tied to the github username. I'm guessing is some more unique identifier?

@dgriffen crates.io still considers the original name and my account as 2 different accounts ( a good thing) so I can't add you as a co-owner. Either Sean has permissions or you could start maintaining a fork and swap it for that (worst-case)

authors: Dylan-DPC azriel91 dgriffen dguo jimmycuadra lweberk sgrif

repo: purpliminal/rust-dotenv


text:
Xcode does not works pac device ios 14.3 JB

Describe the bug
Xcode does not deploy anymore after JB.
Xcode: 12.4
Ios: 14.3
Iphone 11
When the phone is not JB Xcode work perfectly.
When the phone is JB you get the error:
Errors were encountered while preparing your device for development. Please check the Devices and Simulators Window.
or:
Could not start debugserver on "dev name" when trying to launch "app name".
if you reboot the phone ( so no JB) everything work fine

Same problem

+1

if you reboot the phone ( so no JB) everything work fine

Details
Failed to start remote service "com.apple.debugserver" on device.
Domain: com.apple.dtdevicekit
Code: 811
Recovery Suggestion: Please check your connection to your device.
User Info: {
DVTRadarComponentKey = 261622;
}
The service is invalid.
Domain: com.apple.dt.MobileDeviceErrorDomain
Code: -402653150
User Info: {
DVTRadarComponentKey = 261622;
MobileDeviceErrorCode = "(0xE8000022)";
"com.apple.dtdevicekit.stacktrace" = (
0   DTDeviceKitBase                     0x000000012c236c8f DTDKCreateNSErrorFromAMDErrorCode + 220
1   DTDeviceKitBase                     0x000000012c2465a4 __63-[DTDKRemoteDeviceConnection startFirstServiceOf:unlockKeybag:]_block_invoke + 613
2   DTDeviceKitBase                     0x000000012c245c70 __48-[DTDKRemoteDeviceConnection futureWithSession:]_block_invoke_3 + 22
3   DTDeviceKitBase                     0x000000012c238def __DTDKExecuteInSession_block_invoke_2 + 35
4   DTDeviceKitBase                     0x000000012c2384c9 __DTDKExecuteWithConnection_block_invoke_2 + 218
5   DTDeviceKitBase                     0x000000012c2383c6 __DTDKExecuteWithConnection_block_invoke + 106
6   libdispatch.dylib                   0x00007fff202c37c7 _dispatch_client_callout + 8
7   libdispatch.dylib                   0x00007fff202d0605 _dispatch_lane_barrier_sync_invoke_and_complete + 60
8   DVTFoundation                       0x000000010f646cc3 DVTDispatchBarrierSync + 208
9   DVTFoundation                       0x000000010f61dd76 -[DVTDispatchLock performLockedBlock:] + 60
10  DTDeviceKitBase                     0x000000012c2382c7 DTDKExecuteWithConnection + 226
11  DTDeviceKitBase                     0x000000012c238c93 DTDKExecuteInSession + 239
12  DTDeviceKitBase                     0x000000012c245ac2 __48-[DTDKRemoteDeviceConnection futureWithSession:]_block_invoke_2 + 131
13  DVTFoundation                       0x000000010f64417e DVT_CALLING_CLIENT_BLOCK + 7
14  DVTFoundation                       0x000000010f645da0 __DVTDispatchAsync_block_invoke + 1191
15  libdispatch.dylib                   0x00007fff202c25dd _dispatch_call_block_and_release + 12
16  libdispatch.dylib                   0x00007fff202c37c7 _dispatch_client_callout + 8
17  libdispatch.dylib                   0x00007fff202c95fe _dispatch_lane_serial_drain + 606
18  libdispatch.dylib                   0x00007fff202ca0fe _dispatch_lane_invoke + 426
19  libdispatch.dylib                   0x00007fff202d3c5d _dispatch_workloop_worker_thread + 819
20  libsystem_pthread.dylib             0x00007fff2046b499 _pthread_wqthread + 314
21  libsystem_pthread.dylib             0x00007fff2046a467 start_wqthread + 15
);
}
System Information
macOS Version 11.2.1 (Build 20D74)
Xcode 12.4 (17801) (Build 12D4e)
Timestamp: 2021-03-19T12:11:37+03:00

If I restart the phone everything is fine

+1

Just update u0 to 6.1.1 and rejailbreak,it has already solved.

ok, thank you

fixed on 6.1.1 as mention by @wstclzy2010


text:
API

Developer API for My App

Developer API
Uncover
https://clk.sh/api?api=4f614bbab5372e223605a5fee810d641a66d17b5&url=yourdestinationlink.com&ali


text:
Discussion: Examples for mowing automations with this integration

I think it helps all (specialy new users) if we collect and discuss automations (here for mowing control) and rerequisits for those. For irrigation we discuss this here -> #30

Target:
Notification over HomeAssistant Companion App or as Telegram Messenger message or over Amazon Alexa with the help of Alexa Media Player Integration
Why not with the Gardena App:
It is not possible to get Messages from the status of the Gardena Mowers.
It is not possible to get this notifications over Amazon Alexa from the App
** Requirements **

Installed hass-gardena-smart-system integration
Home Assistant Compagnion iOS or Android App
Gardena Mower
(optional) Telegram Messenger Integration
(optional) Alexa Media Player Integration you can find and install this Integration over HACS

Configuration:
to get the activity Statuscode as sensor for the automation trigger you need the activity attribut as seperat template sensor in the configuration.yaml
  - platform: template
    sensors:
      sileno_activity:
        value_template: "{{ states.vacuum.sileno.attributes.activity }}"
        friendly_name: "Sileno Aktivität"

Automation example:
- alias: "Docked with Autotimer"
  trigger:
    platform: state
    entity_id: sensor.sileno_activity
    to: 'PARKED_AUTOTIMER'
  action:
    - service: notify.notify
      data:
        title: "Mower docked"
        message: "Lawn mower was parked due to autotimer"
    - service: notify.telegram_[TELEGRAM_CHANNEL]
      data_template:
        title: '*Mower docked!*'
        message: "Lawn mower was parked due to autotimer -> https://[HA_URL]/lovelace/terrasse"

- alias: "Lawnmower mows"
  trigger:
    platform: state
    entity_id: sensor.sileno_activity
    to: 'OK_CUTTING'
  action:
    - service: notify.notify
      data:
        title: "Mower Status"
        message: "Lawnmower mows"
    - service: notify.telegram_[TELEGRAM_CHANNEL]
      data_template:
        title: '*Mower Status*'
        message: "Lawnmower mows -> https://[HA_URL]/lovelace/terrasse"

- alias: "Lawnmower loads"
  trigger:
    platform: state
    entity_id: sensor.sileno_activity
    to: 'OK_CHARGING'
  action:
    - service: notify.notify
      data:
        title: "Mower Status"
        message: "Lawnmower loads"
    - service: notify.telegram_[TELEGRAM_CHANNEL]
      data_template:
        title: '*Lawnmower loads*'
        message: "Lawnmower loads -> https://[HA_URL]/lovelace/terrasse"

- alias: "Mower Error"
  trigger:
    - platform: state
      entity_id: vacuum.sileno
      to: 'error'
  action:
    - service: notify.notify
      data:
        title: "Mower Fehler"
        message: "Error: {{ states.sensor.sileno_error_status.state }}!"
    - service: notify.telegram_[TELEGRAM_CHANNEL]
      data_template:
        title: '*Mower Error*'
        message: "Error: {{ states.sensor.sileno_error_status.state }}! -> https://[HA_URL]/lovelace/terrasse"

This has been added to a RECIPE.md
Thanks @northpower25


text:
Implement GeoTIFF exporter

Some users have expressed their interest to read the pysteps output in GeoTIFF format. This is because GeoTIFF is the standard format for GIS applications (e.g. QGIS). Being able to easily import pysteps output into GIS applications would be especially helpful for visualization purposes.

@pulkkins You might not want to pull in wradlib as a dependency, but maybe your users (eg. me) already have it installed. Then you might get some inspiration from this wradlib-notebook. This uses GDAL as another big dependency (which QGIS normally also have installed). It can output to all available GDAL raster formats.

PR #118 closes this issue.

Does somebody know that, how can I import fmi_geotiff dataset? when I use "fmi" it uses the pgm. but how can use geotiff dataset?

authors: Afshinshafei aperezhortal kmuehlbauer pulkkins

repo: pySTEPS/pysteps


text:
Implementing containers

Alternative to #1203
This run the whole GA in the MAPDL container as suggested by @koubaa
I think this could be an alternative to the mentioned PR limitations. For Windows, we still need to do #1221

Installing Python using GitHub Actions seems to uncorrectly link the executables.
I'm going with installing manually Python 3.8

I couldn't make XVFB run... assuming it pipes plots to it.
We have a marker for skip_no_xserver which should skip tests that do plots, but then the whole test_post is not using it. And CICD passes. So I guess xvfb should simulate a display (that is what is mean to be).
But in the docker image I just keep getting Core dumped error every time we run a plot function in the CICD.
As a temporal fix, I'm just adding the pytest marker to all the functions that do plots.

Running this locally I see an error when running .ci/display_test.py
(venv) [root@fd92f49248e6 pymapdl]# xvfb-run python .ci/display_test.py
system_supports_plotting True
2022-07-04 12:50:23.809 (   0.687s) [        F7CCF740]vtkXOpenGLRenderWindow.:251    ERR| vtkXOpenGLRenderWindow (0x21351b0): Could not find a decent config

ERROR:root:Could not find a decent config
2022-07-04 12:50:23.810 (   0.688s) [        F7CCF740]vtkXOpenGLRenderWindow.:468    ERR| vtkXOpenGLRenderWindow (0x21351b0): Could not find a decent visual

ERROR:root:Could not find a decent visual
/usr/bin/xvfb-run: line 181:  1321 Aborted                 DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1

https://github.com/pyvista/pyvista/issues/2155
I feel like this probably a Pyvista/vtk issue. I'm still investigating.

Running this locally I see an error when running .ci/display_test.py
(venv) [root@fd92f49248e6 pymapdl]# xvfb-run python .ci/display_test.py
system_supports_plotting True
2022-07-04 12:50:23.809 (   0.687s) [        F7CCF740]vtkXOpenGLRenderWindow.:251    ERR| vtkXOpenGLRenderWindow (0x21351b0): Could not find a decent config

ERROR:root:Could not find a decent config
2022-07-04 12:50:23.810 (   0.688s) [        F7CCF740]vtkXOpenGLRenderWindow.:468    ERR| vtkXOpenGLRenderWindow (0x21351b0): Could not find a decent visual

ERROR:root:Could not find a decent visual
/usr/bin/xvfb-run: line 181:  1321 Aborted                 DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1

pyvista/pyvista#2155
I feel like this probably a Pyvista/vtk issue because it has been recorded by pyvista error file:
ERROR: In ../Rendering/OpenGL2/vtkXOpenGLRenderWindow.cxx, line 251
vtkXOpenGLRenderWindow (0x3867750): Could not find a decent config

ERROR: In ../Rendering/OpenGL2/vtkXOpenGLRenderWindow.cxx, line 468
vtkXOpenGLRenderWindow (0x3867750): Could not find a decent visual

I'm still investigating

Soo........
Discussion
So it seems it is a CentOS issue.
I tried two images:
centos/python-36-centos7 and ubuntu:latest.
I had to install xvfb and pyvista in both (python also in ubuntu).
Outputs
CentOS7
(app-root) xvfb-run python -c "import pyvista;from pyvista.plotting import system_supports_plotting;print("system_supports_plotting", system_supports_plotting());pyvista.OFF_SCREEN = True;pyvista.plot(pyvista.Sphere())"
<function system_supports_plotting at 0x7f0bbee4f730> False
ERROR:root:Could not find a decent config
ERROR:root:Could not find a decent visual
/usr/bin/xvfb-run: line 181:   430 Aborted                 DISPLAY=:$SERVERNUM XAUTHORITY=$AUTHFILE "$@" 2>&1

Ubuntu
root@18378420f4f8:/# xvfb-run python3 -c "import pyvista;from pyvista.plotting import system_supports_plotting;print("system_supports_plotting", system_supports_plotting());pyvista.OFF_SCREEN = True;pyvista.plot(pyvista.Sphere())"
<function system_supports_plotting at 0x7f7dc6f16170> False

Code to replicate
CentOS

Pull image: docker run -i -t  -u root centos/python-36-centos7 /bin/bash
Install pyvista: pip install pyvista
Installing xvfb: yum -y install mesa-libGL xorg-x11-server-Xvfb
Test:

xvfb-run python -c "import pyvista;from pyvista.plotting import system_supports_plotting;print("system_supports_plotting", system_supports_plotting());pyvista.OFF_SCREEN = True;pyvista.plot(pyvista.Sphere())"

Ubuntu

Pull image: docker run -it -u root ubuntu
Update: apt-get update -y
Install python:

apt-get install -y python3
apt-get install -y python3-pip

Install pyvista: pip3 install pyvista
Install xvfb: apt install xvfb.
Test:

xvfb-run python3 -c "import pyvista;from pyvista.plotting import system_supports_plotting;print("system_supports_plotting", system_supports_plotting());pyvista.OFF_SCREEN = True;pyvista.plot(pyvista.Sphere())"

Conclusion
After trying to do my best to fix the issue on the CentOS image, I think this requires quite a few changes in xvfb and maybe CentOS. I am not ready to identify, and implement those changes.
Hence, I'm recommending to CLOSE THIS ISSUE without implement it.
Future
A lot of work has been devoted to this PR, and the situation is that it partially works. You could implement this PR in only the unit tests side, and leave the docs building using the old implementation.
But this will just hide the fact that we are using an OS which is quite old and at the end of its life support.
This PR could be reused in the future, once the MAPDL docker image has been moved to ubuntu. But that will take some time.
So we can close this issue for now, and then reopen once the docker image has been migrated.
FYI: Pinging @akaszynski  and @koubaa

@akaszynski @germa89 Agree that we can try this again with a more modern docker image.  In fact - a lot of the work in this PR has to do with setting up the machine to have both mapdl and pymapdl installed on it.  Maybe it is worth it to generate such a docker image internally so that we can pick a container for CI that already has pymapdl + mapdl.

repo: pyansys/pymapdl


text:
Alternate digest algorithms

Needed SHA-512 digests.  Implemented SHA-1, SHA-256, and SHA-512 instead of hardcoding SHA-256 (The spec offers CRC32C and ADLER and a couple others, but they're in zlib and would require a bit more code -- not sure there'd be many users for the extra work).  Also, the current spec calls for the tokens ('sha-256') to be lowercase with a dash.  Hashlib, unfortunately, returns 'sha256' for .name, so used a dict (known_digests) to apply the correct tokens for the spec.
In addition, I separated out a method to return the request body instead of using it directly for the digest.  The spec does not specify that ONLY the body may be used for the digest (In my case, I need one of the headers included as well), so this allows a developer to override the content that is, er, digested.

Thank you for your contribution! This ended up being implemented a bit differently, but the library now supports SHA-512 digests via subclassing:
class MySigner(HTTPSignatureAuth):
    def add_digest(self, request):
        super().add_digest(request, algorithm="sha-512")

authors: jimduchek kislyuk

repo: pyauth/requests-http-signature


text:
Change StrConstrained name

For https://github.com/pydantic/pydantic-core/issues/386

Right, that was silly :smile:
I think this is a simple way of doing what the issue says.

Codecov Report

Merging #517 (10e2467) into main (1276e37) will increase coverage by 0.00%.
The diff coverage is 100.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #517   +/-   ##
=======================================
  Coverage   94.80%   94.80%
=======================================
  Files          93       93
  Lines       12259    12266    +7
  Branches       25       25
=======================================
+ Hits        11622    11629    +7
  Misses        630      630
  Partials        7        7

Impacted Files
Coverage Δ

src/validators/string.rs
100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1276e37...10e2467. Read the comment docs.

honestly, maybe this isn't worth the effort.

I was also thinking it will a few checks to just change the name dynamically.
So you're thinking to just change constrained-str to str or just close this?

Thanks so much for looking into this.
Let's leave this for now, there' are more impactful things to work on.

repo: pydantic/pydantic-core


text:
Expand validate_call docstring

Expands docstring and moves page in API docs to preserve alpha ordering
Change Summary

Related issue number

Checklist

[ ] Unit tests for the changes exist
[ ] Tests pass on CI and coverage remains at 100%
[x] Documentation reflects the changes where applicable
[ ] changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)
[ ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

please review

repo: pydantic/pydantic


text:
helper function - detect index columns in dataframe

Has anyone seen any method how to detect index column(s) in dataframe based on uniqueness of rows?

if u r asking questions on an issue (not recommended) pls post a complete example otherwise ask on the mailing list (much better)

authors: denfromufa jreback

repo: pydata/pandas


text:
CF conventions for time doesn't support years

CF conventions code supports: {'microseconds': 'us', 'milliseconds': 'ms', 'seconds': 's', 'minutes': 'm', 'hours': 'h', 'days': 'D'}, but not 'years'. See example file https://www.dropbox.com/s/34dcpliko928yaj/histsoc_population_0.5deg_1861-2005.nc4?dl=0

I am not sure to understand what you are asking us to do here. The problem with "years" is that their use is not recommended by the CF conventions.
Very often (and I think your file means it this way), users would like years to be simple "calendar years" , i.e. : 1901-01-01, 1902-01-01, but this is not what the unit "years" means in the CF conventions: see http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#time-coordinate

I think I do mean 'years' in the CF convention sense, in this case the time dimension is:
double time(time=145);
      :standard_name = "time";
      :units = "years since 1860-1-1 12:00:00";
      :calendar = "proleptic_gregorian";

This is correctly interpreted by the NASA Panoply NetCDF file viewer. From glancing at the xarray code, it seems it depends on the pandas Timedelta object which in turn doesn't support years as delta objects (although date ranges can be generated at year intervals so it should be possible to implement).

I think I do mean 'years' in the CF convention sense

Can you pinpoint to which part of the CF convention? From the link I read:  a year is exactly 365.242198781 days, which would lead to highly unlikely calendar dates.
I agree however that interpreting "years" as being "calendar years" is the only way that makes sense.
For the record, netCDF4 also doesn't like "years":
import netCDF4
ds = netCDF4.Dataset('/home/mowglie/Downloads/histsoc_population_0.5deg_1861-2005.nc4')
time = ds.variables['time']
netCDF4.num2date(time[:], units=time.units)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-b38f64c7bce4> in <module>()
      2 ds = netCDF4.Dataset('histsoc_population_0.5deg_1861-2005.nc4')
      3 time = ds.variables['time']
----> 4 netCDF4.num2date(time[:], units=time.units)

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.num2date (netCDF4/_netCDF4.c:66463)()

ValueError: unsupported time units

Although I'm not a specialist of CF conventions, this issue may be related to this one: https://github.com/Unidata/netcdftime/issues/5. The forthcoming NetCDFTimeIndex (#1252) uses the netcdftime package. It's rather about using common_year with noleap calendar, though.

I would think that this sort of feature belongs in netcdftime, not xarray. There are obvious issues with defining what a year (or a month) is but I image we can sort those out.

I actually have a similar issues with respect to 'months'. I want to write out my xarray dataarray as a netcdf file, with months as time intervals (one value per month, doesn't matter what day of the month is used as a reference). As with the 'years' described above, this does not seem to work in the  current framework?

In order to construct a netcdf file with a 2D field on a monthly resolution (for X number of years), I currently use the lines of code mentioned below. Since I do not care about the type of calendar, I just use 360_day, in which each month of the year has 30 days. Perhaps this can be useful for others. In case a better solution is available, please let me know!
import numpy as np
import pandas as pd
import xarray as xr

# 51 years, saving first day of each month.
mmhours = np.arange(0,(51*360*24),30*24)
attrs = {'units': 'Hours since 1955-01-01T12:00:00', 'calendar' : '360_day'}

target = np.random.rand(len(mmhours),10,10)
lat = np.arange(50,51,0.1)
lon = np.arange(3,4,0.1)

target_xr = xr.Dataset({'test': (['time', 'lat', 'lon'],  target)},
                       coords={'time': ('time', mmhours, attrs) ,'lat': lat, 'lon': lon})

target_xr.to_netcdf('test.nc', encoding={'test': {'zlib': True}})

Hi Matthias, I think your solution is fine. The best is simply to avoid "months" as units altogether.
I one has a "real" calendar one can also let pandas and xarray do the job:
t = pd.date_range(start='1980-01', end='2010-12', freq='MS')
target = np.random.rand(len(t), 10, 10)
lat = np.arange(50, 51, 0.1)
lon = np.arange(3, 4, 0.1)
target_xr = xr.Dataset({'test': (['time', 'lat', 'lon'],  target)},
                       coords={'time': ('time', t),
                               'lat': lat, 'lon': lon}
                      )
target_xr.to_netcdf('test_2.nc')

which creates the following time units automatically:
	int64 time(time) ;
		time:units = "days since 1980-01-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;

Month unit support in cftime is being discussed in in https://github.com/Unidata/cftime/pull/69
Perhaps xarray folks would like to weigh in.

I have run into this problem multiple times. The latest example I found were some [CORE ocean model runs] (https://rda.ucar.edu/datasets/ds262.0/index.html#!description).
The time dimension of some (they mix units) of these files is given as
netcdf MRI-A_sigma0_monthly {
dimensions:
	level = 51 ;
	latitude = 368 ;
	longitude = 364 ;
	time = UNLIMITED ; // (720 currently)
	time_bnds = 2 ;
variables:
	double latitude(latitude) ;
		latitude:units = "degrees_north                                                   " ;
		latitude:axis = "Y" ;
	double longitude(longitude) ;
		longitude:units = "degrees_east                                                    " ;
		longitude:axis = "X" ;
	double level(level) ;
		level:units = "m                                                               " ;
		level:axis = "Z" ;
	double time(time) ;
		time:units = "years since 1948-1-1 00:00:00                                   " ;
		time:axis = "T" ;
		time:bounds = "time_bnds" ;
		time:calendar = "noleap" ;
	double time_bnds(time, time_bnds) ;
	float sigma0(time, level, latitude, longitude) ;
		sigma0:units = "kg/m^3          " ;
		sigma0:long_name = "Monthly-mean potential density (sigma-0)                                                                                " ;
		sigma0:missing_value = -9.99e+33f ;
}

I understand that 'fully' supporting to decode this unit is hard and should probably addressed upstream.
But I think it might be useful to have a utility function that converts a dataset with these units into someting quickly useable with xarray?
E.g. one could load the dataset with ds = xr.open_dataset(..., decode_times=False) and then maybe call xr.decode_funky_units(ds, units='calendaryears', ...), which could default to the first day of a year (or the first day of a month for units of months since.
This way the user is aware that something is not decoded exactly, but can work with the data.
Is this something that people could see useful here? Id be happy to give an implementation a shot if there is interest.

authors: benbovy fmaussion jbusecke jhamman mangecoeur matthiasdemuzere rabernat

repo: pydata/xarray


text:
arithmetics should ignore nans created by alignment

Can anybody tell me if there is anybody who benefits from this behaviour? I can't think of any good use cases.
wallet = xarray.DataArray([50, 70], dims=['currency'], coords={'currency': ['EUR', 'USD']})
restaurant_bill = xarray.DataArray([30], dims=['currency'], coords={'currency': ['USD']})
with xarray.set_options(arithmetic_join="outer"):
    print(wallet - restaurant_bill)

<xarray.DataArray (currency: 2)>
array([ nan,  40.])
Coordinates:
  * currency  (currency) object 'EUR' 'USD'

While it is fairly clear why it can be desirable to have nan + not nan = nan as a default in arithmetic when the nan is already present in one of the input arrays, when the nan is introduced as part of an automatic align things become much less intuitive.
Proposal:

add a parameter to xarray.align, fillvalue=numpy.nan, which determines what will appear in the newly created array elements
change __add__, __sub__ etc. to invoke xarray.align(fillvalue=0)
change __mul__, __truediv__ etc. to invoke xarray.align(fillvalue=1)

In theory the setting could be left as an opt-in as set_options(arithmetic_align_fillvalue='neutral'), yet I wonder who would actually want the current behaviour?

Related pure-numpy thread
https://stackoverflow.com/questions/42209838/treat-nan-as-zero-in-numpy-array-summation-except-for-nan-in-all-arrays

This behavior is consistent with the default behavior on pandas, which always does an outer join for arithmetic:
In [6]: wallet.to_series() - restaurant_bill.to_series()
currency
EUR     NaN
USD    40.0
dtype: float64

So I don't think we want to change this in an inconsistent way in xarray.
I do agree that we should have support for an explicit fill value in alignment/reindexing and arithmetic. For consistency with pandas (and elsewhere in xarray), let's call it fill_value.

change __add__, __sub__ etc. to invoke xarray.align(fillvalue=0)
change __mul__, __truediv__ etc. to invoke xarray.align(fillvalue=1)

I can see the logic in using an identity value instead of NaN as a default in arithmetic. One peril of this approach is that it isn't always evident what the right identity is. In fact, according to NumPy:
In [16]: import numpy as np

In [17]: print(np.add.identity)
0

In [18]: print(np.multiply.identity)
1

In [19]: print(np.subtract.identity)
None

In [20]: print(np.divide.identity)
None

Let me give a couple other examples of why I don't think we should use an identity of some sort as the default:

Suppose we are comparing a model to observations. model - observations gives the residuals -- unless the variables are mis-aligned, in which case it would give a residual equal to one of the variables.
Suppose we want to do a weighted average. (observations * weights).sum() gives a sensible answer -- but not if a missing weight defaults to 1 (all the weights put together are supposed to sum to 1!).

Closed by #2876

authors: crusaderky shoyer


text:
Add metadata_only param to .to_zarr?

Is your feature request related to a problem?
A leaf from https://github.com/pydata/xarray/issues/8245, which has a bullet:

compute=False is arguably a less-than-obvious kwarg meaning "write metadata". Maybe this should be a method, maybe it's a candidate for renaming? Or maybe make_template can be an abstraction over it

I've also noticed that for large arrays, running compute=False can take several minutes, despite the indexes being very small. I think this is because it's building a dask task graph — which is then discarded, since the array is written from different machines with the region pattern.
Describe the solution you'd like
Would introducing a metadata_only parameter to to_zarr help here:

Better name
No dask graph

Describe alternatives you've considered
No response
Additional context
No response

Yes, is a great idea!

+1, this is a really nice idea.
Related to this could also be a write-through cache of sorts. For high-latency stores (e.g. S3), synchronously populating the store metadata can really add up. If we knew we were only writing metadata, we could safely populate all the Zarr json objects then send them in one bulk write step.
The combination of these two features would be a lightning fast Zarr initialization routine 🚀

I came across https://github.com/pydata/xarray/issues/8343 recently — this seems to be a similar suggestion to what I was intending. Is that correct?

The challenge is that Xarray needs some way to represent the "schema" for the desired entire dataset. I'm very open to alternatives, but so far, the most convenient way to do this has been to load Dask arrays into an xarray.Dataset.

Is anyone more familiar with whether there is a cost to producing the dask task graph? I'm seeing .to_zarr(compute=False) take well over 1 minutes with large arrays with lots of chunks. And it's only writing very small metadata.

If to_zarr(compute=False) is slow, that's more likely due to Xarray doing lots of sequential blocking IO operations. Once you've made the Dataset object the dask graphs have already been created.

This is an area I don't understand well, so obv I defer.
I had thought that it was relevant to driven by dask since that's most of the time is spent in dask/array/optimization.py. But maybe we're saying that this is because xarray is writing indexes during that time? (Though the indexes are fairly small, and it's spending lots of time in a fuse function?)

  %Own   %Total  OwnTime  TotalTime  Function (filename)
  0.00%   0.00%   14.07s    16.42s   fuse (dask/optimization.py)
  0.00%   0.00%    8.15s     8.15s   reverse_dict (dask/core.py)
  0.00%   0.00%    3.17s     4.17s   make_blockwise_graph (dask/blockwise.py)
  0.00%   0.00%    2.81s     2.94s   _cull_dependencies (dask/blockwise.py)
 20.00%  25.00%    2.50s     3.04s   fuse_slice (dask/array/optimization.py)
  0.00%   0.00%    2.49s     2.49s   keys_in_tasks (dask/core.py)
  0.00%   0.00%    2.19s     2.90s   functions_of (dask/optimization.py)
  0.00%   0.00%    1.32s     9.20s   cull (dask/highlevelgraph.py)
  0.00%   0.00%   0.930s    48.16s   optimize (dask/array/optimization.py)

Max I think you're right. In recent times, dask has a "lazy graph" (HighLevelGraph) that gets lowered down to an old-style graph expressed as dicts. That lowering is still slow and potentially whats happening here.

Yeah here's the call to optimize in dask.array.store
https://github.com/dask/dask/blob/1203b1bb6d52b1fb00d54656af4af8e6e35aa615/dask/array/core.py#L1169-L1171

I really think we're just taking advantage of a side-effect by using compute=False for this use-case. I like the idea of a separate method, say initialize_zarr, for this.

Yes we could either do:

.initialize_zarr / .to_zarr_metadata / xarray_beam.make_template](https://xarray-beam.readthedocs.io/en/latest/_autosummary/xarray_beam.make_template.html)
or .to_zarr(metadata_only=True)

How difficult do we think this is? Is it something I can bang out in 45 mins or is it a bigger effort that requires more context?

FYI — for the moment, xr.ones_like(ds).to_zarr(compute=False) saves a few minutes each go! (though the data needs to be floats)

FYI — for the moment, xr.ones_like(ds).to_zarr(compute=False) saves a few minutes each go! (though the data needs to be floats)

In xarray-beam we use zeros_like, which I believe works for any NumPy dtype.

How difficult do we think this is? Is it something I can bang out in 45 mins or is it a bigger effort that requires more context?

In xarray-beam we use zeros_like, which I believe works for any NumPy dtype.

Could this just be running xr.zero_like(ds).to_zarr(compute=False)?? Are there any data types that zarr supports which wouldn't be covered by zeros_like? Possibly strings...

Possibly strings...

Seem to work!
ds = xr.Dataset(
    {
        "a": (("x", "y"), np.arange(20).reshape(4, 5)),
        "b": ("x", ["aa", "b", "c", "d"]),
        "c": ("x", np.array(["aa", "b", "c", "d"], dtype="S")),
        "time": ("t", pd.date_range("2001-01-01", periods=4, freq='D')),
    },
    coords={"x": np.arange(4), "y": [33, 44, 22, 11, 55]},
)

I'm playing around with this piece of code. Does it make sense? There's a fair bit of complexity around the write_empty_chunks=False optimization.
def make_template(ds, *, encoding=None):
    fillvalues = {
        name: var.encoding["_FillValue"]
        for name, var in ds._variables.items()
        if var.encoding and "_FillValue" in var.encoding
    }
    fillvalues.update(
        {
            name: enc["_FillValue"]
            for name, enc in encoding.items()
            if "_FillValue" in enc
        }
    )

    to_drop = {var for var, varenc in encoding.items() if "_FillValue" in varenc}
    dropped = ds.drop_vars(to_drop)
    template = xr.zeros_like(ds)
    for var in to_drop:
        template[var] = xr.full_like(ds[var], encoding[var]["_FillValue"])
    return template

def initialize_zarr(ds, repo, *, region_dims=None, append_dim=None, **kwargs):
    if "compute" in kwargs:
        raise ValueError("The ``compute`` kwarg is not supported in `initialize_zarr`.")

    if kwargs.get("mode", "w") != "w":
        raise ValueError(
            f"Only mode='w' is allowed for initialize_zarr. Received {kwargs['mode']}"
        )

    encoding = kwargs.get("encoding", {})

    template = make_template(ds, encoding=encoding)

    # TODO: handle `write_empty_chunks` in init_kwargs["encoding"]
    init_kwargs = kwargs.copy()
    init_kwargs.pop("write_empty_chunks", None)

    template.to_zarr(
        repo.store, group="foo/", compute=False, write_empty_chunks=False, **init_kwargs
    )

    if region_dims:
        # At this point, the store has been initialized (and potentially overwritten)
        kwargs.pop("mode")
        dropped = ds.drop_dims(region_dims)
        new_encoding = kwargs.pop("encoding", None)
        if new_encoding:
            new_encoding = {k: v for k, v in new_encoding.items() if k in dropped}

        dropped.to_zarr(
            repo.store,
            group="foo/",
            **kwargs,
            encoding=new_encoding,
            compute=True,
            mode="a",
        )

        # can't use drop_dims since that will also remove any variable
        # with the dims to be dropped
        # even if they have anything in region_dims
        dims_to_drop = set(ds.dims) - set(region_dims)
        vars_to_drop = [
            name
            for name, var in ds._variables.items()
            if set(var.dims).issubset(dims_to_drop)
        ]
        return ds.drop_vars(vars_to_drop)

    elif append_dim:
        # TODO
        pass

    else:
        return ds

encoding = {"a": {"_FillValue": -1}}
initialized = initialize_zarr(ds, store, region_dims="y", mode="w", encoding=encoding)

n00b question — why do we need the code around _FillValue?
If there's a dataset with two dims, and they're both region_dims, does this drop all the vars?
(very minor point — we also want to allow w-, indeed that should be the default, given that Zarr will wipe everything, including a whole bucket, for any path it's given)

why do we need the code around _FillValue?

🤦🏾♂ I was testing with a numpy dataset, so compute=False made no difference :/

If there's a dataset with two dims, and they're both region_dims, does this drop all the vars?

Yes. But the indexes for the dims get written during template.to_zarr. After the drop, we are looking to write non-dim coordinate variables and data variables without region dims. So if there was a big dask array that had no dimensions in common with region_dims it would get written at this point. Perhaps initialize_ isn't a good prefix for this function. Thoughts?

we also want to allow w-, indeed that should be the default, given that Zarr will wipe everything,

:+1:

If there's a dataset with two dims, and they're both region_dims, does this drop all the vars?

Yes. But the indexes for the dims get written during template.to_zarr. After the drop, we are looking to write non-dim coordinate variables and data variables without region dims. So if there was a big dask array that had no dimensions in common with region_dims it would get written at this point.

I think the code is doing the right thing — dims_to_drop = set(ds.dims) - set(region_dims) means that we'll drop anything with no dims in common, as you replied.
A clearer way to ask my initial question was "If there's a dataset with one array with two dims, and both dims are region_dims, does this drop the array?" — it doesn't drop anything (dims_to_drop is empty).

Perhaps initialize_ isn't a good prefix for this function.

I like it! make_template etc seems fine too if anyone has strong preferences.

I see there's a logic branch for append_dim; I haven't used region with append_dim; is that a commonly used pattern?

authors: dcherian jhamman max-sixty shoyer


text:
Interactive cells in test_coordinates

Description
The test functions in test_coordinates.py are separated using interactive cells in the current develop branch. Is this a leftover or intended?

That's a leftover - I think they could be deleted and committed without a pull-request...

That's a leftover - I think they could be deleted and committed without a pull-request...

authors: f-brinkmann sikersten

repo: pyfar/pyfar


text:
[patch] Expose StaticIO.for_node as a shortcut to for-nodes

So you don't need to type body_node_class=...
from pyiron_workflow import Workflow

loop1 = Workflow.create.standard.Add.for_node(
    iter_on="other",
    obj=1,
    other=[1, 2],
    output_as_dataframe=False,
)
loop2 = Workflow.create.standard.Multiply.for_node(
    zip_on=("obj", "other"),
    obj=loop1.outputs.add,
    other=[1, 2],
    output_as_dataframe=False,
)
out = loop2()
out.mul
>>> [2, 6]

Pull Request Test Coverage Report for Build 10754356381
Details

0 of 0   changed or added relevant lines in 0 files are covered.
6 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.01%) to 91.404%

Files with Coverage Reduction
New Missed Lines
%

nodes/static_io.py
6
86.96%

Totals

Change from base Build 10754157007:
0.01%

Covered Lines:
2924

Relevant Lines:
3199

💛  - Coveralls

authors: coveralls liamhuber


text:
Initial structure of the sponsor page

@cmaureir It's so great to see this coming together. Thank you 💯

authors: cmaureir willingc

repo: pyladies/global-conference


text:
WIP: Add Full Rank Approximation

This PR adds Full Rank ADVI interface. It is ready to review. Just to experiment with the API, I have created a gist inspired from @ColCarroll's notebook. I look forward to include mixture comparisons as well.

[X] Add Full Rank ADVI
[X] Add docstrings
[X] Add tests
[X] Change variable flattening to model flattening
[ ] Rerun quick start notebook

Just testing this now. Not sure what to make of this:

Should be 10K loss values, correct?

Hi @fonnesbeck
Thanks for checking this out. The implementation of Full Rank ADVI(especially the model flattening part) is wrong. I am working on this! Soon, this will be resolved.

Mean field ADVI does not run at all. Fails with:
ValueError: Dimensions must be equal, but are 4 and 1086 for '{{node monte_carlo_variational_loss/expectation/JointDistributionSequential/log_prob/add_1}} = AddV2[T=DT_FLOAT](monte_carlo_variational_loss/expectation/JointDistributionSequential/log_prob/add, monte_carlo_variational_loss/expectation/JointDistributionSequential/log_prob/Normal_1/log_prob/sub)' with input shapes: [1,4], [1,1086].

Seems like a shape issue. Can you share a reproducible code snippet?
I suspect sort of adjusting sample_size parameter to fit function.

Is this supposed to be working now? I'm getting the following:

It was running with the original PR, though not getting good results. Now it quits after a handful of iterations with NaNs.

There are some concerns about transformations not being stable

Same behavior for mean-field ADVI

Yes. Some transformations were missing for Bounded Distributions. Highlighting the same, I had opened an issue #283 some time back. Commit d61431b adds the remaining transformations. Before this commit, passing validate_args=True to each distribution while doing VI, leads to various constraint errors from TFP.
But I do not feel like its a good approach to manually add transformations to respective distributions. I tried to make it more general with a single class -
class Interval(BackwardTransform):
    name = "interval"

    def __init__(self, lower_limit, upper_limit):
        transform = tf.cond(
            tf.math.is_inf(lower_limit),
            lambda: tf.cond(
                tf.math.is_inf(upper_limit),
                lambda: tfb.Identity(),
                lambda: tfb.Chain([tfb.Shift(upper_limit), tfb.Scale(-1), tfb.Exp()]),  # upper - exp(x)
            ),
            lambda: tf.cond(
                tf.math.is_inf(upper_limit),
                lambda: tfb.Chain([tfb.Shift(lower_limit), tfb.Exp()]),  # exp(x) + lower
                lambda: tfb.Sigmoid(low=lower_limit, high=upper_limit),  # interval
            ),
        )
        super().__init__(transform)

But this leads to many eager execution issues for lambda functions. Please suggest if there is a way to improve existing approach.

@fonnesbeck, what is park bias model?

@ferrine its a relatively large model that I was using to test ADVI, ported over from PyMC3. Its got a hierarchical component as well as Gaussian proccesses. Samples very slowly, so a good candidate for ADVI.

Hi @fonnesbeck
The model seems interesting. Can you experiment again in your free time with the current state of PR? Issues related to transformations has been resolved.
Thanks

With the current PR, I am getting the following failure:
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-29-6a1ed8631a36> in <module>()
      1 m = park_bias_model()
      2 # trace = pm.sample(m, num_samples=100, burn_in=2000, xla=True)
----> 3 approx = pm.fit(m, num_steps=20000, method='fullrank_advi')

6 frames
/usr/local/lib/python3.6/dist-packages/pymc4/variational/approximations.py in fit(model, method, num_steps, sample_size, random_seed, optimizer, **kwargs)
    246         return losses
    247
--> 248     return ADVIFit(inference, run_approximation())

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781
    782       new_tracing_count = self._get_tracing_count()

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    844               *args, **kwds)
    845       # If we did not create any variables the trace we have is good enough.
--> 846       return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
    847
    848     def fn_with_cond(*inner_args, **inner_kwds):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _filtered_call(self, args, kwargs, cancellation_manager)
   1845                            resource_variable_ops.BaseResourceVariable))],
   1846         captured_inputs=self.captured_inputs,
-> 1847         cancellation_manager=cancellation_manager)
   1848
   1849   def _call_flat(self, args, captured_inputs, cancellation_manager=None):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1921       # No tape is watching; skip to running the function.
   1922       return self._build_call_outputs(self._inference_function.call(
-> 1923           ctx, args, cancellation_manager=cancellation_manager))
   1924     forward_backward = self._select_forward_and_backward_functions(
   1925         args,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    548               inputs=args,
    549               attrs=attrs,
--> 550               ctx=ctx)
    551         else:
    552           outputs = execute.execute_with_cancellation(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Cholesky decomposition was not successful. The input might not be valid.
	 [[{{node fit_surrogate_posterior/while/body/_22/fit_surrogate_posterior/while/StatefulPartitionedCall/monte_carlo_variational_loss/expectation/loop_body/PartitionedCall/pfor/PartitionedCall/Cholesky/pfor/Cholesky}}]]
	 [[fit_surrogate_posterior/while/body/_22/fit_surrogate_posterior/while/StatefulPartitionedCall/Adam/Adam/AssignAddVariableOp/_51]]
  (1) Invalid argument:  Cholesky decomposition was not successful. The input might not be valid.
	 [[{{node fit_surrogate_posterior/while/body/_22/fit_surrogate_posterior/while/StatefulPartitionedCall/monte_carlo_variational_loss/expectation/loop_body/PartitionedCall/pfor/PartitionedCall/Cholesky/pfor/Cholesky}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_run_approximation_207085]

Function call stack:
run_approximation -> run_approximation

Can you rebase to current master so that all the covariance functions are available?

and I also leave this experiment by @Sayam753

This is a major feature, congrats @Sayam753!
@ferrine should we open an issue to keep track of the issue?

In another PR I would add full rank to the example NB, which also still reads "Full Rank ADVI - Coming Soon".

authors: Sayam753 ferrine fonnesbeck twiecki


text:
Consolidate ModelBuilder InferenceData getters (and setters)

Think the ModelBuilder class itself could just get the getters (and setters) for:

prior
prior_predictive
fit_result (posterior)
posterior_predictive

They are defined here for mmm:
https://github.com/pymc-labs/pymc-marketing/blob/664b5caef1f20785a8f73cac3c4f83a13ceeeee7/pymc_marketing/mmm/base.py#L266-L290)
and here for clv:
https://github.com/pymc-labs/pymc-marketing/blob/664b5caef1f20785a8f73cac3c4f83a13ceeeee7/pymc_marketing/clv/models/basic.py#L242-L256
Would the clv modules benefit from this? @ColtAllen
I'm leaning more toward standardization of code and having informative error messages

Would the clv modules benefit from this? @ColtAllen I'm leaning more toward standardization of code and having informative error messages

CLVModel overwrites quite a few of the inherited ModelBuilder methods because the latter is built around an "X,Y" variable convention that doesn't apply to the CLV models. That said, the idata methods could benefit from some cleanup, particularly if it gets rid of the following warning that pops up during testing:
~/site-packages/arviz/data/inference_data.py:1538: UserWarning: The group fit_data is not defined in the InferenceData scheme
This warning is probably extraneous from the ModelBuilder class, but also worth mentioning:
~/site-packages/pymc/model/core.py:518: FutureWarning: All coords are now mutable by default. coords_mutable will be removed in a future release.

This warning is probably extraneous from the ModelBuilder class, but also worth mentioning:
~/site-packages/pymc/model/core.py:518: FutureWarning: All coords are now mutable by default. coords_mutable will be removed in a future release.

Yeah, I think we should get rid of these. Have this in mind #669

CLVModel overwrites quite a few of the inherited ModelBuilder methods because the latter is built around an "X,Y" variable convention that doesn't apply to the CLV models. That said, the idata methods could benefit from some cleanup, particularly if it gets rid of the following warning that pops up during testing:

Maybe there can be two versions of the ModelBuilder? There can be some base class that will handle this common logic of getters and setters. Then two child classes with the X at initialize and X,y in the methods.
Just an initial thought...

repo: pymc-labs/pymc-marketing


text:
pipenv --py crashes

Issue description
Running pipenv --py in a project with no venv setup crashes.
Expected result
No stack trace, just an error message.
Actual result
$ pipenv --py
Traceback (most recent call last):
  File "/home/david/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1114, in invoke
    return Command.invoke(self, ctx)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/cli/command.py", line 140, in cli
    do_py()
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/core.py", line 1635, in do_py
    click.echo(which("python", allow_global=system))
  File "/home/david/.local/lib/python3.6/site-packages/pipenv/core.py", line 108, in which
    raise RuntimeError("location not created nor specified")
RuntimeError: location not created nor specified

Steps to replicate
Using Ubuntu 18.04:
sudo apt update
sudo apt install python3-pip
pip3 install pipenv  # --user is implied so pipenv executable gets installed in /home/$USER/.local/bin/
git clone https://github.com/dlech/vscode-python-issue4866.git
cd vscode-python-issue4866
pipenv --py

Please run $ pipenv --support, and paste the results here.
$ pipenv --support
Pipenv version: '2018.11.26'
Pipenv location: '/home/david/.local/lib/python3.6/site-packages/pipenv'
Python location: '/usr/bin/python3'
Python installations found:

3.6.7: /usr/bin/python3
3.6.7: /usr/bin/python3.6m
2.7.15rc1: /usr/bin/python2
2.7.1: /usr/bin/jython

PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.6.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.18.0-16-generic',
 'platform_system': 'Linux',
 'platform_version': '#17~18.04.1-Ubuntu SMP Tue Feb 12 13:35:51 UTC 2019',
 'python_full_version': '3.6.7',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

CLUTTER_IM_MODULE
NVM_DIR
LS_COLORS
LESSCLOSE
XDG_MENU_PREFIX
LANG
DISPLAY
GTK2_MODULES
DEBFULLNAME
GTK_CSD
NVM_CD_FLAGS
USERNAME
CHROME_DESKTOP
NO_AT_BRIDGE
XDG_VTNR
GIO_LAUNCHED_DESKTOP_FILE_PID
SSH_AUTH_SOCK
PRU_C_DIR
MANDATORY_PATH
XDG_SESSION_ID
USER
DESKTOP_SESSION
RBENV_SHELL
QT4_IM_MODULE
GOPATH
TEXTDOMAINDIR
DROPBOX_USE_LIBAPPINDICATOR
DEFAULTS_PATH
QT_QPA_PLATFORMTHEME
PWD
HOME
TEXTDOMAIN
SSH_AGENT_PID
TERM_PROGRAM
TERM_PROGRAM_VERSION
QT_ACCESSIBILITY
LIBVIRT_DEFAULT_URI
XDG_SESSION_TYPE
XDG_DATA_DIRS
GSETTINGS_SCHEMA_DIR
XDG_SESSION_DESKTOP
GTK_MODULES
WINDOWPATH
TERM
SHELL
QT_IM_MODULE
XMODIFIERS
IM_CONFIG_PHASE
NVM_BIN
XDG_CURRENT_DESKTOP
GPG_AGENT_INFO
GIO_LAUNCHED_DESKTOP_FILE
XDG_SEAT
SHLVL
DEBEMAIL
GDMSESSION
GNOME_DESKTOP_SESSION_ID
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_CONFIG_DIRS
PATH
RBENV_VERSION
SESSION_MANAGER
LESSOPEN
GTK_IM_MODULE
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:
Debug–specific environment variables:

PATH: /home/david/.nvm/versions/node/v10.15.3/bin:/home/david/work/gocode/bin:/home/david/.rbenv/shims:/home/david/bin:/home/david/work/gocode/bin:/home/david/.rbenv/shims:/home/david/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/david/.dotnet/tools:/home/david/.dotnet/tools
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/david/work/junk/vscode-python-issue4866

Contents of Pipfile ('/home/david/work/junk/vscode-python-issue4866/Pipfile'):
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/home/david/work/junk/vscode-python-issue4866/Pipfile.lock'):
{
    "_meta": {
        "hash": {
            "sha256": "415dfdcb118dd9bdfef17671cb7dcd78dbd69b6ae7d4f39e8b44e71d60ca72e7"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

Yes the message RuntimeError: location not created nor specified tells what is happening, do you want to improve the message? Welcome

It seems that most errors result in a stack trace being printed, so I guess this is normal for pipenv.

@frostming
maybe pipenv --venv and pipenv --py should have the same style:

It makes sense to me. Feel free to shoot a PR.

authors: Cologler dlech frostming

repo: pypa/pipenv


text:
Add setuptools.command.build

Summary of changes
In order to override distutils.command.build on downstream projects
it is good to have a setuptools specific command which allows
downstream projects to avoid importing distutils.
Closes
Pull Request Checklist

[x] Changes have tests
[x] News fragment added in changelog.d/.
(See documentation for details)

Thank you very much @isuruf, sorry for the confusion with the extra commits, I was trying to fix the error with flake8, I hope that is OK with you.
I am OK with this change, but I will let the other maintainers have a look because I don't know if there are other implications.

Ping on this

Thank you very much @isuruf and sorry for the delay.
Since no one has found any problem with this implementation, I believe we can go ahead...
Should we add any protection/migration warnings for the case an user was previously relying on distutils.command.build to setup subcommands?
For example, how about something like:
_ORIGINAL_SUBCOMMANDS = {"build_py", "build_clib", "build_ext", "build_scripts"}

class build(_build):
    def run(self):
       subcommands = {cmd[0] for cmd in _build.sub_commands}
       if subcommands - _ORIGINAL_SUBCOMMANDS:
           msg = """
           It seems that you are using `distutils.command.build.build` to add new subcommands.
           Using `distutils` directly is considered deprecated, please use `setuptools.command.build`.
           """
           warning.warns(msg, SetuptoolsDeprecationWarning)
           self.subcommands = _build.sub_commands
       super().run()
  ...

What do you think?

Thank you very much @isuruf and sorry for the delay.
Since no one has found any problem with this implementation, I believe we can go ahead...
Should we add any protection/migration warnings for the case an user was previously relying on distutils.command.build to setup subcommands?
For example, how about something like:
_ORIGINAL_SUBCOMMANDS = {"build_py", "build_clib", "build_ext", "build_scripts"}

class build(_build):
    def run(self):
       subcommands = {cmd[0] for cmd in _build.sub_commands}
       if subcommands - _ORIGINAL_SUBCOMMANDS:
           msg = """
           It seems that you are using `distutils.command.build.build` to add new subcommands.
           Using `distutils` directly is considered deprecated, please use `setuptools.command.build`.
           """
           warning.warns(msg, SetuptoolsDeprecationWarning)
           self.subcommands = _build.sub_commands
       super().run()
  ...

What do you think?

Makes sense to me. There's a small typo in the line before last. self.subcommands should be self.sub_commands. Can you push into this PR or should I?

I can do that later today, but if you would like to go ahead and work on it, that would be great (since I would have to spend some time to create a simple test case).

Hi @isuruf, I tried to fix the tests failing in the CI.
Please feel free to revert my changes and do something different.

authors: abravalheri isuruf

repo: pypa/setuptools


text:
Update dependency management docs

Summary of changes

Extract depencency_liks section to a new file:
deprecated/dependency_links.rst
Add note about directly URLs not being accepted in PyPI.
Simplify intro about build system requirement.
Simplify intro about optional dependencies.
Fix confusion in example about "Project" and "Package".
"Demote" section about extras in entry-points to a note.

Pull Request Checklist

[ ] Changes have tests
[ ] News fragment added in changelog.d/.
(See documentation for details)

@abravalheri thanks for this PR.  Now that dependency_links is deprecated, is there an alternative way to specify a different non-pypi package index url [eg in artifactory/jfrog] to setuptools.setup()?
I've tried the install_requires argument to setuptools.setup() like so:
install_requires=[mypackage @ http://:@:8082/api/pypi/pypi/simple]
but install_requires seems to expect to download source code as a [.zip] archive.
Ideally, I'm looking for something equivalent to pip's extra-index-url setting, which is specifiable directly to setuptools.setup().

authors: abravalheri dkondoetsy


text:
"beta" badge for WebAuthn beta feature within Account Settings

What's the problem this feature will solve?
The new WebAuthn two-factor auth method is going to -- initially -- be in beta, and we want to alert users to that when letting them use it.
Describe the solution you'd like
For the WebAuthn rollout, @ewdurbin suggested that we initially add a badge in the Account Settings marking the WebAuthn 2FA method as "beta". This badge should be on the same line as the WebAuthn entry (and clearly NOT apply to the TOTP method) and should probably link to an FAQ entry or GitHub issue or wiki page about the beta.
Additional context
Because we don't have #5869 yet.

Fixed in #5977! Thank you @nlhkabu!

repo: pypa/warehouse