text:
The results of evaluate.py about entity

After running evaluate.py, I got the result about entity as follow. It seems different from the result in your paper.
Could you tell me what's wrong with it? Thanks！
Info (entity)
Number of unique entities: 652
Number of malicious entities: 11
Result (entity)
TP: 11
TN: 641
FP: 0
FN: 0

It seems evaluate.py for S1 does not show the right numbers, the evaluate.py shows smaller number than the actual numbers.
The reason for this failure is that if you read the generated evaluation file:
output/eval_seq_graph_testing_preprocessed_logs_S1-CVE-2015-5122_windows.dot.txt.json
You will see all the entities lumped together, thus, many entities can be grouped as one entity like this "Entity1;Entity2; ...":
c:/users/aalsahee/appdata/roaming/microsoft/windows/recent/customdestinations/atd2u3m1o6uzd4q4dugw.temp**;**c:/users/aalsahee/appdata/roaming/mozilla/firefox/profiles/a6gl280b.default/sessionstore.js.tmp**;**c:/users/aalsahee/a....
evaluate.py still counts these lumped entites as one entity! that's way these numbers generated by evaluate.py are smaller than the actual numbers.
During writing the paper I must have implemented another script to consider the numbers of these lumped entities. I don't have it anymore, but evaluate.py can be easily fixed to consider the numbers of these lumped entities. I hope this helps. Thanks.

authors: cssaheel zlsfe

repo: purseclab/ATLAS


text:
How can I stream Bitstamp's websocket?

I just couldn't figure out how to hook myself to bitstamp's websocket. Please help and it will be very very much appreciated! Thank You
https://www.bitstamp.net/websocket/

Pusher libraries are either "http" or "websocket". This is a http library, so you can use it to trigger events and authenticate private channels, but you can't use it to subscribe to websockets. For a list of our websocket libraries see here.

source: gharchive/issue

repo: pusher/pusher-http-python

license: mit

license_type: permissive

license_source: bigquery


text:
⚠ Jurnal has degraded performance

In 85484be, Jurnal (https://jurnal.unmer.ac.id) experienced degraded performance:

HTTP code: 200
Response time: 17686 ms

Resolved: Jurnal performance has improved in 90cc2d0.

repo: pusimgit/upptime

license: MIT

license_source: github-api


text:
Extension not working in Edge browser and local GitLab Instance

I have specified the local instance URL as described but the extension does not work in Edge Browser Version 127.0.2651.74

Again sorry for the late response :sweat_smile:
I guess it probably has something to do with the trailing slash, can you try removing the URL from the instance input, save it and re-add it without the trailing slash?

thanks to the suggestions. tried it but it does not work. root cause bight be the same as for this addin https://gitlab.com/potato-oss/gitlab-details-sidebar/-/issues/57. however the following addin works fine with custom/lokal instance URL https://github.com/tavyandy97/span-tree

@xChillout The new version has been accepted

authors: puyt xChillout

repo: puyt/chrome-gitlab-enhancer


text:
Irradiance calculation produces figures below real figures

I have the following script:
import itertools
import pandas as pd
import pvlib
from pvlib import clearsky, atmosphere
from pvlib.location import Location

cs = pvlib.location.get_clearsky_test(50.430104, -5.006048, 'ineichen',
1525953600000000000, 'test')
print("ghi = " + str(cs['ghi'][0]))
print("dni = " + str(cs['dni'][0]))
print("dhi = " + str(cs['dhi'][0]))

# Parameters: Latitude, Longitude, Tilt, Azimuth, ghi, dni, dhi, time (Epoch)
result = pvlib.irradiance.globalinplane_test_output(50.430104, -5.006048,
24, 180, cs['ghi'][0], cs['dni'][0], cs['dhi'][0], 1525953600000000000, .25,
'grass')
print (result)

I have some wrapper methods here but the underlying methods will be called. This produces the following results:
ghi = 861.5440345805347
dni = 893.789882747542
dhi = 111.15085948276032
Solar Position azimuth =  172.79194271695087
Solar Position apparent_zenith =  32.90596711532058
AOI =  9.530542460995187
Airmass =  1.19028427148373
Gr-grass =  7.448439523040739
Time =  2018-05-10 12:00:00
Diff Perez =  125.92483957453679
Global in plane =                        poa_global  poa_direct  poa_diffuse
2018-05-10 12:00:00  1014.826609   881.45333   133.373279
poa_global  poa_direct  poa_diffuse
2018-05-10 12:00:00  1014.826609   881.45333   133.373279

The irradiance figure for the 10th May is about 10% lower than that measured by a Pyranometer at that location. From what I have read this model is generally lower than measured data.
Is there anything that I can try to improve the "theoretical" figures being generated as these seem lower than the real figures?

@Spanner1969 thanks for the question. There are some alternate model choices which might close the gap between the model result and your measurements, but keep in mind, that irradiance models generally are 'average' models, over conditions and locations, and are not precise for any location or specific time. We default to the Ineichen clearsky model both for its simplicity and its performance relative to other simple models (see the Reno et al reference for the comparison).  Including altitude may help a bit, although the site you are modeling is already near sea level so this effect will be minor.
At some point in your modeling process you are transposing from horizontal to plane-of-array (probably with  get_total_irradiance) and that transposition defaults to the isotropic model, which is known to be biased towards low values.  You can select among other models for the transposition: I'd recommend haydavies for transposing modeled irradiance (and perez for transposing measurements).
Over a long period of time, a bias in modeled irradiance as compared to measurement could result from several reasons, including model bias and biased calibration of the irradiance instrument. Instrument issues might be identified and corrected by deploying redundant sensors. I've seen people address model bias by applying an empirical scalar to the clear sky irradiance.
Please note that we have a google group pvlib-python for pvlib use questions, and we prefer to keep this forum for issues and discussion of the code itself.
@wholmgren we should add a link to the google group to the home page here.  We provide only a link to this Issues page currently.

@cwhanse are you suggesting that we replace the README.md's Contact section with the documentation's Getting support section? I am in favor of that.

@wholmgren yes

authors: Spanner1969 cwhanse wholmgren

repo: pvlib/pvlib-python


text:
Added Code for Issues(#20, #114, #124) Resolved merge conflicts of previous PR

I'm closing previous PR's because of Merge Conflicts with the added support for polymon.
for Issues resolved with this PR, refer: Polymon #124 , Base 64 #20, papersscissors.rock #114
for previous PR's closed, refer: #140 , #144
(PS I'll probably stop contributing to this project due to inactivity of the maintainers so I was just wrapping up all the local branches to my master branch, thought I should ease the task by creating a single PR).

Hey cool game! Could we chat? tim@wickeyappstore.com We are making a store for progressive web apps and would love to have your game!

source: gharchive/pull-request


text:
🛑 kosilum.com-http is down

In b6317b8, kosilum.com-http (http://www.kosilum.com) was down:

HTTP code: 403
Response time: 31 ms

Resolved: kosilum.com-http is back up in 813e4d4.

repo: pwmonitor/monikosi


text:
Iphone xs Jailbreak power off problem

when i am to jaibreak status with uncover 3.7.0 and i decide to power off my iphone, when i do this the screen goes black and/ BUT the side button its not working. So we going in 2 results .                                                         1  results) 50%  of tries the iphone stay to black screen ( maybe its power off) and dont open again with the side button.Only if i do a hard reset can open it.( force restart).                                                 2 results) 50% of tries the iphone opens alone after about 2 minuites without press the side bottum . This happend only in jailbreak status        BUT if the iphone it is NOT in jailbreak status , the iphone power off and power on normaly......
Steps to reproduce the behavior:

Jailbreak
Power off iphone
Black screen
It opens alone after 2 minuites or if its not open alone , i must hard reset( force restart) it.

Device (please complete the following information):

iOS Version: 12.2
iOS Device: iphone xs
unc0ver Version: 3.70

Place an "x" between the brackets if true:

[ ] this is a bug others will be able to reproduce
[ ] this issue is present with all tweaks uninstalled(except for default packages) or disabled
[ ] this issue is present after a rootfs restore
[ x] this issue is present on the latest version of unc0ver

Logs
If applicable, add logs or error messages here.

I put my phone in the frezzer for 2 hours straight, the battery was completly drained, then a pluged the phone, then I was fonctionnal

Same here iphone xs max ios 12.4

Can confirm. Same phone/ios, same problem.

Same problem, XR, iOS 12.2

Hi
Same for me
When a i turn off my phone for using camera
« Camera don’t working with JB »
My iphone stay black until battery off.
And i can’t use my phone xs
iPhone xs 12.4 Unc0ver
Thx

authors: ImVxping Santacruz78 activoss kkraku mzxnbcv realHyperMatrix


text:
Ability to toggle the logging window that's seen on Beta versions in the Release Version.

I think it'd be pretty neat to be able to see the logging window in release versions of Unc0ver.
Make it a toggle or something like that, perhaps? However you want to execute the idea or not at all is fine with me. I just like to watch the text scroll as it's actually jailbreaking. I'd make it opt-in though. If the user wants to see the log window, then they have to go and switch it on.
Call it "Developer Mode", "Verbose Mode", or whatever.

Just use the Beta version while it isn't merged with the Master Version? There's no point in using current release versions. Since B29, all of these betas could already be latest release version, but they're not because they chose to only make them release version whenever A12 support is added.

@clarityzzz I was meaning this be a feature in the final version of 3.0.0. I'm using Beta 50 as of this posting. I was just saying that I enjoy seeing the logging window as do many other users of the betas, I'm sure. So I thought it'd be a good idea.

3.0.0 adds this feature. Thanks Unc0ver team!


text:
[Cluster.NotALeader] No write operations are allowed directly on this database. Writes must pass through the leader

Hi,
I am getting this exception when running a write query against a neo4j causal cluster version 4.3.6, this work fine with causal cluster of version 4.1.1. In both cases the query is sent to a follower server, py2neo is 2021.2.3.
Graph is initialized using:
Graph('neo4j://<dns_address>:7687', **settings)
settings:
{'user': 'neo4j', 'password': password, 'name': <db_name>, 'routing': True, 'routing_refresh_ttl': 300}
Any idea what can cause this behavior?
Thanks

We have the same issue with dbms.routing.default_router=SERVER (which looks to be mandatory for exposing Neo4j Browser using LoadBalancer in Kubernetes).

How are you running the query?

Thank for answering, @technige!
This is how we run the query:
self.graph.create(user)
We have a Model
class User(GraphObject):
    __primarykey__ = "email"

    createdAt = Property()
    name = Property()
    deactivatedEmail = Property()

We have a Graph:
self.graph = Graph(
    'neo4j://' + NEO4J_URL + ':7687',
    user='neo4j', password='password', name='neo4j', routing=True, routing_refresh_ttl=300
)

We have this error in both 4.3.9 and 4.4.1 with dbms.routing.default_router=SERVER
Failed to execute update after 3 tries

This is complete example of failing code. This code is failing with the latest py2neo and with the neo4j 4.3.9 or 4.4.1 deployed using community based helm chart with default configuration (including default_router=SERVER).
root@backend-568c7588b-thhpt:/usr/src/app# python
Python 3.7.12 (default, Dec 21 2021, 11:35:10)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from py2neo import Graph
>>> graph = Graph('neo4j://neo4j-host:7687', user='neo4j', password='password', routing=True)
>>> from models import User
>>> user = User('andrey', 'test-user@gmail.coom')
>>> graph.create(user)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 591, in create
    self.update(lambda tx: tx.create(subgraph))
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 445, in update
    self._update(cypher, timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 490, in _update
    raise WriteServiceUnavailable("Failed to execute update after %r tries" % n)
py2neo.errors.WriteServiceUnavailable: Failed to execute update after 3 tries

@kostia-sukernik and I talking about another issue, the error we see after running this is different:
    graph = Graph(f"neo4j://neo4j:7687", name=database, password=password)
    graph.run("create (x)")

Internal server error: [Cluster.NotALeader] No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER
But with the official neo4j python driver and the same cluster configuration we are not getting this error on write queries.

@kostia-sukernik and I talking about another issue, the error we see after running write queries is different:
graph = Graph(f"neo4j://neo4j:7687", name=database, password=password)
graph.run("create (x)")

Internal server error: [Cluster.NotALeader] No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER
But with the official neo4j python driver and the same cluster configuration we are not getting this error.

I can only confirm that when I run query your way I have the same issue:
>>> graph = Graph('neo4j://neo4j:7687', user='neo4j', password=password)
>>> graph.run("create (x)")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 405, in run
    return self.auto().run(cypher, parameters, **kwparameters)
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 980, in run
    readonly=self.readonly)
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/__init__.py", line 1331, in auto_run
    cx.pull(result, n=pull)
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 950, in pull
    self._audit(self._transaction)
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 759, in _audit
    task.audit()
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 1101, in audit
    item.audit()
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 1101, in audit
    item.audit()
  File "/usr/local/lib/python3.7/site-packages/py2neo/client/bolt.py", line 1264, in audit
    raise self._failure
py2neo.errors.ClientError: [Cluster.NotALeader] No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER

I can confirm that when setting up a cluster through docker using the official image I get the `
My conf file:

# Setting that specifies how much memory Neo4j is allowed to use for the page cache.
dbms.memory.pagecache.size=100M

# Setting that specifies the initial JVM heap size.
dbms.memory.heap.initial_size=100M

# Strategy that the instance will use to determine the addresses of other members.
causal_clustering.discovery_type=DNS

# The network addresses of an initial set of Core cluster members that are available to bootstrap this Core or Read Replica instance.
# If the DNS strategy is used, the addresses are fetch using the DNS A records.
causal_clustering.initial_discovery_members=neo4j-network:5000

# Address (the public hostname/IP address of the machine)
# and port setting that specifies where this instance advertises for discovery protocol messages from other members of the cluster.
causal_clustering.discovery_advertised_address=$(hostname -i)

# Address (the public hostname/IP address of the machine)
# and port setting that specifies where this instance advertises for Raft messages within the Core cluster.
causal_clustering.raft_advertised_address=$(hostname)

 # Address (the public hostname/IP address of the machine)
 # and port setting that specifies where this instance advertises for requests for transactions in the transaction-shipping catchup protocol.
causal_clustering.transaction_advertised_address=$(hostname)

# Enable server side routing
dbms.routing.enabled=true

# Use server side routing for neo4j:// protocol connections.
dbms.routing.default_router=SERVER

# The advertised address for the intra-cluster routing connector.
dbms.routing.advertised_address=0.0.0.0

The Docker compose file:
version: '3.8'

# Custom top-level network
networks:
  neo4j-internal:

services:

  core1:
    # Docker image to be used
    image: ${NEO4J_DOCKER_IMAGE}

    # Hostname
    hostname: core1

    # Service-level network, which specifies the networks, from the list of the top-level networks (in this case only neo4j-internal), that the server will connect to.
    # Adds a network alias (used in neo4j.conf when configuring the discovery members)
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network

    # The ports that will be accessible from outside the container - HTTP (7474) and Bolt (7687).
    ports:
      - "7474:7474"
      - "7687:7687"
      - "7688:7688"

    # Uncomment the volumes to be mounted to make them accessible from outside the container.
    volumes:
      - ./conf/:/conf/
      - ./core1/data:/var/lib/neo4j/data
      - ./core1/logs:/var/lib/neo4j/logs
      - ./core1/conf:/var/lib/neo4j/conf
      - ./core1/import:/var/lib/neo4j/import
      #- ./core1/metrics:/var/lib/neo4j/metrics
      #- ./core1/licenses:/var/lib/neo4j/licenses
      #- ./core1/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins

    # Passes the following environment variables to the container
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=CORE

    # Simple check testing whether the port 7474 is opened.
    # If so, the instance running inside the container is considered as "healthy".
    # This status can be checked using the "docker ps" command.
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]

    # Set up the user
    user: ${USER_ID}:${GROUP_ID}

  core2:
    image: ${NEO4J_DOCKER_IMAGE}
    hostname: core2
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network
    ports:
      - "7574:7474"
      - "7787:7687"
      - "7788:7688"
    volumes:
      - ./conf/:/conf/
      - ./core2/data:/var/lib/neo4j/data
      - ./core2/logs:/var/lib/neo4j/logs
      - ./core2/conf:/var/lib/neo4j/conf
      - ./core2/import:/var/lib/neo4j/import
      #- ./core2/metrics:/var/lib/neo4j/metrics
      #- ./core2/licenses:/var/lib/neo4j/licenses
      #- ./core2/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=CORE
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
    user: ${USER_ID}:${GROUP_ID}

  core3:
    image: ${NEO4J_DOCKER_IMAGE}
    hostname: core3
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network
    ports:
      - "7674:7474"
      - "7887:7687"
      - "7888:7688"
    volumes:
      - ./conf/:/conf/
      - ./core3/data:/var/lib/neo4j/data
      - ./core3/logs:/var/lib/neo4j/logs
      - ./core3/conf:/var/lib/neo4j/conf
      - ./core3/import:/var/lib/neo4j/import
      #- ./core3/metrics:/var/lib/neo4j/metrics
      #- ./core3/licenses:/var/lib/neo4j/licenses
      #- ./core3/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=CORE
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
    user: ${USER_ID}:${GROUP_ID}

  readreplica1:
    image: ${NEO4J_DOCKER_IMAGE}
    hostname: replica1
    networks:
      neo4j-internal:
        aliases:
          - neo4j-network
    ports:
      - "7477:7474"
      - "7690:7687"
      - "7988:7688"
    volumes:
      - ./conf/:/conf/
      - ./replica1/data:/var/lib/neo4j/data
      - ./replica1/logs:/var/lib/neo4j/logs
      - ./replica1/conf:/var/lib/neo4j/conf
      - ./replica1/import:/var/lib/neo4j/import
      #- ./replica1/metrics:/var/lib/neo4j/metrics
      #- ./replica1/licenses:/var/lib/neo4j/licenses
      #- ./replica1/ssl:/var/lib/neo4j/ssl
      #- ./plugins:/var/lib/neo4j/plugins
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT
      - NEO4J_AUTH
      - EXTENDED_CONF
      - NEO4J_EDITION
      - NEO4J_dbms_mode=READ_REPLICA
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider localhost:7474 || exit 1"]
    user: ${USER_ID}:${GROUP_ID}

@technige technige We are also facing the same issue (py2neo.errors.WriteServiceUnavailable) after switching to cluster and enabling routing, could you please give any insight on this?
repo= Repository(ServiceProfile(profile=neo4j://neo4j:password@localhost:7687, routing=True, protocol='neo4j'))
repo.save(node)

Neo4j's official driver is not giving any error.
Thank you for this wonderful library.

authors: RazLightlytics SimonThordal abronin iamyoudi kostia-sukernik technige


text:
Bug in TLStretch?

Hello!
I was going through the new transfer learning estimators and I noticed that in TLStretch the dispersions are computed as the sum of the squared distance to the mean:
https://github.com/pyRiemann/pyRiemann/blob/master/pyriemann/transfer/_estimators.py#L289-L294
Isn't there a 1/N term missing? The way it is defined will not work when there are different numbers of points in the source and target.

I am on it

Thanks!

Thanks @apmellot ! Good catch.

When all weights are equal to 1 / n, arg min S2 / n and arg min S2 are the same (S2 denoting the sum of squares).

Closed by #258

authors: apmellot plcrodrigues qbarthelemy sylvchev

repo: pyRiemann/pyRiemann


text:
No longer in pip

Can't find module in pip

jeah not sure what happend. I did not remove it on purpose. Will add it again once I find a bit of time

authors: DerThorsten matteoturini


text:
Issue 3114 check coord

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please add a line in the relevant section of CHANGELOG.md to document the change (include PR #) - note reverse order of PR #s. If necessary, also add to the list of breaking changes.

[ ] New feature (non-breaking change which adds functionality)
[ ] Optimization (back-end change that speeds up the code)
[ ] Bug fix (non-breaking change which fixes an issue)

Key checklist:

[ ] No style issues: $ pre-commit run (or $ nox -s pre-commit) (see CONTRIBUTING.md for how to set this up to run automatically when committing locally, in just two lines of code)
[ ] All tests pass: $ python run-tests.py --all (or $ nox -s tests)
[ ] The documentation builds: $ python run-tests.py --doctest (or $ nox -s doctests)

You can run integration tests, unit tests, and doctests together at once, using $ python run-tests.py --quick (or $ nox -s quick).
Further checks:

[ ] Code is commented, particularly in hard-to-understand areas
[ ] Tests added that prove fix is effective or that feature works

sorry I accidentally create another one

I think this was the correct way. @RuiheLi, you may reopen this pill request and cherry-pick your commits from the develop branch (#3378) to this branch so that you don't lose progress

I think this was the correct way, because it is from a feature branch to the develop branch. @RuiheLi, you may reopen this pull request and cherry-pick your commits from the develop branch (#3378, which can be closed afterwards) to this branch so that you don't lose progress

Thank you so much for the kind suggestion. Reopen now.
All the commits before should be alright because I cereated the branch issue_3114_check_coord based on the develop branch.

@all-contributors please add @RuiheLi for code and tests

@all-contributors please add @RuiheLi for code and tests

Please update the PR description so that 3114 gets closed by this change (use the fixes tag in the template)

@RuiheLi Are you still working on this?

@RuiheLi Are you still working on this?

I have added the final else as suggested. But I had a try and still cannot write a proper test for the divergence as it will require to define a spatial variable with incorrect coordinate system first which will trigger an early error.

I have added the final else as suggested. But I had a try and still cannot write a proper test for the divergence as it will require to define a spatial variable with incorrect coordinate system first which will trigger an early error.

@kratman what's the best way to approach that? Should we add a # pragma: no cover tag?

I will take a look at this closer in the afternoon and give my opinion on the tests

I have added the final else as suggested. But I had a try and still cannot write a proper test for the divergence as it will require to define a spatial variable with incorrect coordinate system first which will trigger an early error.

@kratman what's the best way to approach that? Should we add a # pragma: no cover tag?

now that the coordinate system is already checked should the logic in discretisation be changed? i.e. we don't need to throw that error and the final else can just be the Cartesian case? alternatively, if we want to keep the error and have a test you could create a spatial variable with the correct domain, then manually overwrite cood_sys for the submesh later and then create the divergence object. seems clunky though

@RuiheLi Sorry it took so long for me to respond, but I think there is a nice workaround to test this change without ignoring anything lines.
If you are having trouble testing that function, then it might get easier if you create a function for the block in question. If you create a function called something along the lines of "check_coordinate_system()" for the if-elif-else block, then it can take all things it is supposed to check as arguments. That might make it easier to write a test without adding making the test too complicated. I did something similar in this commit: https://github.com/pybamm-team/PyBaMM/pull/3682/commits/fa6bdc22eac87e63a3a200bc826ed1eb97b2ebea
In general short functions are easy to test. So breaking up complicated functions is probably a good thing anyway. Whenever it is difficult to test a function and we consider adding a pragma to ignore it. It is often a sign that we need to break up the function into smaller testable methods.

thanks @kratman, i like this approach as we check coordinates a few times in that file, so will reduce duplication too

@RuiheLi Are you still working on this?

authors: RuiheLi agriyakhetarpal brosaplanella kratman rtimms

repo: pybamm-team/PyBaMM


text:
Added support for the Hitch integration testing framework.

See http://hitchtest.com for details and videos.

@crdoconnor rdoconnor, thanks for this PR! Hitch appeared on my radar recently, and I've been looking for a chance to check it out. I do have some questions:

Does it work on OSX? I see a lot of libraries in requirements.apt that I don't think are in most OSX installations.
Does it work on Windows? cookiecutter-django provides minimal support for Windows and I doubt all the libs work there either.
What about Travis, Jenkins, AppVeyor (Windows), and other CI tools? How does Hitch interact with them?

If the answer is 'no' to any of those questions, that doesn't mean Hitchtest can't be included. Instead, what it means:

The dependency on Linux will need to be specified in the documentation.
Instructions for running on CI tools should also be specified. I've debated adding .travis and .appveyor settings files, and this seems like the perfect time to focus on it.

Yes, it's regularly tested and it runs fine on OS X (as well as Arch, Debian, CentOS and Fedora). Cookie cutter did not seem to have an equivalent of requirements.apt for Mac OS X, though (or for various other linux distributions). What should I do about this?

No, unfortunately it will not work on Windows and it would be very difficult to make it do so :( I will add a note to the documentation if you like.

I haven't done much work on testing integrations with CI tools yet, but that's my next top priority. I'll try make sure I have instructions for jenkins and travis in the next few days.

Any thoughts @luzfcb or @theskumar? We're supposed to dally with the bleeding edge, and Hitch looks promising.

How well does it work in virtual environments a'la Docker/Vagrant?

The tests I did on other linux distributions apart from my own (Debian, Arch, CentOS, Fedora) were done in Vagrant. Works just fine.
I have not tried it in Docker yet. I'll add that to my to do list.

I have also made it work in Travis CI, incidentally. An example of that is here: https://travis-ci.org/hitchtest/django-remindme
I'll update the pull request to include an appropriate config for travis tonight.

@luzfcb, @theskumar Any comments? I like what I see, since end-to-end is a really nice feature. Also, it's totally opt-in: you can use it any time you want. Barring any further comments or objections I want to merge this PR into cookiecutter-django.
There is the Windows issue (it won't work there). But once we get this merged in and I get my Windows laptop running again (used solely for testing), I'll add in logic for windows users to not get quite so many hitch-focused files.
Again, let me know your thoughts.

No, not strictly speaking necessary. Actually I use pipsi to install it, much like I do with cookiecutter.

@pydanny I have tested it in docker too now. It seems to run fine.

Grats and thanks @crdoconnor for giving cookiecutter Django end-to-end testing functionality. :ship:

Thanks for accepting it!

@crdoconnor a good candidate for plugin is Splinter [1][2].
[1] http://splinter.readthedocs.org/en/latest/
[2] https://github.com/cobrateam/splinter

@luzfcb Actually you wouldn't even need a plugin to use splinter, since it can use a selenium remote webdriver (see here https://splinter.readthedocs.org/en/latest/drivers/remote.html). Just do "hitch install splinter", and in a step definition you can do the following (taken from the docs above):
from splinter import Browser

def step_definition(self):
    with Browser(driver_name="remote",
                 url=self.services['Firefox'].logs.json()[0]['uri']) as browser:
        print("Link to job: https://saucelabs.com/jobs/{}".format(
              browser.driver.session_id))
        browser.visit("http://splinter.readthedocs.org")
        browser.find_link_by_text('documentation').first.click()

@crdoconnor a question:
You have any plans to include cache of non-python packages, something like pip currently does.
For example, I already have a project with Hitch. after execute hitch test stub.test, Hitch download the Python, Postgresql and Redis packages.
I start a new project. When running Hitch in the new project, it will download again all non-python packages. This takes a lot of time on slow connections.
I sometimes suffer with bad internet connection.
I suppose several other people may have the same problem

@luzfcb It already does this. The second time you run stub.test and python, postgres & redis are already built, it won't try to re-download or recompile them.

@crdoconnor yes, he will not re-download if I'm in the same project, but if I create a new project in a totally different directory, add Hitch with exactly the same dependencies.
Hitch will download the non-python dependencies again, ( the python dependencies will not be downloaded again because pip already caches automatically)

Got it. I've created an issue: https://github.com/hitchtest/hitchtest/issues/1
Will fix it over the next few days.

authors: crdoconnor luzfcb pydanny

repo: pydanny/cookiecutter-django


text:
Use Griffe's public API

Closes #548.

Thank you so much.

authors: pawamoy samuelcolvin

repo: pydantic/pydantic-ai


text:
BaseModel only matches other BaseModel's (not even unittest.mock.ANY)

Initial Checks

[X] I confirm that I'm using Pydantic V2 installed directly from the main branch, or equivalent

Description
I've encountered some unexpected behavior while using Pydantic 2.0, specifically with the __eq__ method in the BaseModel class. The current implementation of the method causes it to not match anything other than instances of BaseModel. This behavior might cause issues when using custom validation classes, e.g. unittest.mock.ANY.
Here's the code snippet of the __eq__ method in Pydantic's BaseModel:
def __eq__(self, other) -> bool:
    if not isinstance(other, BaseModel):
        return False
    ...

This implementation checks whether the other object is an instance of BaseModel, and immediately returns False if it is not. Due to this, when comparing a BaseModel instance with custom validation classes, like unittest.mock.ANY, the comparison will always return False.
While I understand that this check is in place to ensure that only instances of BaseModel can be compared (and potentially for performance), it would be helpful to have a more flexible implementation that allows compatibility with custom validation classes.
Also I do not want to simply reorder the comparison since then pytest will display the expected and actual results in the opposite order of what I want.
Possible solution:
One potential solution is to update the __eq__ method in BaseModel to delegate the comparison to the other object if it is not an instance of BaseModel:
def __eq__(self, other) -> bool:
    if not isinstance(other, BaseModel):
        return NotImplemented

By returning NotImplemented, we allow Python to delegate the comparison to the other object's __eq__ method. If that method is capable of handling the comparison with a BaseModel instance, it will return the appropriate result. This makes the comparison more flexible and compatible with custom validation classes.
With this change, users can implement their own custom validation classes and handle comparisons with BaseModel instances in the custom class's __eq__ method:
class CustomValidationClass:
    def __eq__(self, other) -> bool:
        if isinstance(other, BaseModel):
            # Implement custom comparison logic here
            ...

Performance wise I don't think this will affect much. In the case of BaseModel's it has no change. For other types it may need to do a few more steps but nothing considerable (I have not performed any benchmarks in this regard).
If this solution or others turn out to have bad effects, performance wise or in some other way, is it possible to add a config option to enable this?
Please let me know if you have any questions or require further information on this issue. Thank you for creating this awesome project :pray:
Side Note:
I orginally discovered this kink with a Custom Validation class for Pydantic V1. The Custom Validation class would check for an instance of BaseModel. However the implementation in V1 first converts the model to a dict so it was no longer an instance of BaseModel and failed the test.
Example Code
from unittest.mock import ANY

from pydantic import BaseModel

class MyModel(BaseModel):
    foo: str

if __name__ == "__main__":
    assert MyModel(foo="bar") == ANY

Python, Pydantic & OS Version
pydantic version: 2.0a3
pydantic-core version: 0.25.0 release build profile
install path: /Users/ulfur/projects/pydantic-test/.venv/lib/python3.10/site-packages/pydantic
python version: 3.10.1 (v3.10.1:2cd268a3a9, Dec  6 2021, 14:28:59) [Clang 13.0.0 (clang-1300.0.29.3)]
platform: macOS-12.5.1-arm64-arm-64bit
optional deps. installed: ['typing-extensions']

I did a little benchmarking of this:
import time
from dataclasses import dataclass

from pydantic import BaseModel

class MyModel(BaseModel):
    a: int

@dataclass
class MyDataclass:
    a: int

m1 = MyModel(a=1)
m2 = MyDataclass(a=1)
t0 = time.time()
for _ in range(10_000_000):
    m1 == m2
t1 = time.time()

print(f"elapsed: {t1-t0:.3f}s")
# with return False:          elapsed: 1.793s
# with return NotImplemented: elapsed: 2.581s, 43% slower

As you can see, it runs significantly slower with return NotImplemented vs. return False. However, I think there are several good arguments for us making this change:

It does improve testability of various things, as you've noted
It's only slower in the case that you are doing an == comparison against something that isn't even a BaseModel; this seems pretty unlikely to be happening in "hot" code
Even though it's noticeably slower, it's still pretty fast, at approximately 4M comparisons/second on my local machine.
This is what dataclasses does: https://github.com/python/cpython/blob/d2e2e53f733f8c8098035bbbc452bd1892796cb3/Lib/dataclasses.py#L669-L671

Given all the above points, I'll open a PR making this change.

authors: UlfurOrn dmontagu

repo: pydantic/pydantic


text:
AnyUrl not serialized as expected

Initial Checks

[X] I confirm that I'm using Pydantic V2

Description
When using AnyUrl fields with model_dump, the output is of type pydantic_core.Url and not str as expected. Trying to jsonify the result leads to an error as Url is not serializable.
This problem does not occur when using model_dump_json.
This seems like a bug, as the documentation states that The .model_dump_json() method serializes a model directly to a JSON-encoded string that is equivalent to the result produced by .model_dump()  and defines serializing as transforming models into a less structured form comprised of Python built-ins.
Example Code
import json
from pydantic import BaseModel, AnyUrl
from pydantic_core import Url

class MyModel(BaseModel):
    url: AnyUrl

my_model = MyModel(url="http://example.com/")

# model_dump does not provide python built-ins
assert isinstance(my_model.model_dump()["url"], Url)

# trying to write the result leads to an error
try:
    json.dumps(my_model.model_dump())
except TypeError as err:
    assert err.args[0] == 'Object of type Url is not JSON serializable'

# there is no problem when using model_dump_json
assert isinstance(json.loads(my_model.model_dump_json())["url"], str)

# all assertions are true

Python, Pydantic & OS Version
pydantic version: 2.5.3
pydantic-core version: 2.14.6
pydantic-core build: profile=release pgo=true
install path: /usr/local/lib/python3.12/site-packages/pydantic
python version: 3.12.0 (main, Nov 29 2023, 03:23:09) [GCC 12.2.0]
platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.36
related packages: mypy-1.7.1 pydantic-settings-2.1.0 fastapi-0.105.0 typing_extensions-4.9.0 email-validator-2.1.0.post1

You can do:
json.dumps(my_model.model_dumps(mode="json")

See https://github.com/pydantic/pydantic/issues/7632 for more details.

authors: Kludex Leobouloc


text:
Data corruption during Excel export

In some case a DataFrame exported to excel present some bad values.
It's is not a problem of Excel reading (the data inside the sheet1.xml of the .xlsx file is also incorrect).
The same DataFrame exported to ".csv" is correct.
The problem could be "solved" by renaming the column header  as [col-1, col-2,...]. Maybe an encoding problem ?
The issue is that there is no warning/error during the export. It's very easy to miss it.
To reproduce:
import pandas as pd

df = pd.read_pickle('problematic_df.pkl')
df.to_excel('problematic_df.xlsx')
df.to_csv('problematic_df.csv')

with the file available here: https://drive.google.com/file/d/0Bzz_ZaP_wS_HMFdlMkVzaTR0cjA/view?usp=sharing
Note that the content of cell M14 is different in both file (at least when run on my computer)
Using:

Python 3.4.3 |Anaconda 2.3.0 (64-bit)
pandas 0.16.2
Windows 7 64 bits

can you show the dataframe in question above (.info and .dtypes), and a sample .head()

df.info()
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 15 entries, 2015-08-11 00:04:59.999000 to 2015-08-11 01:14:59.999000
Freq: 5T
Data columns (total 20 columns):
TEMP 1 (A11)_°C     3 non-null float64
TEMP 2 (A12)_°C     3 non-null float64
TEMP 3 (A21)_°C     3 non-null float64
TEMP 4 (A23)_°C     3 non-null float64
TEMP 5 (A14)_°C     3 non-null float64
TEMP 6 (A15)_°C     3 non-null float64
TEMP 7 (A16)_°C     3 non-null float64
TEMP 8 (A25)_°C     3 non-null float64
TEMP 9 (B11)_°C     3 non-null float64
TEMP 10 (B21)_°C    3 non-null float64
Ref #11/ADIT_°C     3 non-null Ref #11/ADIT_°C    float64
Ref #11/ADIT_°C    float64
dtype: object
TEMP 11 (B14)_°C    3 non-null float64
TEMP 12 (B24)_°C    3 non-null float64
TEMP 13 (C11)_°C    3 non-null float64
TEMP 14 (C21)_°C    3 non-null float64
TEMP 15 (C14)_°C    3 non-null float64
TEMP 16_°C          3 non-null float64
TEMP 17 (C24)_°C    3 non-null float64
TEMP 18_°C          3 non-null float64
Ref #11/ADIT_°C     3 non-null Ref #11/ADIT_°C    float64
Ref #11/ADIT_°C    float64
dtype: object
dtypes: float64(20)
memory usage: 2.5 KB
None

df.dtypes
TEMP 1 (A11)_°C     float64
TEMP 2 (A12)_°C     float64
TEMP 3 (A21)_°C     float64
TEMP 4 (A23)_°C     float64
TEMP 5 (A14)_°C     float64
TEMP 6 (A15)_°C     float64
TEMP 7 (A16)_°C     float64
TEMP 8 (A25)_°C     float64
TEMP 9 (B11)_°C     float64
TEMP 10 (B21)_°C    float64
Ref #11/ADIT_°C     float64
TEMP 11 (B14)_°C    float64
TEMP 12 (B24)_°C    float64
TEMP 13 (C11)_°C    float64
TEMP 14 (C21)_°C    float64
TEMP 15 (C14)_°C    float64
TEMP 16_°C          float64
TEMP 17 (C24)_°C    float64
TEMP 18_°C          float64
Ref #11/ADIT_°C     float64
dtype: object

df.head()
                        TEMP 1 (A11)_°C  TEMP 2 (A12)_°C  TEMP 3 (A21)_°C  \
2015-08-11 00:04:59.999              NaN              NaN              NaN
2015-08-11 00:09:59.999              NaN              NaN              NaN
2015-08-11 00:14:59.999              NaN              NaN              NaN
2015-08-11 00:19:59.999              NaN              NaN              NaN
2015-08-11 00:24:59.999              NaN              NaN              NaN
2015-08-11 00:29:59.999              NaN              NaN              NaN
2015-08-11 00:34:59.999              NaN              NaN              NaN
2015-08-11 00:39:59.999              NaN              NaN              NaN
2015-08-11 00:44:59.999              NaN              NaN              NaN
2015-08-11 00:49:59.999              NaN              NaN              NaN
2015-08-11 00:54:59.999              NaN              NaN              NaN
2015-08-11 00:59:59.999              NaN              NaN              NaN
2015-08-11 01:04:59.999        28.743081         5.851268        26.925449
2015-08-11 01:09:59.999        18.642970         3.182361         6.454038
2015-08-11 01:14:59.999        24.646857        24.838391        21.959323

                         TEMP 4 (A23)_°C  TEMP 5 (A14)_°C  TEMP 6 (A15)_°C  \
2015-08-11 00:04:59.999              NaN              NaN              NaN
2015-08-11 00:09:59.999              NaN              NaN              NaN
2015-08-11 00:14:59.999              NaN              NaN              NaN
2015-08-11 00:19:59.999              NaN              NaN              NaN
2015-08-11 00:24:59.999              NaN              NaN              NaN
2015-08-11 00:29:59.999              NaN              NaN              NaN
2015-08-11 00:34:59.999              NaN              NaN              NaN
2015-08-11 00:39:59.999              NaN              NaN              NaN
2015-08-11 00:44:59.999              NaN              NaN              NaN
2015-08-11 00:49:59.999              NaN              NaN              NaN
2015-08-11 00:54:59.999              NaN              NaN              NaN
2015-08-11 00:59:59.999              NaN              NaN              NaN
2015-08-11 01:04:59.999        10.989247        17.369523        10.847034
2015-08-11 01:09:59.999         8.634052         2.338829         5.450807
2015-08-11 01:14:59.999        21.646028         9.444759        25.819109

                         TEMP 7 (A16)_°C  TEMP 8 (A25)_°C  TEMP 9 (B11)_°C  \
2015-08-11 00:04:59.999              NaN              NaN              NaN
2015-08-11 00:09:59.999              NaN              NaN              NaN
2015-08-11 00:14:59.999              NaN              NaN              NaN
2015-08-11 00:19:59.999              NaN              NaN              NaN
2015-08-11 00:24:59.999              NaN              NaN              NaN
2015-08-11 00:29:59.999              NaN              NaN              NaN
2015-08-11 00:34:59.999              NaN              NaN              NaN
2015-08-11 00:39:59.999              NaN              NaN              NaN
2015-08-11 00:44:59.999              NaN              NaN              NaN
2015-08-11 00:49:59.999              NaN              NaN              NaN
2015-08-11 00:54:59.999              NaN              NaN              NaN
2015-08-11 00:59:59.999              NaN              NaN              NaN
2015-08-11 01:04:59.999         7.031353        14.956100         4.171982
2015-08-11 01:09:59.999         0.147043        21.383754        23.385664
2015-08-11 01:14:59.999        21.723265        19.405757        20.091859

                         TEMP 10 (B21)_°C  Ref #11/ADIT_°C  TEMP 11 (B14)_°C  \
2015-08-11 00:04:59.999               NaN              NaN               NaN
2015-08-11 00:09:59.999               NaN              NaN               NaN
2015-08-11 00:14:59.999               NaN              NaN               NaN
2015-08-11 00:19:59.999               NaN              NaN               NaN
2015-08-11 00:24:59.999               NaN              NaN               NaN
2015-08-11 00:29:59.999               NaN              NaN               NaN
2015-08-11 00:34:59.999               NaN              NaN               NaN
2015-08-11 00:39:59.999               NaN              NaN               NaN
2015-08-11 00:44:59.999               NaN              NaN               NaN
2015-08-11 00:49:59.999               NaN              NaN               NaN
2015-08-11 00:54:59.999               NaN              NaN               NaN
2015-08-11 00:59:59.999               NaN              NaN               NaN
2015-08-11 01:04:59.999          0.696912         4.977991         10.410759
2015-08-11 01:09:59.999         10.104453         9.947120         19.196995
2015-08-11 01:14:59.999         14.178148        38.991964         12.298831

                         TEMP 12 (B24)_°C  TEMP 13 (C11)_°C  TEMP 14 (C21)_°C  \
2015-08-11 00:04:59.999               NaN               NaN               NaN
2015-08-11 00:09:59.999               NaN               NaN               NaN
2015-08-11 00:14:59.999               NaN               NaN               NaN
2015-08-11 00:19:59.999               NaN               NaN               NaN
2015-08-11 00:24:59.999               NaN               NaN               NaN
2015-08-11 00:29:59.999               NaN               NaN               NaN
2015-08-11 00:34:59.999               NaN               NaN               NaN
2015-08-11 00:39:59.999               NaN               NaN               NaN
2015-08-11 00:44:59.999               NaN               NaN               NaN
2015-08-11 00:49:59.999               NaN               NaN               NaN
2015-08-11 00:54:59.999               NaN               NaN               NaN
2015-08-11 00:59:59.999               NaN               NaN               NaN
2015-08-11 01:04:59.999         15.923412         16.179728         18.776198
2015-08-11 01:09:59.999          7.768040         12.242002         19.361663
2015-08-11 01:14:59.999         21.629414         27.349045          9.152102

                         TEMP 15 (C14)_°C  TEMP 16_°C  TEMP 17 (C24)_°C  \
2015-08-11 00:04:59.999               NaN         NaN               NaN
2015-08-11 00:09:59.999               NaN         NaN               NaN
2015-08-11 00:14:59.999               NaN         NaN               NaN
2015-08-11 00:19:59.999               NaN         NaN               NaN
2015-08-11 00:24:59.999               NaN         NaN               NaN
2015-08-11 00:29:59.999               NaN         NaN               NaN
2015-08-11 00:34:59.999               NaN         NaN               NaN
2015-08-11 00:39:59.999               NaN         NaN               NaN
2015-08-11 00:44:59.999               NaN         NaN               NaN
2015-08-11 00:49:59.999               NaN         NaN               NaN
2015-08-11 00:54:59.999               NaN         NaN               NaN
2015-08-11 00:59:59.999               NaN         NaN               NaN
2015-08-11 01:04:59.999         11.898102           0         11.688812
2015-08-11 01:09:59.999         26.681062           0         14.318932
2015-08-11 01:14:59.999         13.217890           0         13.049905

                         TEMP 18_°C  Ref #11/ADIT_°C
2015-08-11 00:04:59.999         NaN              NaN
2015-08-11 00:09:59.999         NaN              NaN
2015-08-11 00:14:59.999         NaN              NaN
2015-08-11 00:19:59.999         NaN              NaN
2015-08-11 00:24:59.999         NaN              NaN
2015-08-11 00:29:59.999         NaN              NaN
2015-08-11 00:34:59.999         NaN              NaN
2015-08-11 00:39:59.999         NaN              NaN
2015-08-11 00:44:59.999         NaN              NaN
2015-08-11 00:49:59.999         NaN              NaN
2015-08-11 00:54:59.999         NaN              NaN
2015-08-11 00:59:59.999         NaN              NaN
2015-08-11 01:04:59.999           0        20.809154
2015-08-11 01:09:59.999           0        27.623193
2015-08-11 01:14:59.999           0         4.088537

For information the value incorrectly exported to Excel is TEMP 11 (B14)_°C at time 11/08/2015  01:05:00.
In the .csv file and as printed by ".head()" the value is 10.4107588318426.
In the .xlsx file the value is 20.8091542146332

This might be the old excel-duplicate-column-name problem.  The problem goes away if I rename the frame so that no columns are duplicated, and it looks like the errors come in after the first duplicate (Ref #11/ADIT_°C).

I've just tried to modify the name of the last column (the second Ref #11/ADIT_°C ) and indeed it solved the problem.

authors: bertrandhaut dsm054 jreback

repo: pydata/pandas


text:
Series passed to function in DataFrame.apply on empty dataframe lack index

Code Sample, a copy-pastable example if possible
from StringIO import StringIO
import pandas as pd
from random import random
s = "a,b,c"
df = pd.read_csv(StringIO(s))
f1 = lambda row: random()
new_col1 = df.apply(f1, axis=1)
df["d"] = new_col1
f2 = lambda row: 0 if row["a"] > 10 else 1
new_col2 = df.apply(f2, axis=1)
df["e"] = new_col2
df

Expected Output
Empty DataFrame
Columns: [a, b, c, d, e]
Index: []

The problem seems to be that a Series row is passed to the lambda and that 'a' can't be used to index into it. This is only breaks for an empty DataFrame. If it has values the output is as expected.
According to the docs:

Objects passed to functions are Series objects having index either the DataFrame’s index (axis=0) or the columns (axis=1)

output of pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-21-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: None
pip: 8.1.2
setuptools: 20.7.0
Cython: None
numpy: 1.11.0
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: None

you need to pass reduce=True

In [14]: new_col2 = df.apply(f2, axis=1, reduce=True)

In [15]: new_col2
Out[15]: Series([], dtype: float64)

authors: archevel jreback


text:
BUG: Rolling negative window issue fix #13383

[x] closes #13383
[x] tests added / passed
[x] passes git diff upstream/master | flake8 --diff
[x] whatsnew entry

Added functionality in validate function of Rolling to ensure that window size is non-negative.

[Current coverage][cc-pull] is 84.25%

Merging [#13441][cc-pull] into [master][cc-base-branch] will increase coverage by <.01%

@@             master     #13441   diff @@
==========================================
  Files           138        138
  Lines         50810      50812     +2
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits          42809      42812     +3
+ Misses         8001       8000     -1
  Partials          0          0

Powered by Codecov. Last updated by [07761c5...e000807][cc-compare]
[cc-base-branch]: https://codecov.io/gh/pydata/pandas/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/pydata/pandas/compare/07761c519154a6ed8a9345512476152226499ba6...e0008070703ac745642a55701fc5ade428216ae3
[cc-pull]: https://codecov.io/gh/pydata/pandas/pull/13441?src=pr

thanks!

authors: codecov-io jreback priyankjain


text:
FIX: Search bar and navigation dropdown style

These are a few UX/UI improvements to the search bar and the navigation dropdown styling. Here are the major improvements:

Fixes the overlap of UI items
Makes the dropdown menu items show up as regular links on mobile (no hiding)
Makes the header menu become scrollable if it exceeds 90% of the vertical height (in case there are many menu links)

closes https://github.com/pydata/pydata-sphinx-theme/issues/769

Don't know if this is because of this PR, but the dropdown now disappears on "medium size" screens

Hmm I think that's because of:
https://github.com/pydata/pydata-sphinx-theme/pull/770/files#diff-56f26cf6476156a4e40851251ff7f6c361c38ad8dfb404d87e7738e52cc31721R104-R105
We should make sure that the breakpoint for "collapsing the header" and for "turning the dropdown into a list of links" is the same.

I dug a little more and realized that we were slightly mis-using the media-breakpoint-down mixin, because it actually triggers on the next highest breakpoint, rather than the one given as an argument (that is super confusing and they've change the behavior in BS5, I linked the PR where they discuss this in the top comment). I think that is now fixed

Alrighty we've got two approvals here so I'm gonna merge and we can iterate in subsequent PRs as we wish

authors: choldgraf chrisjsewell

repo: pydata/pydata-sphinx-theme


text:
concat_dim getting added to all variables of multifile datasets

Code Sample
Using the following example data set:
example_jan.nc
#!/usr/bin/env python3

import xarray
ds = xarray.open_mfdataset('example_jan.nc', concat_dim='Time')
print(ds)

The result from xarray 0.10.2 (and all previous various xarray versions we've worked with):
Dimensions:                                                      (Time: 1, nOceanRegions: 7, nOceanRegionsTmp: 7, nVertLevels: 100)
Dimensions without coordinates: Time, nOceanRegions, nOceanRegionsTmp, nVertLevels
Data variables:
    time_avg_avgValueWithinOceanLayerRegion_avgLayerTemperature  (Time, nOceanRegionsTmp, nVertLevels) float64 dask.array<shape=(1, 7, 100), chunksize=(1, 7, 100)>
    time_avg_avgValueWithinOceanRegion_avgSurfaceTemperature     (Time, nOceanRegions) float64 dask.array<shape=(1, 7), chunksize=(1, 7)>
    time_avg_daysSinceStartOfSim                                 (Time) timedelta64[ns] dask.array<shape=(1,), chunksize=(1,)>
    xtime_end                                                    (Time) |S64 dask.array<shape=(1,), chunksize=(1,)>
    xtime_start                                                  (Time) |S64 dask.array<shape=(1,), chunksize=(1,)>
    refBottomDepth                                               (nVertLevels) float64 dask.array<shape=(100,), chunksize=(100,)>
Attributes:
    history:  Tue Dec  6 04:49:14 2016: ncatted -O -a ,global,d,, acme_alaph7...
    NCO:      "4.6.2"

The results with xarray 0.10.3:
<xarray.Dataset>
Dimensions:                                                      (Time: 1, nOceanRegions: 7, nOceanRegionsTmp: 7, nVertLevels: 100)
Dimensions without coordinates: Time, nOceanRegions, nOceanRegionsTmp, nVertLevels
Data variables:
    time_avg_avgValueWithinOceanLayerRegion_avgLayerTemperature  (Time, nOceanRegionsTmp, nVertLevels) float64 dask.array<shape=(1, 7, 100), chunksize=(1, 7, 100)>
    time_avg_avgValueWithinOceanRegion_avgSurfaceTemperature     (Time, nOceanRegions) float64 dask.array<shape=(1, 7), chunksize=(1, 7)>
    time_avg_daysSinceStartOfSim                                 (Time) timedelta64[ns] dask.array<shape=(1,), chunksize=(1,)>
    xtime_end                                                    (Time) |S64 dask.array<shape=(1,), chunksize=(1,)>
    xtime_start                                                  (Time) |S64 dask.array<shape=(1,), chunksize=(1,)>
    refBottomDepth                                               (Time, nVertLevels) float64 dask.array<shape=(1, 100), chunksize=(1, 100)>
Attributes:
    history:  Tue Dec  6 04:49:14 2016: ncatted -O -a ,global,d,, acme_alaph7...
    NCO:      "4.6.2"

Problem description
The expected behavior for us was that refBottomDepth should not have Time as a dimension.  It does not vary with time and does not have a Time dimension in the input data set.
It seems like https://github.com/pydata/xarray/issues/1988 and https://github.com/pydata/xarray/pull/2048 were intended to address cases where the concat_dim was not yet present in the input files.  But in cases where concat_dim is already in the input files, it seems like only those fields that include this dimensions should be concatenated and other fields should remain free of concat_dim.  Part of the problem for us is that the number of dimensions of some of our variables change depending on which xarray version is being used.
Expected Output
That for 0.10.2 (see above)
Output of xr.show_versions()

/home/xylar/miniconda2/envs/mpas_analysis_py3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
INSTALLED VERSIONS
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-38-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.3
pandas: 0.22.0
numpy: 1.14.2
scipy: 1.0.1
netCDF4: 1.3.1
h5netcdf: 0.5.1
h5py: 2.7.1
Nio: None
zarr: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.17.2
distributed: 1.21.6
matplotlib: 2.2.2
cartopy: 0.16.0
seaborn: None
setuptools: 39.0.1
pip: 9.0.3
conda: None
pytest: 3.5.0
IPython: None
sphinx: 1.7.2

cc @pwolfram

What happens if you open multiple files with open_mfdataset(), e.g., for both January and February. Does it result in a dataset with the right dimensions on each variable?

@shoyer, in that case as well, Time is added to refBottomDepth in v 0.10.3, which was not the case in previous xarray versions.

👍 This is a persistent problem for me as well.
I often find myself writing a preprocessor function like this
def process_coords(ds, concat_dim='time', drop=True):
    coord_vars = [v for v in ds.data_vars if concat_dim not in ds[v].dims]
    if drop:
        return ds.drop(coord_vars)
    else:
        return ds.set_coords(coord_vars)
ds = xr.open_mfdataset('*.nc', preprocess=process_coords)

The reason to drop the coordinates is to avoid the comparison that happens when you concatenate coords.

@shoyer, I stand corrected.  in 0.10.1, I also see the Time variable getting added to refBottomDepth when I open multiple files.  So maybe this is not in fact a new problem but an existing issue that happened to behave as I expected only when opening a single file.

@rabernat, so you're fooling xarray into not including the time dimension in your non-time variables by making them coordinates in the above example?

so you're fooling xarray into not including the time dimension in your non-time variables by making them coordinates in the above example?

Exactly. They are coordinates. Those variables are usually related to grid geometry or constants, as I presume is refBottomDepth in your example.

Yes, true.  I'm trying to think if there are any examples where the fixed-in-time variables would not be coordinates.  So far, none come to mind.  Thanks for the tip.

But this issue raises an important basic point: we might want different behavior for variables in which concat_dim is already a dimension vs. variables for which it is not.

I stand corrected. in 0.10.1, I also see the Time variable getting added to refBottomDepth when I open multiple files. So maybe this is not in fact a new problem but an existing issue that happened to behave as I expected only when opening a single file in previous versions. Sorry for not noticing that sooner.

OK, in that case I think #2048 was still the right change/bug-fix, making multi-file and single-file behavior consistent.
But you certainly have exposed a real issue here.

But this issue raises an important basic point: we might want different behavior for variables in which concat_dim is already a dimension vs. variables for which it is not.

Yes, we shouldn't implicitly add a new dimensions to variables in the case where the dimension already exists in the dataset. We only need the heuristics/comparisons when an entirely new dimension is being added.

@rabernat, your suggestion above has worked perfectly to get our unit tests working again in MPAS-Analysis so that will tide us over until this issue can be addressed directly in xarray.  Thanks!

I'm glad!
FWIW, I think this is a relatively simple fix within xarray. @xylar, if you are game, we would love to see a PR from you. Could be a good opportunity to learn more about xarray internals.

Hmm, I agree that it shouldn't be hard but I don't really have time to do this right now.  If no one has had a chance to look into it by mid May I might be able to take it on then.

I recently ran into a similar issue and found a potential solution.
The functionality, as far as I understand, is already in the open_mfdataset function in the data_vars ='minimal' argument, in this case variables without concat_dim are included without adding the dimension. The current default is data_vars ='all' which include all variables with the added dimension. If the desired functionality shouldn't implicitly add new dimensions shouldn't the default be set to 'minimal' instead?
I think this is a very non-intrusive solution since it only affects the open_mfdataset function, and if you for some reason want the old behavior it is still there. An alternative way is to rewrite the _dataset_concat function I guess.
This is my first time attempting to contribute, does this sound like a good idea? I can try to make a pull request but would very much value some input first.

I'm a new developer at the SciPy 2019 Sprints, and I'm interested in making a pull request for this issue as a learning step.
@henrikca Would this be useful?  Or would this conflict with your work?

So there are some units tests that assert the behavior for open_mfdataset() is identical to the behavior for concat().   This implies that if we change the default data_vars value from "all" to "minimal" for one function, we need to change it for both functions.
@shoyer I think you suggested that concat() default behavior should change in #2145, in the same way it will change for open_mfdataset.   So I am going to add this change to the pull request.

@bonnland I don't think you want to change the default data_vars but instead update the heuristics as in this comment

we shouldn't implicitly add a new dimensions to variables in the case where the dimension already exists in the dataset. We only need the heuristics/comparisons when an entirely new dimension is being added.

@dcherian . I believe you are correct in principle, but there is a logical problem that is expensive to evaluate.  The difficult case is when two datasets have a variable with the same name, and that variable does not include the concatenation dimension.  In order to align the datasets for concatenation, both variables would need to be identical.   The resulting dataset would just have one (unchanged) instance of that variable, say from the first dataset.   I think someone along the way decided this operation was too expensive.   This is from concat.py, lines 302-307:
    # stack up each variable to fill-out the dataset (in order)
    for k in datasets[0].variables:
        if k in concat_over:
            vars = ensure_common_dims([ds.variables[k] for ds in datasets])
            combined = concat_vars(vars, dim, positions)
            insert_result_variable(k, combined)

So I think some consensus needs to be reached, about whether it is a good idea to load these variables into memory to check for identical-ness between them.
Or another possibility is that we leave "unique" variables alone:  if a variable exists only once across all datasets being concatenated, we do not add the concatenation dimension to it.   This might solve the original poster's issue when opening a single dataset.

The logic for determining which variables to concatenate is in the _calc_concat_over helper function:
https://github.com/pydata/xarray/blob/539fb4a98d0961c281daa5474a8e492a0ae1d8a2/xarray/core/concat.py#L146
Only "different" is supposed to load variables into memory to determine which ones to concatenate.
Right now we also have "all" and "minimal" options:

"all" attempts to concatenate every variable that can be broadcast to a matching shape: https://github.com/pydata/xarray/blob/539fb4a98d0961c281daa5474a8e492a0ae1d8a2/xarray/core/concat.py#L188-L190
"minimal" only concatenates variables that already have the matching dimension.

Recall that concat handles two types of concatenation: existing dimensions (corresponding to np.concatenate) and new dimensions (corresponding to np.stack). Currently, this is all done together in one messy codebase, but logically it would be cleaner to separate these modes into two separate function:

In "existing dimensions" mode:

"all" is currently broken, because it will also concatenate variables that don't have the dimension.
"minimal" does the right thing, concatenating only variables with the dimension.

In "new dimensions" mode:

"all" will add the dimension to all variables.
"minimal" raise an error if any variables have different values. If you're datasets have any data variables with different values at all, it raises an error. This is pretty much useless.

Here's my notebook testing this out: https://gist.github.com/shoyer/f44300eddda4f7c476c61f76d1df938b
So I'm thinking that we probably want to combine "all" and "minimal" into a single mode to use as the default, and remove the other behavior, which is either useless or broken. Maybe it would make sense to come up with a new name for this mode, and to make both "all" and "minimal" deprecated aliases for it? In the long term, this leaves only two "automatic" modes for xarray.concat, which should make things simpler for users trying to figure this out.

@shoyer Your explanation makes sense, but there are unit tests that expect the default concat() behavior to be the same as default behavior for Pandas concat(), which tries to perform an "outer" join between dataframes.
Therefore, from my limited understanding, the default behavior for xarray concat() should be to preserve all variables.   If this default behavior changes, then it may break code making these expectations.
Can we get a perspective from the author of concat.py, @TomNicholas ?   Thanks.

@shoyer I'm sorry I didn't look at your examples more closely at first.  I see now that your first example of using data_vars='minimal' is already preserving one instance of the variable x, and I was suggesting earlier that this variable was not being included in the concatenation.
So I am not clear on why so many unit tests fail when I switch the default value for data_vars to 'minimal'.   The output from your examples seems compatible with Pandas concat, though I don't understand Pandas very well yet.
I wonder if the unit tests that fail are written correctly.   I have to add that I spent an entire day trying to understand the code in concat.py, by stepping through it for several unit tests.    I found the code quite difficult to understand.

Specifically, what should the default behavior of concat() be, when both datasets include a variable that does not include the concatenation dimension? Currently, the concat dimension is added, and the result is a "stacked" version of the variable. Others have argued that this variable should not be included in the concat() result by default, but this appears to break compatibility with Pandas concat().

Can you give a specific example of the behavior in question?

Can you give a specific example of the behavior in question?

Here is the most specific thing I can say:  If I switch the default value for data_vars to 'minimal' for concat() and open_mfdataset(), then I get a lot of failing unit tests (when running "pytest xarray -n 4".  I may be wrong about why they are failing.  The unit tests have comments in them, like "Check pandas compatibility"; see for example, line 370 in test_duck_array_ops.py for an example instruction that raises a ValueError exception.  Many failures appear to be caused by a ValueError exception being raised, like in the final example you have in your notebook.
I hope this is specific enough; I realize that I'm not deeply comprehending what the unit tests are actually supposed to be testing.

UPDATE: @shoyer it could be that unit tests are failing because, as your final example shows, you get an error for data_vars='minimal' if any variables have different values across datasets, when adding a new concatentation dimension. If this is the reason so many unit tests are failing, then the failures are a red herring and should probably be ignored/rewritten.

This seems very likely to me. The existing behavior of data_vars='minimal' is only useful in "existing dimensions mode".
Xarray's unit test suite is definitely a good "smoke test" for understanding the impact of changes to concat on our users. What it tells us is that we can't change the default value from "all" to "minimal" without breaking existing code. Instead, we need to change how "all" or "minimal" works, or switch to yet another mode for the new behavior.
The tests we should feel free to rewrite are cases where we set data_vars="all" or data_vars="minimal" explicitly for verifying the weird edge behaviors that I noted in my earlier comments. There shouldn't be too many of these tests.

Maybe it would make sense to come up with a new name for this mode, and to make both "all" and "minimal" deprecated aliases for it?

I'm in favour of this. What should we name this mode?
One comment on "existing dimensions" mode:

"minimal" does the right thing, concatenating only variables with the dimension.

For variables without the dimension, this will still raise a ValueError because compat can only be 'equals' or 'identical'. It seems to me like we need compat='override' and/or compat='tolerance', tolerance=... that would use numpy's approximate equality testing. This checking of non-dimensional coordinates is a common source of mfdataset issues. What do you think?

I have a draft solution in #3239. It adds a new mode called "sensible" that acts like "all" when the concat dimension doesn't exist in the dataset and acts like "minimal" when the dimension is present. We can decide whether this is the right way i.e. add a new mode but the more fundamental problem is below.
The issue is dealing with variables that should not be concatentated in "minimal" mode (e.g. time-invariant non dim coords when concatenating in time). In this case, we want to skip the equality checks in _calc_concat_over. This is a common reason for poor open_mfdataset performance.
I thought the clean way to do this would be to add the compat kwarg to concat and then add compat='override' since the current behaviour is effectively compat='equals'.
However, merge takes compat too and concat and merge support different compat arguments at present. This makes it complicated to easily thread compat down from combine or open_mfdataset without adding concat_compat and merge_compat which is silly.
So do we want to support all the other compat modes in concat? Things like broadcast_equals or no_conflicts are funny because they're basically merge operations and it means concat acts like both stack, concat and merge. OTOH if you have a set of variables with the same name from different datasets and you want to pick one of those (i.e. no concatenation), then you're basically doing merge anyway. This would require some refactoring since concat assumes the first dataset is a template for the rest.
@shoyer What do you think?

I have tried to understand why the xarray developers decided to provide their own options for concatenation.   I am not an experienced user of xarray, but I can't find any discussion of how the current options for concatenation were derived.
The pandas concat() function uses the option join = {'inner', 'outer', 'left', 'right'} in order to mimic logical database join operations.    If there is a reason that xarray cannot do the same, it is not obvious to me.   I think the pandas options have the advantage of logical simplicity and traditional usage within database systems.

Thanks for your input @bonnland.

The pandas concat() function uses the option join = {'inner', 'outer', 'left', 'right'} in order to mimic logical database join operations. If there is a reason that xarray cannot do the same, it is not obvious to me. I think the pandas options have the advantage of logical simplicity and traditional usage within database systems.

We do have a join argument that takes these arguments + 'override' which was added recently to skip expensive comparisons. This works for "indexes" or "dimension coordinates". An example: if you have 2 dataarrays, one on a coordinate x=[1, 2, 3] and the other on x=[2,3,4], join lets you control the x coordinate of the output. This is done by xr.align.
What's under discussion here is what to do about variables duplicated across datasets or indeed, how do we know that these variables are duplicated across datasets when concatenating other variables.

#3239 has been merged. Now minimal is more useful since you can specify compat="override" to skip compatibility checking.
What's left is to change defaults to implement @shoyer's comment

So I'm thinking that we probably want to combine "all" and "minimal" into a single mode to use as the default, and remove the other behavior, which is either useless or broken. Maybe it would make sense to come up with a new name for this mode, and to make both "all" and "minimal" deprecated aliases for it? In the long term, this leaves only two "automatic" modes for xarray.concat, which should make things simpler for users trying to figure this out.

authors: bonnland dcherian henrikca rabernat shoyer xylar

repo: pydata/xarray


text:
xr.dot requires equal indexes (join="exact")

MCVE Code Sample
import xarray as xr
import numpy as np

d1 = xr.DataArray(np.arange(4), dims=["a"], coords=dict(a=[0, 1, 2, 3]))
d2 = xr.DataArray(np.arange(4), dims=["a"], coords=dict(a=[0, 1, 2, 3]))

# note: different coords
d3 = xr.DataArray(np.arange(4), dims=["a"], coords=dict(a=[1, 2, 3, 4]))

(d1 * d2).sum() # -> array(14)
xr.dot(d1, d2) # -> array(14)

(d2 * d3).sum() # -> array(8)
xr.dot(d2, d3) # -> ValueError

Expected Output
<xarray.DataArray ()>
array(8)

Problem Description
The last statement results in an
ValueError: indexes along dimension 'a' are not equal

because xr.apply_ufunc defaults to join='exact'. However, I think this should work -
but maybe there is a good reason for this to fail?
This is a problem for #2922 (weighted operations) - I think it is fine for the weights and data to not align.
Fixing this may be as easy as specifying join='inner' in
https://github.com/pydata/xarray/blob/e0fd48052dbda34ee35d2491e4fe856495c9621b/xarray/core/computation.py#L1181-L1187
@fujiisoup
Output of xr.show_versions()

INSTALLED VERSIONS
commit: 5afc6f32b18f5dbb9a89e30f156b626b0a83597d
python: 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 21:52:21)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.12.14-lp151.28.36-default
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.6.2
xarray: 0.14.0+164.g5afc6f32.dirty
pandas: 0.25.2
numpy: 1.17.3
scipy: 1.3.1
netCDF4: 1.5.1.2
pydap: installed
h5netcdf: 0.7.4
h5py: 2.10.0
Nio: 1.5.5
zarr: 2.3.2
cftime: 1.0.4.2
nc_time_axis: 1.2.0
PseudoNetCDF: installed
rasterio: 1.1.0
cfgrib: 0.9.7.2
iris: 2.2.0
bottleneck: 1.2.1
dask: 2.6.0
distributed: 2.6.0
matplotlib: 3.1.1
cartopy: 0.17.0
seaborn: 0.9.0
numbagg: installed
setuptools: 41.6.0.post20191029
pip: 19.3.1
conda: None
pytest: 5.2.2
IPython: 7.9.0
sphinx: None

I have no strong opinion, but if most of the arithmetic in xarray uses join='inner', then it would be nicer to do so here too.

authors: fujiisoup mathause


text:
Three test_aggregation[int-method_median] tests failing

Follow-up of #3777.
The three failing tests seems to be failing because dask_array=None, which is likely the same kind of issue as #3777: dask-dependant tests ran while dask is not available. The other printed error is strange to me, because numpy is at version 1.18.1 on this system.
_______________ TestVariable.test_aggregation[int-method_median] _______________

values = array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1]), axis = None, skipna = None
kwargs = {}, func = <function _dask_or_eager_func.<locals>.f at 0x7f3927bdbe50>
msg = 'median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None'

    def f(values, axis=None, skipna=None, **kwargs):
        if kwargs.pop("out", None) is not None:
            raise TypeError(f"`out` is not valid for {name}")

        values = asarray(values)

        if coerce_strings and values.dtype.kind in "SU":
            values = values.astype(object)

        func = None
        if skipna or (skipna is None and values.dtype.kind in "cfO"):
            nanname = "nan" + name
            func = getattr(nanops, nanname)
        else:
            func = _dask_or_eager_func(name, dask_module=dask_module)

        try:
>           return func(values, axis=axis, **kwargs)

xarray/core/duck_array_ops.py:307:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1]),), kwargs = {'axis': None}
dispatch_args = (array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1]),)

    def f(*args, **kwargs):
        if list_of_args:
            dispatch_args = args[0]
        else:
            dispatch_args = args[array_args]
>       if any(isinstance(a, dask_array.Array) for a in dispatch_args):

xarray/core/duck_array_ops.py:40:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <tuple_iterator object at 0x7f3927966f70>

>   if any(isinstance(a, dask_array.Array) for a in dispatch_args):
E   AttributeError: 'NoneType' object has no attribute 'Array'

xarray/core/duck_array_ops.py:40: AttributeError

During handling of the above exception, another exception occurred:

self = <xarray.tests.test_units.TestVariable object at 0x7f3927966640>
func = method_median, dtype = <class 'int'>

    @pytest.mark.parametrize(
        "func",
        (
            method("all"),
            method("any"),
            method("argmax"),
            method("argmin"),
            method("argsort"),
            method("cumprod"),
            method("cumsum"),
            method("max"),
            method("mean"),
            method("median"),
            method("min"),
            pytest.param(
                method("prod"),
                marks=pytest.mark.xfail(reason="not implemented by pint"),
            ),
            method("std"),
            method("sum"),
            method("var"),
        ),
        ids=repr,
    )
    def test_aggregation(self, func, dtype):
        array = np.linspace(0, 1, 10).astype(dtype) * (
            unit_registry.m if func.name != "cumprod" else unit_registry.dimensionless
        )
        variable = xr.Variable("x", array)

        units = extract_units(func(array))
>       expected = attach_units(func(strip_units(variable)), units)

xarray/tests/test_units.py:1389:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xarray/tests/test_units.py:374: in __call__
    return func(*all_args, **all_kwargs)
xarray/core/common.py:46: in wrapped_func
    return self.reduce(func, dim, axis, skipna=skipna, **kwargs)
xarray/core/variable.py:1537: in reduce
    data = func(input_data, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

values = array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1]), axis = None, skipna = None
kwargs = {}, func = <function _dask_or_eager_func.<locals>.f at 0x7f3927bdbe50>
msg = 'median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None'

    def f(values, axis=None, skipna=None, **kwargs):
        if kwargs.pop("out", None) is not None:
            raise TypeError(f"`out` is not valid for {name}")

        values = asarray(values)

        if coerce_strings and values.dtype.kind in "SU":
            values = values.astype(object)

        func = None
        if skipna or (skipna is None and values.dtype.kind in "cfO"):
            nanname = "nan" + name
            func = getattr(nanops, nanname)
        else:
            func = _dask_or_eager_func(name, dask_module=dask_module)

        try:
            return func(values, axis=axis, **kwargs)
        except AttributeError:
            if isinstance(values, dask_array_type):
                try:  # dask/dask#3133 dask sometimes needs dtype argument
                    # if func does not accept dtype, then raises TypeError
                    return func(values, axis=axis, dtype=values.dtype, **kwargs)
                except (AttributeError, TypeError):
                    msg = "%s is not yet implemented on dask arrays" % name
            else:
                msg = (
                    "%s is not available with skipna=False with the "
                    "installed version of numpy; upgrade to numpy 1.12 "
                    "or newer to use skipna=True or skipna=None" % name
                )
>           raise NotImplementedError(msg)
E           NotImplementedError: median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None

xarray/core/duck_array_ops.py:321: NotImplementedError
______________ TestDataArray.test_aggregation[int-method_median] _______________

values = array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), axis = None, skipna = None
kwargs = {}, func = <function _dask_or_eager_func.<locals>.f at 0x7f39286dbc10>
msg = 'median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None'

    def f(values, axis=None, skipna=None, **kwargs):
        if kwargs.pop("out", None) is not None:
            raise TypeError(f"`out` is not valid for {name}")

        values = asarray(values)

        if coerce_strings and values.dtype.kind in "SU":
            values = values.astype(object)

        func = None
        if skipna or (skipna is None and values.dtype.kind in "cfO"):
            nanname = "nan" + name
            func = getattr(nanops, nanname)
        else:
            func = _dask_or_eager_func(name, dask_module=dask_module)

        try:
>           return func(values, axis=axis, **kwargs)

xarray/core/duck_array_ops.py:307:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),), kwargs = {'axis': None}
dispatch_args = (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),)

    def f(*args, **kwargs):
        if list_of_args:
            dispatch_args = args[0]
        else:
            dispatch_args = args[array_args]
>       if any(isinstance(a, dask_array.Array) for a in dispatch_args):

xarray/core/duck_array_ops.py:40:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <tuple_iterator object at 0x7f39288ad880>

>   if any(isinstance(a, dask_array.Array) for a in dispatch_args):
E   AttributeError: 'NoneType' object has no attribute 'Array'

xarray/core/duck_array_ops.py:40: AttributeError

During handling of the above exception, another exception occurred:

self = <xarray.tests.test_units.TestDataArray object at 0x7f39288ad190>
func = method_median, dtype = <class 'int'>

    @pytest.mark.parametrize(
        "func",
        (
            pytest.param(
                function("all"),
                marks=pytest.mark.xfail(reason="not implemented by pint yet"),
            ),
            pytest.param(
                function("any"),
                marks=pytest.mark.xfail(reason="not implemented by pint yet"),
            ),
            function("argmax"),
            function("argmin"),
            function("max"),
            function("mean"),
            pytest.param(
                function("median"),
                marks=pytest.mark.xfail(reason="not implemented by xarray"),
            ),
            function("min"),
            pytest.param(
                function("prod"),
                marks=pytest.mark.xfail(reason="not implemented by pint yet"),
            ),
            function("sum"),
            function("std"),
            function("var"),
            function("cumsum"),
            pytest.param(
                function("cumprod"),
                marks=pytest.mark.xfail(reason="not implemented by pint yet"),
            ),
            pytest.param(
                method("all"),
                marks=pytest.mark.xfail(reason="not implemented by pint yet"),
            ),
            pytest.param(
                method("any"),
                marks=pytest.mark.xfail(reason="not implemented by pint yet"),
            ),
            method("argmax"),
            method("argmin"),
            method("max"),
            method("mean"),
            method("median"),
            method("min"),
            pytest.param(
                method("prod"),
                marks=pytest.mark.xfail(
                    reason="comparison of quantity with ndarrays in nanops not implemented"
                ),
            ),
            method("sum"),
            method("std"),
            method("var"),
            method("cumsum"),
            pytest.param(
                method("cumprod"),
                marks=pytest.mark.xfail(reason="pint does not implement cumprod yet"),
            ),
        ),
        ids=repr,
    )
    def test_aggregation(self, func, dtype):
        array = np.arange(10).astype(dtype) * (
            unit_registry.m if func.name != "cumprod" else unit_registry.dimensionless
        )
        data_array = xr.DataArray(data=array, dims="x")

        # units differ based on the applied function, so we need to
        # first compute the units
        units = extract_units(func(array))
>       expected = attach_units(func(strip_units(data_array)), units)

xarray/tests/test_units.py:2226:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xarray/tests/test_units.py:374: in __call__
    return func(*all_args, **all_kwargs)
xarray/core/common.py:46: in wrapped_func
    return self.reduce(func, dim, axis, skipna=skipna, **kwargs)
xarray/core/dataarray.py:2235: in reduce
    var = self.variable.reduce(func, dim, axis, keep_attrs, keepdims, **kwargs)
xarray/core/variable.py:1537: in reduce
    data = func(input_data, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

values = array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), axis = None, skipna = None
kwargs = {}, func = <function _dask_or_eager_func.<locals>.f at 0x7f39286dbc10>
msg = 'median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None'

    def f(values, axis=None, skipna=None, **kwargs):
        if kwargs.pop("out", None) is not None:
            raise TypeError(f"`out` is not valid for {name}")

        values = asarray(values)

        if coerce_strings and values.dtype.kind in "SU":
            values = values.astype(object)

        func = None
        if skipna or (skipna is None and values.dtype.kind in "cfO"):
            nanname = "nan" + name
            func = getattr(nanops, nanname)
        else:
            func = _dask_or_eager_func(name, dask_module=dask_module)

        try:
            return func(values, axis=axis, **kwargs)
        except AttributeError:
            if isinstance(values, dask_array_type):
                try:  # dask/dask#3133 dask sometimes needs dtype argument
                    # if func does not accept dtype, then raises TypeError
                    return func(values, axis=axis, dtype=values.dtype, **kwargs)
                except (AttributeError, TypeError):
                    msg = "%s is not yet implemented on dask arrays" % name
            else:
                msg = (
                    "%s is not available with skipna=False with the "
                    "installed version of numpy; upgrade to numpy 1.12 "
                    "or newer to use skipna=True or skipna=None" % name
                )
>           raise NotImplementedError(msg)
E           NotImplementedError: median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None

xarray/core/duck_array_ops.py:321: NotImplementedError
_______________ TestDataset.test_aggregation[int-method_median] ________________

values = <Quantity([0 0 0 0 0 0 0 0 0 1], 'pascal')>, axis = 0, skipna = None
kwargs = {}, func = <function _dask_or_eager_func.<locals>.f at 0x7f392619b820>
msg = 'median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None'

    def f(values, axis=None, skipna=None, **kwargs):
        if kwargs.pop("out", None) is not None:
            raise TypeError(f"`out` is not valid for {name}")

        values = asarray(values)

        if coerce_strings and values.dtype.kind in "SU":
            values = values.astype(object)

        func = None
        if skipna or (skipna is None and values.dtype.kind in "cfO"):
            nanname = "nan" + name
            func = getattr(nanops, nanname)
        else:
            func = _dask_or_eager_func(name, dask_module=dask_module)

        try:
>           return func(values, axis=axis, **kwargs)

xarray/core/duck_array_ops.py:307:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<Quantity([0 0 0 0 0 0 0 0 0 1], 'pascal')>,), kwargs = {'axis': 0}
dispatch_args = (<Quantity([0 0 0 0 0 0 0 0 0 1], 'pascal')>,)

    def f(*args, **kwargs):
        if list_of_args:
            dispatch_args = args[0]
        else:
            dispatch_args = args[array_args]
>       if any(isinstance(a, dask_array.Array) for a in dispatch_args):

xarray/core/duck_array_ops.py:40:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <tuple_iterator object at 0x7f39269995e0>

>   if any(isinstance(a, dask_array.Array) for a in dispatch_args):
E   AttributeError: 'NoneType' object has no attribute 'Array'

xarray/core/duck_array_ops.py:40: AttributeError

During handling of the above exception, another exception occurred:

self = <xarray.tests.test_units.TestDataset object at 0x7f3927adf880>
func = method_median, dtype = <class 'int'>

    @pytest.mark.parametrize(
        "func",
        (
            pytest.param(
                function("all"),
                marks=pytest.mark.xfail(reason="not implemented by pint"),
            ),
            pytest.param(
                function("any"),
                marks=pytest.mark.xfail(reason="not implemented by pint"),
            ),
            function("argmax"),
            function("argmin"),
            function("max"),
            function("min"),
            function("mean"),
            pytest.param(
                function("median"),
                marks=pytest.mark.xfail(
                    reason="np.median does not work with dataset yet"
                ),
            ),
            function("sum"),
            pytest.param(
                function("prod"),
                marks=pytest.mark.xfail(reason="not implemented by pint"),
            ),
            function("std"),
            function("var"),
            function("cumsum"),
            pytest.param(
                function("cumprod"),
                marks=pytest.mark.xfail(reason="fails within xarray"),
            ),
            pytest.param(
                method("all"), marks=pytest.mark.xfail(reason="not implemented by pint")
            ),
            pytest.param(
                method("any"), marks=pytest.mark.xfail(reason="not implemented by pint")
            ),
            method("argmax"),
            method("argmin"),
            method("max"),
            method("min"),
            method("mean"),
            method("median"),
            method("sum"),
            pytest.param(
                method("prod"),
                marks=pytest.mark.xfail(reason="not implemented by pint"),
            ),
            method("std"),
            method("var"),
            method("cumsum"),
            pytest.param(
                method("cumprod"), marks=pytest.mark.xfail(reason="fails within xarray")
            ),
        ),
        ids=repr,
    )
    def test_aggregation(self, func, dtype):
        unit_a = (
            unit_registry.Pa if func.name != "cumprod" else unit_registry.dimensionless
        )
        unit_b = (
            unit_registry.kg / unit_registry.m ** 3
            if func.name != "cumprod"
            else unit_registry.dimensionless
        )
        a = xr.DataArray(data=np.linspace(0, 1, 10).astype(dtype) * unit_a, dims="x")
        b = xr.DataArray(data=np.linspace(-1, 0, 10).astype(dtype) * unit_b, dims="x")
        x = xr.DataArray(data=np.arange(10).astype(dtype) * unit_registry.m, dims="x")
        y = xr.DataArray(
            data=np.arange(10, 20).astype(dtype) * unit_registry.s, dims="x"
        )

        ds = xr.Dataset(data_vars={"a": a, "b": b}, coords={"x": x, "y": y})

>       actual = func(ds)

xarray/tests/test_units.py:3733:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
xarray/tests/test_units.py:374: in __call__
    return func(*all_args, **all_kwargs)
xarray/core/common.py:83: in wrapped_func
    return self.reduce(
xarray/core/dataset.py:4230: in reduce
    variables[name] = var.reduce(
xarray/core/variable.py:1535: in reduce
    data = func(input_data, axis=axis, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

values = <Quantity([0 0 0 0 0 0 0 0 0 1], 'pascal')>, axis = 0, skipna = None
kwargs = {}, func = <function _dask_or_eager_func.<locals>.f at 0x7f392619b820>
msg = 'median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None'

    def f(values, axis=None, skipna=None, **kwargs):
        if kwargs.pop("out", None) is not None:
            raise TypeError(f"`out` is not valid for {name}")

        values = asarray(values)

        if coerce_strings and values.dtype.kind in "SU":
            values = values.astype(object)

        func = None
        if skipna or (skipna is None and values.dtype.kind in "cfO"):
            nanname = "nan" + name
            func = getattr(nanops, nanname)
        else:
            func = _dask_or_eager_func(name, dask_module=dask_module)

        try:
            return func(values, axis=axis, **kwargs)
        except AttributeError:
            if isinstance(values, dask_array_type):
                try:  # dask/dask#3133 dask sometimes needs dtype argument
                    # if func does not accept dtype, then raises TypeError
                    return func(values, axis=axis, dtype=values.dtype, **kwargs)
                except (AttributeError, TypeError):
                    msg = "%s is not yet implemented on dask arrays" % name
            else:
                msg = (
                    "%s is not available with skipna=False with the "
                    "installed version of numpy; upgrade to numpy 1.12 "
                    "or newer to use skipna=True or skipna=None" % name
                )
>           raise NotImplementedError(msg)
E           NotImplementedError: median is not available with skipna=False with the installed version of numpy; upgrade to numpy 1.12 or newer to use skipna=True or skipna=None

xarray/core/duck_array_ops.py:321: NotImplementedError

However I’m not much knowledgeable on all this, so I’ll defer to you for finding the root cause.

Thanks for reporting, we don't seem to be testing a lot without dask installed (me especially, in this case).
These fail because _dask_or_eager_func uses dask_array.Array for instance checks instead of the already imported dask_array_type. Also, we should probably go through dask_array_compat and replace the references to da.Array (where da can be None) with dask_array_type.

Well, the thing is we don’t package dask (yet!), so I tested with all we had (i.e. 7 of the 20 existing optdeps), which is a configuration quite different from your usual test matrix. ;)
I might package dask in the future, but they are too much missing dependencies for it in our repos for now, and I don’t have enough time currently to go into packaging each of them and their own dependencies, etc.

authors: ArchangeGabriel keewis