text:
Added type check disable to ActivitySurrogateSelector generators

Reference: https://silentbreaksecurity.com/re-animating-activitysurrogateselector/
The current changes are very minimal in terms of "integration". I had some additional thoughts that might be good to discuss before merging:

Currently both surrogate payloads will always be paired with the disable, but this would only be required if you were targeting newer framework versions. I don't suspect it would cause issues in earlier versions, but I hate making that assumption.
I had considered some sort of flag argument to control code flow, but didn't want to alter the core Generate(input, fmt, test) format. If you are okay with this, maybe a flag for either version support or "mitigation bypasses". I'm happy to go any direction.
Because we have to modify the AppSettings in ysoserial itself before serializing, testing becomes inaccurate (we don't know if the bypass is working). During my testing I just reset the flag after the fmt.Serialize call, but this would be repetitive due to the way the internal Serialize() function is currently layed out.

I think perhaps adding a note would be suffice otherwise this may break some of the plugins that rely on this gadget too? Or maybe we should disable them too?

Because we have to modify the AppSettings in ysoserial itself before serializing, testing becomes inaccurate (we don't know if the bypass is working). During my testing I just reset the flag after the fmt.Serialize call, but this would be repetitive due to the way the internal Serialize() function is currently layed out.

IMHO, test flag was always just for quick PoC, because anyway the payload is deserialized on another computer with potentially different .NET framework installed and can't be a real test.

Thanks for the PR!
I think it would make sense to create two new separate generators, something such as WrappedActivitySurrogateSelectorGenerator.cs and WrappedActivitySurrogateSelectorGenerator.cs, thoughts?

IMHO, test flag was always just for quick PoC

I'm in agreement there, might be worth adding a note at the very least for user edification.

I think it would make sense to create two new separate generators, something such as WrappedActivitySurrogateSelectorGenerator.cs

Creating two more generators might be the cleanest way to go for now. Minimal core code changes and disambiguation between functionality. The FromFileGenerator variation could be seen as following the same strategy if we are aiming for consistency.
Really the worst con is excessive/repetitive information while printing the gadgets. The help text gets quite a bit longer to show very similar information. Also worth noting that pairing/augmenting existing gadgets might become a more common occurrence. For that it might be best to consider some kind of "inheritance respect" for the GenericGenerator classes. Maybe the help text could reflect that one gadget is just an alternative to another gadget.
An even larger consideration might be to expand gadgets out like sub-commands, and allow them to have contextually different arguments and help text.
I'd be happy to help with any of this.

After some additional testing, it appears my initial assumption was incorrect regarding the ability to stack the objects in a list.
I erroneously assumed that I could flip the AppSetting disableTypeCheck back to false before testing deserialization, but the global flag is only read once. After the settings are "initialized", they are no longer referenced and the internal boolean holds the value. It appeared to work only because I was testing in the same app instance.
To actually reset the setting, you have to add reflection:
ConfigurationManager.AppSettings.Set("microsoft:WorkflowComponentModel:DisableActivitySurrogateSelectorTypeCheck", "false");

Type appSettings = Type.GetType("System.Workflow.ComponentModel.AppSettings, System.Workflow.ComponentModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35");
if (appSettings != null)
{
    var disableTypeCheckFlag = appSettings.GetField("disableActivitySurrogateSelectorTypeCheck", BindingFlags.NonPublic | BindingFlags.Static);
    disableTypeCheckFlag.SetValue(null, false);

    var initializedFlag = appSettings.GetField("settingsInitialized", BindingFlags.NonPublic | BindingFlags.Static);
    initializedFlag.SetValue(null, false);
}

The disable payload still works correctly, but only when triggered first via fmt.Deserialize, following by a separate call for the real payload. My assumption is that once the disable payload is finished, the process stops due to the exception raised. Therefore the formatter never gets to the second object in the list.
I'll keep working on this to try and get a single "wrapped" payload. If anyone has any ideas, let me know.

For that it might be best to consider some kind of "inheritance respect" for the GenericGenerator classes.

That sounds good, I dont see any problems doing that.

An even larger consideration might be to expand gadgets out like sub-commands, and allow them to have contextually different arguments and help text.

This sounds similar to plugins, but I think in this case it makes more sense to use raw generators, even if two more are added to the help text.

I'll keep working on this to try and get a single "wrapped" payload. If anyone has any ideas, let me know.

Im not sure I followed correctly, so I may be saying something stupid here, but what about adding the resting flag reflective code as part of the ActivitySurrogate payload class?

Im not sure I followed correctly, so I may be saying something stupid here, but what about adding the resting flag reflective code as part of the ActivitySurrogate payload class?

In essence, I was hoping to avoid requiring that multiple payloads be used sequentially (separate calls to fmt.Deserialize), but it appears that's the case. My theory of using a List<Object> = {disable, payload} fails due to the exception being raised when the first object is deserialized, but before the second is touched.
I might be missing something here myself, but provided there is no workaround, it might be best to create a separate generator called ActivitySurrogateDisableTypeCheck which you would send to a target initially before using the traditional ActivitySurrogateSelector.

Ok, then it would make more sense to create the ActivitySurrogateDisableTypeCheck generator unless you can find a new gadget to handle the exception

Alright, changes are complete, this should be good for the time being. I also expanded the gadget to use reflection to support situations where the setting has already been loaded into the WorkFlow..AppSettings boolean.

Awesome, thanks!

authors: JarLob irsdl monoxgas pwntester


text:
🛑 NARLabs is down

In c34ba29, NARLabs (https://www.narlabs.org.tw/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: NARLabs is back up in 6575965.

source: gharchive/issue

repo: pwtsai/upptime

license: MIT

license_type: permissive

license_source: github-api


text:
Result object serialization

Hi everyone!
Current state
Right now the results object gotten by running something like result = mapdl.result is probably the default way to post process models, and it works great for that. In my experience, it produces awesome plots, better than pure APDL would, and you get pyvista to manipulate them.
If you try to store a result object in a variable, you get the object "saved", but as soon as you run another model within your instance, you lose access to the results in the original object.

Proposed feature
The suggestion I'd have for this, and I don't know if this is doable at all, would be to implement result object serialization, making it compatible with something like pickle, in order to have a more tightly knit integration with the rest of python. Right now, the result object as I understand it (from a users perspective, so I might be wrong) is tightly knit with an active MAPDL session and a result file that's somewhere. I'm guessing this makes the object much lighter in memory (although more reliant on I/O), only pulling the data from the result files when asked (for example, when plotting, or when running result.nodal_solution(0)). However, these instructions get the data "pythonized", and out of the result file.
It would be nice if we could get the result object as an independent and serializable object. This would allow things like running a batch of models with either one or multiple MAPDL instances and then postprocess results independently without having to resort to more complicated file handling and explicit ansys.mapdl.reader use.
Again, I don't know how doable this is. Maybe it also wouldn't be useful as a default, so maybe a separate result object (pythonized_result, or mem_result) could be made, inheriting the postprocessing from the current result object. That way we'd get the best from both worlds, the simple manipulation of the result object, and compatibility with stablished python data types, such as dictionaries (which would be really interesting to store parametric sweep outputs in structures like { inputs : [outputs, rst_object]} )

Duplicate of #727

Hi @CesarRodriguezPereira
Thank you for posting this issue. This is something have thought about it, you can see more details of this in the mentioned issue.
So far, there is no intention to implement this. We might have a look in the future though.
Thank you again!

repo: pyansys/pymapdl


text:
Try using towncrier for changelog

https://towncrier.readthedocs.io/en/stable/
This might be easier to use than manually changing the changelog file. Could be done as part of the documentation gsoc project

Sure, we can do this as a stretch goal. The changelog is in md (markdown format), we need to change it to rst (reStructured Text) before using towncrier.

can i work on this ??

sure

@arunavabasu-03 are you still working on this?

@tinosulzer , we have to do this with jinija templateing or <version>.md file for every release .

Can you pls clarify this @tinosulzer ?

authors: arunavabasu-03 jeromtom tinosulzer

repo: pybamm-team/PyBaMM


text:
Form factors

Add simple form factor calculations for cylindrical and for stacked, rectangular cells

this seems to have changes mixed in from multiple PRs? what order should they be reviewed and merged in?

source: gharchive/pull-request

authors: julian-evers rtimms

repo: pybamm-team/pybamm-tea


text:
run stubtest on _pydantic_core.pyi

Change Summary
This PR adds use of mypy's stubtest tool to validate the .pyi file for pydantic-core, and fixes the errors detected.
Most of this is mechanical fixups which are good to have correct but not interesting to debate. The interesting things of note:

Two errors related to methods I didn't want in the stub, see TODO notes in .mypy-stubtest-allowlist.
I moved the TypedDict definitions to pydantic_core/__init__.py as these were reported at missing at runtime, and I think it's fair users may want to import these.
Many classes marked @final, because PyO3 types are final by default. We should review if we want to allow subtyping any of them.

Related issue number
N/A
Checklist

[ ] Unit tests for the changes exist
[ ] Documentation reflects the changes where applicable
[ ] Pydantic tests pass with this pydantic-core (except for expected changes)
[ ] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @adriangb

please review

Codecov Report

Merging #702 (db90671) into main (c1c27d4) will increase coverage by 0.00%.
The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #702   +/-   ##
=======================================
  Coverage   93.73%   93.74%
=======================================
  Files          99       99
  Lines       13806    13821   +15
  Branches       25       25
=======================================
+ Hits        12941    12956   +15
  Misses        859      859
  Partials        6        6

Impacted Files
Coverage Δ

pydantic_core/__init__.py
88.23% <100.00%> (+6.41%)
:arrow_up:

src/argument_markers.rs
87.09% <100.00%> (+0.21%)
:arrow_up:

src/url.rs
99.59% <100.00%> (+<0.01%)
:arrow_up:

src/validators/mod.rs
98.03% <100.00%> (+<0.01%)
:arrow_up:

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1c27d4...db90671. Read the comment docs.

Lint error is annoying consequence of installed wheel not placing native module inside the source tree, I'll push a proposed fix.

Rebased on #705, which should make CI green.

authors: codecov-commenter davidhewitt

repo: pydantic/pydantic-core


text:
Getting TypeError when using a field with a reserved keyword

Initial Checks

[X] I confirm that I'm using Pydantic V2 installed directly from the main branch, or equivalent

Description
Getting this error when migrating my project to Pydantic V2, I can use aliases to workaround this, but this works with Pydantic V1 so just wanted to make sure you guys are aware.
stack trace:
list: list[Any] | None = None
E   TypeError: 'NoneType' object is not subscriptable

Example Code
from pydantic import BaseModel
from typing import Any

class MyModel(BaseModel):
    list: list[Any] | None = None
    datetime: datetime | None = None

MyModel.model_validate({'list': [1,2,3]})

Python, Pydantic & OS Version
pydantic version: 2.0
pydantic-core version: 2.0.1 release build profile install path: /Users/Jose/Workspace/TestProject/venv/lib/python3.11/site-packages/pydantic
python version: 3.11.4 (main, Jun  7 2023, 00:34:59) [Clang 14.0.3 (clang-1403.0.22.14.1)]
platform: macOS-13.4.1-arm64-arm-64bit
optional deps. installed: ['typing-extensions']

Selected Assignee: @Kludex

This doesn't seem to work on v1 as well:
from pydantic import BaseModel
from datetime import datetime
from typing import Any

class MyModel(BaseModel):
    list: list[Any] | None = None  # You can't use `"list"` as field name
    d: datetime | None = None

MyModel(list=[1, 2, 3])

You are using list and datetime as attributes, which are the same as the types you are trying to use. You can change the field names, and it will work as expected.

All good with closing it, just figured out that's not actually the Pydantic version, it's related to nested models, see this example:
from pydantic import BaseModel
from datetime import datetime

class Child(BaseModel):
    list: list[int]
    datetime: datetime | None

class Parent(BaseModel):
    child: Child

Parent.model_validate({'child': {'list': [1, 2, 3, 4, 5], 'datetime': '2023-06-03 00:00:00'}})

The above example is also working... What was supposed to happen?

See the example, of how I'm using the field names list and datetime, they don't throw a TypeError: 'NoneType' object is not subscriptable when the model is nested, if I do the same in the parent model it fails with that error.

To be clear, this is not a bug in pydantic — it's an issue with python:
from datetime import datetime
from typing import Any

class MyModel:
    list: list[Any] | None = None
    datetime: datetime | None = None
"""
TypeError: 'NoneType' object is not subscriptable
"""

It seems the annotations get evaluated in the presence of the class namespace. And because pydantic needs to actually introspect them, it will error when building the model even if you try to do something like list: 'list[Any] | None' = None.
If you need this to be the field name, I would suggest using an alias:
from datetime import datetime
from typing import Any

from pydantic import BaseModel, Field

class MyModel(BaseModel):
    l: list[Any] | None = Field(default=None, alias='list')
    d: datetime | None = Field(default=None, alias='datetime')

print(MyModel.model_validate({"list": [1, 2, 3]}).model_dump(by_alias=True))
#> {'list': [1, 2, 3], 'datetime': None}

Oh, better yet — if you just pre-assign an alias that won't conflict with the namespace in the class definition, it works as desired:
from datetime import datetime
from typing import Any

from pydantic import BaseModel

List = list
Datetime = datetime

class MyModel(BaseModel):
    list: List[Any] = None
    datetime: Datetime = None

print(MyModel.model_validate({"list": [1, 2, 3]}))
#> list=[1, 2, 3] datetime=None

I think that's as good as you could possibly hope for given the fact that the original code example raises an error even if the class isn't a pydantic model.

Very impressive explanation @dmontagu! Thank you, it's definitely an edge case, addressed by using a pydantic alias similar to the example you posted above 🚀

authors: Kludex dmontagu josecsotomorales

repo: pydantic/pydantic


text:
Fix the un-hashability of various annotation types, for use in caching generic containers

skip change file check

please review


text:
Fix float -> Decimal coercion precision loss

Fixes #6807

please review cc @hramezani


text:
Expand test coverage for experimental Pipeline API

Change Summary

I have added the following test cases to test_pipeline.py:

test_ge_le_gt_lt

Previously there were only ge and le, so I added test codes for gt and lt.

test_parse_multipleOf

Previously there were only cases for int, so I added test codes for float and decimal.

test_len_constraints

Newly added.

test_interval_constraints

Newly added.

I'm not confident if this test case is correctly constructed, so I would like you to review it.

test_parse_tz

Added because there was no pattern for ValueError.

Pre-PR Coverage of pipeline.py: 83.30%
Current Coverage of pipeline.py w/ this PR: 93.01%
Related issue number

fix https://github.com/pydantic/pydantic/issues/9524
Checklist

[x] The pull request title is a good summary of the changes - it will be used in the changelog
[x] Unit tests for the changes exist
[x] Tests pass on CI
[x] Documentation reflects the changes where applicable
[x] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @sydney-runkle

please review

@sydney-runkle
Thank you for merging the PR!

What did you think about this API? We're looking for feedback on this new experimental feature 🚀 !

I genuinely think this is a great feature.
In the past, when defining complex constraints and transformations for model fields in Pydantic, we had to use separate Field Validators and Model Validators. However, with the introduction of this pipeline API, I believe it has enhanced type safety and improved readability with more concise syntax.
If there are any other benefits or effective ways to use this pipeline API, I would love to learn about them. :)
I understand this is currently an experimental feature, but I look forward to incorporating it once it is officially released. Thank you for implementing such a useful feature!

@K-dash,
Wonderful, thanks so much for your feedback! We're super excited to get your feedback. @adriangb did all of the initial work on this awesome feature, and even wrote a PEP to help support this! https://peps.python.org/pep-0746/
I think the main benefits are:

Type safety
Readability
Ease of reuse
Customizing constraint orders (like for string constraints, which isn't currently easily doable without this feature)

I'm sure there are even more things, but these were the main motivating factors!

@sydney-runkle
Thank you for your response.
I have looked at PEP 746 created by Adrian.
I see, in order to validate the consistency between the type annotations specified in Annotated and the associated metadata, the metadata needs to implement the  __supports_type__  protocol.
I hadn't understood that part.

Ease of reuse
Customizing constraint orders (like for string constraints, which isn't currently easily doable without this feature)

Indeed, I also think the above two benefits are significant. Thank you.
Through the implementation of these test codes, I have learned a lot not only about the pipeline API but also about type annotations. I am very grateful!

authors: K-dash sydney-runkle


text:
DOC: document level-dropping for empty columns

I might have missed it, but I couldn't find this behavior documented anywhere.  It is occasionally useful, if a bit magical.
In [1]: df1 = pd.DataFrame([1,2,3], columns=[['a'],['b']])

In [2]: df2 = pd.DataFrame([1,2,3], columns=[['a'],['']])

In [5]: df1['a'], type(df1['a'])
Out[5]:
(   b
 0  1
 1  2
 2  3, pandas.core.frame.DataFrame)

In [6]: df2['a'], type(df2['a'])
Out[6]:
(0    1
 1    2
 2    3
 Name: a, dtype: int64, pandas.core.series.Series)

yeh '' is an empty level for dropping purposes. The empty level concept is much better now-adays IOW NaN is pretty good. So not really sure this is necessary any longer.
All of that said. This is pretty buried in pretty tangled code.

repo: pydata/pandas


text:
v0.10.5 release

We had several regressions in v0.10.4, so I'd like to issue a v0.10.5 release shortly with some fixes.
I also had a couple of highly productive plane rides to/from vacation, so I found time to put together a number of bug-fix PRs. If anyone has time to help review these, it would be greatly appreciated!
Issues to resolve:

How should Dataset.update() handle conflicting coordinates? #2180 (resolved for now)

PRs for regressions in v0.10.4:

[x] Fix writing to groups with h5netcdf #2181
[x] Fix DataArray.stack() with non-unique coordinates on pandas 0.23 #2168
[ ] Fix dtype=S1 encoding in to_netcdf() #2158

PRs for older bugs:

[x] Fix DataArray.groupby().reduce() mutating input array #2169
[x] Bug fixes for Dataset.reduce() and n-dimensional cumsum/cumprod #2156
[x] Validate output dimension sizes with apply_ufunc #2155
[x] Fix string slice indexing for a length-1 CFTimeIndex #2166
[x] Fix unlimited dims bug #2154

cc @pydata/xarray @crusaderky

OK, that took a little longer than I thought!
I plan to merge https://github.com/pydata/xarray/pull/2158/ and issue the 0.10.5 end of day today unless anyone has objections.

I tagged v0.10.5 and uploaded it to pypi before noticing that due to a bug in our versioneer setup, our version number was v0.10.5 instead of the correct 0.10.5. Oops!
Pypi doesn't allow overwriting files, so I just tagged and uploaded v0.10.6 as well.
Hopefully this version builds properly on conda-forge: https://github.com/conda-forge/xarray-feedstock/pull/33

repo: pydata/xarray


text:
DataArray.loc fails for duplicates where DataFrame works

import pandas as pd
import xarray as xr

df = pd.DataFrame(data=[0, 1], index=list("aa"))
da = xr.DataArray(df)
print(df.loc[list("a")])  # works
print(da.loc[{"dim_0": list("a")}])  # fails

Xarray fails with the exception
Traceback (most recent call last):
  File "session4.py", line 7, in <module>
    print(da.loc[{"dim_0": list("a")}])  # fails
  File "/usr/local/anaconda3/lib/python3.6/site-packages/xarray/core/dataarray.py", line 104, in __getitem__
    return self.data_array.sel(**key)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/xarray/core/dataarray.py", line 784, in sel
    indexers=indexers, drop=drop, method=method, tolerance=tolerance)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/xarray/core/dataset.py", line 1509, in sel
    self, indexers=indexers, method=method, tolerance=tolerance)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/xarray/core/coordinates.py", line 355, in remap_label_indexers
    obj, v_indexers, method=method, tolerance=tolerance
  File "/usr/local/anaconda3/lib/python3.6/site-packages/xarray/core/indexing.py", line 250, in remap_label_indexers
    dim, method, tolerance)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/xarray/core/indexing.py", line 186, in convert_label_indexer
    indexer = get_indexer_nd(index, label, method, tolerance)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/xarray/core/indexing.py", line 117, in get_indexer_nd
    flat_indexer = index.get_indexer(flat_labels, **kwargs)
  File "/usr/local/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 3244, in get_indexer
    raise InvalidIndexError('Reindexing only valid with uniquely'
pandas.core.indexes.base.InvalidIndexError: Reindexing only valid with uniquely valued Index objects

Output of xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_GB.UTF-8
LOCALE: en_GB.UTF-8

xarray: 0.10.8
pandas: 0.23.4
numpy: 1.15.1
scipy: 1.1.0
netCDF4: 1.4.1
h5netcdf: None
h5py: 2.8.0
Nio: None
zarr: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.19.0
distributed: 1.23.0
matplotlib: 2.2.3
cartopy: None
seaborn: 0.9.0
setuptools: 40.2.0
pip: 18.0
conda: 4.5.11
pytest: 3.7.3
IPython: 6.5.0
sphinx: 1.7.7

Thanks for the report!
This was actually a somewhat intentional omission in xarray, but if would not be particularly difficult to add in this feature if we want it. At the very least, we should note this deviation somewhere in the docs.
There are two potentially problematic aspects to the pandas behavior:

It means that you cannot count on indexing a dataframe with its own index to return something equivalent to the original dataframe, e.g., consider df.loc[['a', 'a']] in your example, which returns a dataframe with 4 rows.
More generally, it means you can't count on indexing a dataframe with an array to return an object of the same size as the indexer. This is particularly problematic for xarray, because we support vectorized indexing with multi-dimensional indexers. I don't know how we could define a multi-dimensional equivalent of this -- what shape should the result have if you indexed with a multi-dimensional array instead, e.g., da.loc[{"dim_0": xr.DataArray([['a']]}]? With multiple dimensions involved, it's not clear where the extra introduced dimensions should go.

Now that you bring this up, I wonder how the existing supporting for indexing like da.loc[{"dim_0": "a"}] would work if there are other multi-dimensional indexers. I don't know if we have test coverage for this...

Thanks for the feedback!

You can count on indexing if the is_unique flag is checked beforehand. The way pandas does indexing seems to be both clear to the user and powerful. It seems clear because indexing is the result of a Cartesian product after filtering for matching values. It is powerful because it allows indexing as complex as SQL INNER JOIN, which covers the trivial case of unique elements. For example, the following operation

import pandas as pd

df = pd.DataFrame(data=[0, 1, 2], index=list("aab"))
print(df.loc[list("ab")])
#    0
# a  0
# a  1
# b  2

is an INNER JOIN between the two indexes
INNER((a, b) x (a, a, b)) = INNER(aa, aa, ab, ba, ba, bb)
                          = (aa, aa, bb)

Another example:
import pandas as pd

df = pd.DataFrame(data=[0, 1], index=list("aa"))
print(df.loc[list("aa")])
#    0
# a  0
# a  1
# a  0
# a  1

is again an INNER JOIN between the two indexes
INNER((a, a) x (a, a)) = INNER(aa, aa, aa, aa)
                       = (aa, aa, aa, aa)

Assume a bidimensional array with the following indexing:

  0 1
a ! @
a # $

This translate into an unidimensional index: (a, 0), (a, 1), (a, 0), (a, 1). As such, it can be treated as usual. Assume you index the above matrix using [('a', 0), ('a', 0)]. This implies
INNER( ((a, 0), (a, 0)) x ((a, 0), (a, 1), (a, 0), (a, 1)) ) = INNER( (a,0)(a,0),
    (a,0)(a,1), (a,0)(a,0), (a,0)(a,1), (a,0)(a,0), (a,0)(a,1),
    (a,0)(a,0), (a,0)(a,1) )
  = ((a,0)(a,0), (a,0)(a,0), (a,0)(a,0), (a,0)(a,0))

Converting it back to the matricial representation:
  0 0
a ! !
a # #

In summary, my suggestion is to consider the possibility of defining indexing B by using A (i.e., B.loc(A)) as a Cartesian product followed by match filtering. Or in SQL terms, an INNER JOIN.
The multi-dimensional indexing, as far as I can see, can always be transformed into the uni-dimensional case and treated as such.

Let me give a more concrete example of the issue for multi-dimensional indexing:
da_unique = xr.DataArray([0, 1], dims=['x'], coords={'x': ['a', 'b']})
da_nonunique = xr.DataArray([0, 1], dims=['x'], coords={'x': ['a', 'a']})
indexer = xr.DataArray([['a']], dims=['y', 'z'])

With a unique index, notice how the result takes on the dimensions of the indexer:
>>> da_unique.loc[indexer]
<xarray.DataArray (y: 1, z: 1)>
array([[0]])
Coordinates:
    x        (y, z) object 'a'
Dimensions without coordinates: y, z

What would you propose for the result of da_nonunique.loc[indexer]?

Now I see the problem. But I think it is solvable.
I will ignore the dimension names for now as I don't have
much experience with xarray yet.
The code
da_nonunique = xr.DataArray([0, 1], dims=['x'], coords={'x': ['a', 'a']}
indexer = xr.DataArray([['a']], dims=['y', 'z'])

can be understood as defining two indexed arrays:
[a, a] and [[a]]. As we are allowing for non-unique indexing,
I will denote unique array elements as [e_0, e_1] and [[r_0]]
interchangeably.
Algorithm:

Align. [[a], [a]] and [[a]].
Ravel. [(a,a), (a,a)] and [(a,a)].
Join. [(a,a), (a,a)]. I.e., [e_0, e_1].
Unravel. [[e_0, e_1]]. Notice that [e_0, e_1] has been
picked up by r_0.
Reshape. [[e_0, e_1]] (solution).

Concretely, the solution is a bi-dimensional, 1x2 array:
| 0 1 |.
There is another relevant example. Let the code be
da_nonunique = xr.DataArray([0, 1, 2], dims=['x'], coords={'x': ['a', 'a', 'b']}
indexer = xr.DataArray([['a', 'b']], dims=['y', 'z'])

We have [a, a, b] and [[a, b]], also denoted as [e_0, e_1, e_2]
and [[r_0, r_1]].
Algorithm:

Align. [[a], [a], [b]] and [[a, b]].
Ravel. [(a,a), (a,a), (b,b)] and [(a,a), (b,b)].
Join. [(a,a), (a,a), (b,b)]. I.e., [e_0, e_1, e_2].
Unravel. [[e_0, e_1, e_2]]. Notice now that [e_0, e_1] has been
picked up by r_0 and [e_2] by r_1.
Reshape. [[e_0, e_1, e_2]].

The solution is a bi-dimensional, 1x3 array:
| 0 1 2 |
Explanation

Align recursively adds a new dimension in the array with lower dimensionality.
Ravel recursively removes a dimension by converting elements into tuples.
SQL Join operation: Cartesian product plus match.
Unravel performs the inverse of 2.
Reshape converts it to the indexer's dimensionality.

Please take a look at xarray's detailed indexing rules: http://xarray.pydata.org/en/stable/indexing.html#indexing-rules

I will ignore the dimension names for now as I don't have
much experience with xarray yet.

I think this is the crux of the problem. Put another way: why should the result of indexing be a 1x2 array instead of a 2x1 array? Currently (with the exception of indexing by a scalar with an index with duplicates), xarray determines the shape/dimensions resulting from indexing from the shape/dimensions of the indexers not the array being indexed.

I see. Now I read about it, let me give another shot.
Let i be
<xarray.DataArray (y: 1, z: 1)>
array([['a']], dtype='<U1')
Dimensions without coordinates: y, z

and d be
<xarray.DataArray (x: 2)>
array([0, 1])
Coordinates:
  * x        (x) <U1 'a' 'a'

The result of d.loc[i] is equal to d.sel(x=i). Also, it seems reasonable to expect the its result should be the same as d0.sel(x=i) for d0 given by
<xarray.DataArray (x: 2, dim_1: 1)>
array([[0],
       [1]])
Coordinates:
  * x        (x) <U1 'a' 'a'
Dimensions without coordinates: dim_1

as per column vector representation assumption.
Answer
Laying down the first dimension gives

y
z
x

a
a
a

a

By order, x will match with y and therefore we will append a new dimension after x to
match with z:

y
z
x
dim_1

a
a
a
?

a
?

where ? means any. Joining the first and second halves of the table gives

y
z
x
dim_1

a
a
a
?

a
a
a
?

And here is my suggestions. Use the mapping y|->x and z|->dim_1 to decide which
axis to expand for the additional element. I will choose y-axis because the additional a was
originally appended to the x-axis.
The answer is
<xarray.DataArray (y: 2, z: 1)>
array([[0],
       [1]])
Coordinates:
    x        (y, z) <U1 'a' 'a'
Dimensions without coordinates: y, z

for
>>> ans.coords["x"]
<xarray.DataArray 'x' (y: 2, z: 1)>
array([['a'],
       ['a']], dtype='<U1')
Coordinates:
    x        (y, z) <U1 'a' 'a'
Dimensions without coordinates: y, z

Hi again. I'm working on a precise definition of xarray and indexing. I find the official one a bit hard to understand. It might help me come up with a reasonable way to handle duplicate indices. https://drive.google.com/file/d/1uJ_U6nedkNe916SMViuVKlkGwPX-mGK7/view?usp=sharing

CC @fujiisoup who implemented much of this. I will also take a look at your doc when I have the chance.
I do think that handling duplicate matches with indexing is an important use-case. This comes up with nearest neighbor matching as well -- it would be useful to be able to return the full set of matches within a given distance, not just the nearest match.
I wonder if it would be more productive to consider a new indexing API for one -> many matches. sel/loc is already quite complex.

Sorry that I couldn't join the discussion here.
Thanks, @horta, for giving the nice document.
We tried to use the consistent terminology in the docs, but I agree that it would be nice to have a list of the definitions.
I think it might be better to discuss in another issue. See #2410.
For loc and sel issues.
One thing I don't agree is

The result of d.loc[i] is equal to d.sel(x=i). Also, it seems reasonable to expect the its result should be the same as d0.sel(x=i) for d0 given by

As xarray inherits not only from pandas but also from numpy's multi-dimensional array.
That is, we need to be very consistent with the resultant shape of indexing.
It would be confusing if a selection from different dimensional arrays becomes the same.

I do think that handling duplicate matches with indexing is an important use-case. This comes up with nearest neighbor matching as well -- it would be useful to be able to return the full set of matches within a given distance, not just the nearest match.

I also think that what is lacking in xarray is this functionality.
Any interest to help us for this?

authors: fujiisoup horta shoyer


text:
open_rasterio does not read coordinates from netCDF file properly with netCDF4>=1.4.2

MCVE Code Sample

Adapted from the test_serialization unit test at here.
import xarray as xr
from xarray.tests.test_backends import assert_identical, create_tmp_geotiff, create_tmp_file

with create_tmp_geotiff(additional_attrs={}) as (tmp_file, expected):
    # write it to a netcdf and read again (roundtrip) using open_rasterio
    with xr.open_rasterio(tmp_file) as rioda:
        with create_tmp_file(suffix='.nc') as tmp_nc_file:
            # Write geotiff to netcdf file
            rioda.to_netcdf(tmp_nc_file)

            # Read using open_dataarray works
            with xr.open_dataarray(tmp_nc_file) as ncds:
                assert_identical(rioda, ncds)

            # Read using open_rasterio doesn't work!!
            with xr.open_rasterio(tmp_nc_file) as ncds:
                assert_identical(rioda, ncds)

Actual Output (using netCDF4>=1.4.2)
AssertionError: Left and right DataArray objects are not identical

Differing coordinates:
L * x        (x) float64 5.5e+03 6.5e+03 7.5e+03 8.5e+03
R * x        (x) float64 0.5 1.5 2.5 3.5
L * y        (y) float64 7.9e+04 7.7e+04 7.5e+04
R * y        (y) float64 0.5 1.5 2.5
Differing attributes:
L   transform: (1000.0, 0.0, 5000.0, 0.0, -2000.0, 80000.0)
R   transform: (1.0, 0.0, 0.0, 0.0, 1.0, 0.0)
L   res: (1000.0, 2000.0)
R   res: (1.0, -1.0)
Attributes only on the left object:
    crs: +init=epsg:32618

Expected Output (using netCDF4==1.4.1)
AssertionError: Left and right DataArray objects are not identical

Differing attributes:
L   nodatavals: (nan, nan, nan)
R   nodatavals: (nan, nan, nan)
Attributes only on the left object:
    crs: +init=epsg:32618

Problem Description

I have a script which takes in either NetCDF or GeoTIFF files as an input and I've been using xr.open_rasterio to read them quite successfully before, as I'm basically just interested in parsing the correct XY coordinates out. However, upgrading from NetCDF 1.4.1 to 1.4.2 breaks this behaviour (xr.open_rasterio no longer parses the coordinates properly from a .nc file).
My hunch is that it has something to do with different NetCDF4 formats (e.g. ‘NETCDF4’, 'NETCDF4_CLASSIC’, ‘NETCDF3_64BIT’) but looking at the xr.open_rasterio code, I'm not too sure what's going on... Also not very sure if this is an upstream netcdf4-python or rasterio issue but I thought I'd report it here first.
Output of xr.show_versions()

INSTALLED VERSIONS
commit: None
python: 3.6.7 | packaged by conda-forge | (default, Jul  2 2019, 02:18:42)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 4.14.127+
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: C.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.8.18
libnetcdf: 4.4.1.1
xarray: 0.12.3
pandas: 0.25.0
numpy: 1.17.0rc2
scipy: 1.3.0
netCDF4: 1.4.1
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.0.24
cfgrib: None
iris: None
bottleneck: None
dask: 2.1.0
distributed: None
matplotlib: 3.1.1
cartopy: None
seaborn: None
numbagg: None
setuptools: 41.0.1
pip: 19.2.1
conda: None
pytest: 5.0.1
IPython: 7.6.1
sphinx: None

Thanks @weiji14 for opening this issue. When we implemented open_rasterio, I don't think we intended this to be used for netCDF data so I'm surprised it works (worked) at all. I think we'll need someone to dig into the xarray implementation a bit to see if we can pull out the correct coordinates (metadata) using rasterio.

Well open_rasterio did have an "experimental" warning on it in the docs :laughing:, but it was really nice having it work on GeoTIFFs and NetCDF files. I've forked the repo and will try to debug the situation a bit more. If anyone who's worked on that part of the codebase before has any pointers on what might be the cause of this issue / where to start that would be great.

Hold on, the coordinates seems to be parsed out correctly from the netCDF file (even with netCDF==1.5.1.2) when I have a clean conda installation created following the instructions at https://xarray.pydata.org/en/latest/contributing.html#creating-a-python-environment.
I've isolated the issue and think the problem arises when I also import salem (an xarray accessor)... Will try to narrow this down before I close this issue.

I guess GDAL can also read netCDF files :).
Indeed, we did not anticipate people using open_rasterio this way.

GDAL can open so many formats (https://gdal.org/drivers/raster/index.html). We've really only tested against TIFF like formats though. I think the assumption will need to be is that rasterio will provide standard metadata for each of these drivers.

Yes, there's https://gdal.org/drivers/raster/netcdf.html :smile: I've done a bit more debugging (having temporarily isolated salem from my script) and am still having issues with my setup.
The clean xarray-tests conda environment that works with netcdf==1.5.1.2 has libnetcdf: 4.6.2, but for some strange reason, running xr.show_versions() on my setup shows libnetcdf: 4.6.3 even though conda list | grep libnetcdf shows that I've installed libnetcdf  4.6.2  h056eaf5_1002  conda-forge.
Not sure if this libnetcdf 4.6.3 version is the problem, but it stands out the most (to me at least) when looking at the diff between my setup and the clean one. Is there a way to check the order in which xarray looks for the netcdf binaries as I feel it might be a PATH related issue. Also not sure if this issue fits here in xarray or somewhere else now...

The clean xarray-tests conda environment that works with netcdf==1.5.1.2 has libnetcdf: 4.6.2, but for some strange reason, running xr.show_versions() on my setup shows libnetcdf: 4.6.3 even though conda list | grep libnetcdf shows that I've installed libnetcdf 4.6.2 h056eaf5_1002 conda-forge.

xr.show_versions() gets its libnetcdf version from netCDF4 (specially netCDF4.__netcdf4libversion__). So I'm guessing that somehow netCDF4 is picking up libnetcdf from somewhere else -- maybe you pip installed it? It might be worth trying another fresh conda environment...

xr.show_versions() gets its libnetcdf version from netCDF4 (specially netCDF4.__netcdf4libversion__). So I'm guessing that somehow netCDF4 is picking up libnetcdf from somewhere else -- maybe you pip installed it? It might be worth trying another fresh conda environment...

I'm not sure where it's picking up the libnetcdf 4.6.3 version from, but I found your comment at https://github.com/pydata/xarray/issues/2535#issuecomment-445944261 and think it might indeed be an incompatibility issue with rasterio and netCDF4 binary wheels (do rasterio wheels include netcdf binaries?). Probably somewhat related to https://github.com/mapbox/rasterio/issues/1574 too.
Managed to get things to work by combining the workaround in this Pull Request and StackOverflow post, basically having pip compile the netcdf python package from source instead of using the wheel:
HDF5_DIR=$CONDA_PREFIX pip install --no-binary netCDF4 netCDF4==1.4.2

where $CONDA_PREFIX is the path to the conda environment e.g. /home/jovyan/.conda/envs/name-of-env. I've tested my MCVE code sample above and it works up to the latest netCDF4==1.5.1.2 version!

I'm seeing a problem with geotiffs that I think might be related:
$ gdalinfo /home/nedcr/cr/software/ana/lib/geo/tests/data/Urandangi_MGA55.tif
Driver: GTiff/GeoTIFF
Files: /home/nedcr/cr/software/ana/lib/geo/tests/data/Urandangi_MGA55.tif
       /home/nedcr/cr/software/ana/lib/geo/tests/data/Urandangi_MGA55.tif.aux.xml
Size is 10, 10
Coordinate System is:
PROJCS["GDA94 / MGA zone 55",
    GEOGCS["GDA94",
        DATUM["Geocentric_Datum_of_Australia_1994",
            SPHEROID["GRS 1980",6378137,298.257222101,
                AUTHORITY["EPSG","7019"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6283"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4283"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",147],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","28355"]]
Origin = (-406507.954209543997422,7588834.152862589806318)
Pixel Size = (263.500000000000000,-265.500000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( -406507.954, 7588834.153) (138d16' 6.99"E, 21d34'24.95"S)
Lower Left  ( -406507.954, 7586179.153) (138d16' 1.84"E, 21d35'50.30"S)
Upper Right ( -403872.954, 7588834.153) (138d17'37.56"E, 21d34'29.73"S)
Lower Right ( -403872.954, 7586179.153) (138d17'32.42"E, 21d35'55.09"S)
Center      ( -405190.454, 7587506.653) (138d16'49.70"E, 21d35'10.02"S)
Band 1 Block=10x10 Type=Float32, ColorInterp=Gray
  Min=0.069 Max=8.066
  Minimum=0.069, Maximum=8.066, Mean=2.556, StdDev=1.749
  NoData Value=-3.40282306073709653e+38
  Metadata:
    STATISTICS_MAXIMUM=8.06591796875
    STATISTICS_MEAN=2.5563781395387
    STATISTICS_MINIMUM=0.068740844726562
    STATISTICS_STDDEV=1.7493082797107
    STATISTICS_VALID_PERCENT=89

import xarray as xr
from osgeo.gdal import Open

ras = Open('/home/nedcr/cr/software/ana/lib/geo/tests/data/Urandangi_MGA55.tif')
ras.GetGeoTransform()
# (-406507.954209544, 263.5, 0.0, 7588834.15286259, 0.0, -265.5)

ds = xr.open_rasterio('/home/nedcr/cr/software/ana/lib/geo/tests/data/Urandangi_MGA55.tif')
ds.transform
# (263.5, 0.0, -406507.954209544, 0.0, -265.5, 7588834.15286259)

The transform in the xarray dataset is transposed, and not really useful anymore.
Output of xr.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 5.0.0-37-lowlatency
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_AU.UTF-8
LOCALE: en_AU.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.2
xarray: 0.14.1
pandas: 0.25.1
numpy: 1.16.4
scipy: 1.3.0
netCDF4: 1.5.1.2
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.0.22
cfgrib: None
iris: None
bottleneck: None
dask: 1.2.0
distributed: 1.27.1
matplotlib: 3.0.3
cartopy: 0.17.0
seaborn: None
numbagg: None
setuptools: 40.8.0
pip: 19.0.3
conda: None
pytest: 4.3.1
IPython: 7.3.0
sphinx: 2.1.2

@naught101 Looks like those coordinates are now in the order as declared in the affine package (since rasterio >= 1.0). See also: https://rasterio.readthedocs.io/en/latest/topics/migrating-to-v1.html.
All coordinates are created correctly, so from my point of view, only the docs here have to be changed.
from affine import Affine
da = xr.open_rasterio('path_to_file.tif')
transform = Affine(*da.attrs['transform'])
nx, ny = da.sizes['x'], da.sizes['y']
x, y = np.meshgrid(np.arange(nx)+0.5, np.arange(ny)+0.5) * transform

yes, I would think so. Thanks for the reminder, @gabriel-abrahao.
The original issue was that libnetcdf picked up by netCDF4 didn't match the version installed by the netCDF4 wheels, which is not really xarray's fault.

authors: jhamman keewis kmuehlbauer naught101 shoyer weiji14


text:
WIP: Rolling minimum dependency versions policy

Closes #3222
Closes #3293

Drop support for Python 3.5
Downgrade numpy to 1.14, pandas to 0.21, scipy to 1.0 (24 months old)
Downgrade dask to 1.2 (6 months old)
Other upgrades/downgrades to comply with the policy
CI tool to verify that the minimum dependencies requirements in CI are compliant with the policy

TODO:

documentation
seek and destroy all hardcoded versions in documentation and comments
test failures:  AttributeError: module 'pandas.core.common' has no attribute 'count_not_none'
test failures: test_strftime_of_cftime_array

(help is very welcome!)
Out of scope:

Purge away all OrderedDict's
Clean up# type: annotations for instance variables

A note: it seems impossible to install dask >=2.0 together with these old versions of numpy/pandas - at least with conda. I did not investigate why.

Hello @crusaderky! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file ci/min_deps_check.py:

Line 5:1: F401 'os.path' imported but unused

In the file xarray/core/dataarray.py:

Line 2:1: F401 'sys' imported but unused

I was bored, so I wrote a CI tool to keep track of the situation :)
Package       Required          Policy            Status
------------- ----------------- ----------------- ------
python        3.6  (2016-12-23) 3.6  (2016-12-23) =
boto3         1.9  (2018-09-10) 1.9  (2018-09-10) =
bottleneck    1.2  (2017-09-14) 1.2  (2017-09-14) =
cartopy       0.17 (2018-11-17) 0.17 (2018-11-17) =
cdms2         3.1  (2018-11-02) 3.1  (2018-11-02) =
cfgrib        0.9  (2019-02-25) 0.9  (2019-02-25) =
cftime        1.0  (2018-04-27) 1.0  (2018-04-27) =
dask          1.2  (2019-04-13) 1.2  (2019-04-13) =
distributed   1.27 (2019-04-12) 1.27 (2019-04-12) =
h5netcdf      0.7  (2019-02-27) 0.7  (2019-02-27) =
h5py          2.9  (2018-12-20) 2.10 (2019-09-10) <
hdf5          1.10 (2017-09-22) 1.10 (2017-09-22) =
iris          2.2  (2018-10-12) 2.2  (2018-10-12) =
lxml          4.4  (2019-07-27) 4.4  (2019-07-27) =
matplotlib    3.1  (2019-05-18) 3.1  (2019-05-18) =
nc-time-axis  1.2  (2019-01-25) 1.2  (2019-01-25) =
netcdf4       1.4  (2018-05-08) 1.4  (2018-05-08) =
numba         0.44 (2019-06-18) 0.44 (2019-06-18) =
numpy         1.14 (2018-01-09) 1.14 (2018-01-09) =
pandas        0.21 (2017-11-04) 0.21 (2017-11-04) =
pseudonetcdf  3.0  (2019-07-18) 3.0  (2019-07-18) =
pydap         3.2  (2018-10-16) 3.2  (2018-10-16) =
pynio         1.5  (2017-12-01) 1.5  (2017-12-01) =
rasterio      1.0  (2018-12-21) 1.0  (2018-12-21) =
scipy         1.0  (2017-11-07) 1.0  (2017-11-07) =
seaborn       0.9  (2018-07-17) 0.9  (2018-07-17) =
toolz         0.10 (2019-07-12) 0.10 (2019-07-12) =
zarr          2.3  (2019-03-25) 2.3  (2019-03-25) =

That's awesome, I've never seen this done so systematically—great idea!
Let's put dropping 3.5 fairly prominently in the release notes
Thanks @crusaderky !

Missing only CI integration of the new tool; waiting for #3366. All tests pass locally. You can start reviewing...

Evidence of the new CI script at work:
https://dev.azure.com/xarray/xarray/_build/results?buildId=975&view=logs

I have one outstanding error
>       abs_diff = np.atleast_1d(abs(actual - expected)).astype(np.timedelta64)
E       TypeError: Cannot cast datetime.timedelta object from metadata [Y] to [us] according to the rule 'same_kind'

xarray/tests/test_coding_times.py:122: TypeError

The error disappears if I change the repositories of py36-min-all-deps from [conda-forge] to [anaconda, conda-forge]. I tried comparing the two conda list but I could not spot any glaring differences. Help appreciated!

@shoyer how do we test the readthedocs CI build before merging into main?

Thanks @crusaderky . Can you replace the last line with just loffset=loffset
https://github.com/pydata/xarray/blob/dd2b803a28cdf2f36210f5a7a46897cc47584ea6/xarray/core/common.py#L1043-L1049

Ready for final review and merge.
I'm trying to test the readthedocs CI but it's getting stuck:
https://readthedocs.org/projects/crusaderky-xarray/builds/
Maybe it's because I'm not a paying user?

I'm getting a crash in readthedocs because of versioneer:
https://readthedocs.org/projects/crusaderky-xarray/builds/9748865/
In the RTD CI, versioneer incorrectly sets the version as 0+untagged.111.g6d60700, which while it is compliant to PEP-440, it obviously fails a minimum version check in pandas.DataFrame.to_xarray.
On top of this, there's a bug in distutils (!!!) that causes an ugly crash instead of a clean error message:
pandas/compat/_optional.py(100)import_optional_dependency()

     98     if minimum_version:
     99         version = _get_version(module)
--> 100         if distutils.version.LooseVersion(version) < minimum_version:
    101             assert on_version in {"warn", "raise", "ignore"}

if distutils.version.LooseVersion(version) < minimum_version:
To replicate locally: manually
xarray: 0+untagged.111.g6d60700
[...]
>>>-------------------------------------------------------------------------
Exception in /home/docs/checkouts/readthedocs.org/user_builds/crusaderky-xarray/checkouts/mindeps/doc/pandas.rst at block ending on line 105
Specify :okexcept: as an option in the ipython:: block to suppress this message
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-7f86783d8836> in <module>
----> 1 s[::2].to_xarray()

~/checkouts/readthedocs.org/user_builds/crusaderky-xarray/conda/mindeps/lib/python3.7/site-packages/pandas/core/generic.py in to_xarray(self)
   2904             speed    (date, animal) int64 350 18 361 15
   2905         """
-> 2906         xarray = import_optional_dependency("xarray")
   2907
   2908         if self.ndim == 1:

~/checkouts/readthedocs.org/user_builds/crusaderky-xarray/conda/mindeps/lib/python3.7/site-packages/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version)
     98     if minimum_version:
     99         version = _get_version(module)
--> 100         if distutils.version.LooseVersion(version) < minimum_version:
    101             assert on_version in {"warn", "raise", "ignore"}
    102             msg = version_message.format(

~/checkouts/readthedocs.org/user_builds/crusaderky-xarray/conda/mindeps/lib/python3.7/distutils/version.py in __lt__(self, other)
     50
     51     def __lt__(self, other):
---> 52         c = self._cmp(other)
     53         if c is NotImplemented:
     54             return c

~/checkouts/readthedocs.org/user_builds/crusaderky-xarray/conda/mindeps/lib/python3.7/distutils/version.py in _cmp(self, other)
    335         if self.version == other.version:
    336             return 0
--> 337         if self.version < other.version:
    338             return -1
    339         if self.version > other.version:

TypeError: '<' not supported between instances of 'str' and 'int'

Not sure if the above is caused by the fact that I'm not running RTD in the main xarray repo?

I think I worked around it - logged in #3369

RTD build successful!
https://crusaderky-xarray.readthedocs.io/en/mindeps

This is super! I think it'd be fine to merge before all the dependency changes are in; I see this is getting to be a mammoth PR.
I'm +0.5 on moving to 0.14 given the benefits of this and the breaking changes. Any other thoughts?

The only thing I see is the big bump in matplotlib. Should we make that a 12 month policy like scipy and pandas? ( I don't have strong opinions on this)

@crusaderky - this is getting hard to review at this point. Is there more to do or can we merge this in its current state?

I'm +0.5 on moving to 0.14 given the benefits of this and the breaking changes. Any other thoughts?

See #3380

@jhamman it's ready for merge if everybody's happy with it

Okay, planning to merge c.COB tomorrow unless there are additional comments.

@shoyer any comments from you?

Thanks @crusaderky. Good to have this policy solidified.

Thanks all!

Thanks @crusaderky. That was a major lift.

Thanks @crusaderky !
+1 to @dcherian 's comment!

authors: crusaderky dcherian jhamman max-sixty pep8speaks


text:
Fix duckdb example documentation

Checklist

[ ] Added a docs/source/changelog.md entry

@NMAC427 @nicolasmueller I will fix this documentation hickup right away for readthedocs (push to main is sufficient). But I won't take this as a reason to make a new release. The documentation on pypi is wrong though until until 0.7.1 is released for other reasons.

I tested both pip and conda-forge installation techniques and now it works for me at least. CI tested it previously as well, but there was a pipedag.yaml in the discovery path which wasn't supposed to be used.


text:
Units with onto

Now I implemented what @liamhuber and I talked about.
@units
def get_speed_onto(
    distance: u(float, "meter"), time: u(float, "second")
) -> u(float, "meter/second"):
    return distance / time

The functionality is the same as shown in the previous PRs. What happens internally is:
print(u(float, "meter/second"))

Output: typing.Annotated[float, {'units': 'meter/second', 'otype': None}]
And it's up to the parser to figure out what to do with the additional information.

Pull Request Test Coverage Report for Build 10845198040
Details

0 of 0   changed or added relevant lines in 0 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.03%) to 98.099%

Files with Coverage Reduction
New Missed Lines
%

units.py
1
99.13%

Totals

Change from base Build 10844452784:
0.03%

Covered Lines:
671

Relevant Lines:
684

💛  - Coveralls

So this one works only for python 3.11 and above. That's fine by me but maybe not so interesting for other packages...

So this one works only for python 3.11 and above. That's fine by me but maybe not so interesting for other packages...

Have you checked if from future import annotations magically fixes it for us?

So this one works only for python 3.11 and above. That's fine by me but maybe not so interesting for other packages...

Have you checked if from future import annotations magically fixes it for us?

Ok, I checked if from __future__ import annotations fixes #49 and it doesn't. I don't know what the source of trouble is

Sorry for a couple of reasons I couldn't work on this today. I'll come back to it probably on Monday. Anyway thanks for the suggestion!

Sorry for a couple of reasons I couldn't work on this today. I'll come back to it probably on Monday. Anyway thanks for the suggestion!

No worries! I had two minutes waiting for a meeting to start and it was a super easy push. I'll still leave debugging the tricky stuff to you anyhow 😂

Interesting. duration: u(float | None, "second") = 0 apparently works but duration: u(float | None, "second") = None not.

In the end I came back to the same problem...

Now it boils down to the conversion, as we encountered in #49

In the end I implemented it here, and since I don't have the feeling that this one is going to pass, I suggest to close it. What do you think? @liamhuber

In the end I implemented it here, and since I don't have the feeling that this one is going to pass, I suggest to close it. What do you think? @liamhuber

Sounds good to me. The linked on looks good. I'd still be interested to move towards a single dataclass (or maybe pydantic model?) to hold the pyiron-allowable annotations, but that's some that can and probably should be left for the future as we develop some experience.

authors: coveralls liamhuber samwaseda

repo: pyiron/elaston


text:
Get rid of double deploy entry in Travis config file

http://stackoverflow.com/questions/36539398/deploying-different-files-based-on-build-os/ could work.

Worked with a test release.

repo: pylaligand/dart-sqlite

license_source: bigquery


text:
Forward sampling returning an empty dictionary

I am trying a forward sampling on mixture model example from pymc4 examples, but getting an empty dict as a result.
The model is defined as -
@pm.model
def mixture(n_groups, n_points):
    centers = pm.Normal('centers', loc=tf.zeros((n_groups, 2)), scale=1)
    scales = pm.HalfNormal('scales', 0.4 * tf.ones(n_groups))
    rates = pm.Dirichlet('rates', concentration=tf.ones(n_groups))

    group_assignments = pm.Multinomial('group_assignments', total_count=n_points, probs=rates)

    for idx in range(n_groups):
        count = tf.to_int32(group_assignments[idx])
        pm.Normal(f'group_{idx}', loc=centers[idx] * tf.ones((count, 2)), scale=scales[idx])

# Now I configured the model -
model = mixture.configure(n_groups=5, n_points=10_000)

I checked the tensorflow graph but it's not empty. The graph node names are-
['zeros',
'centers/scale',
'centers/Identity',
'centers/Identity_1',
'ones/shape_as_tensor',
'ones/Const',
'ones',
'mul/x',
'mul',
'scales/scale',............]
What could be the reasons ?

Closing as this relates to the old context based API. It would be nice to check out this example in the current coroutine API though

authors: Adarsh-kumar lucianopaz


text:
Adding 'Doing Bayesian Data Analysis'?

Are you guys planning to add a new resource to this page? The book I have in mind is 'Doing Bayesian Data Analysis' from John K. Kruschke. There are at least two repositories in github.com where you can find pymc3 as a replacement for Stan:

https://github.com/aloctavodia/Doing_bayesian_data_analysis
https://github.com/JWarmenhoven/DBDA-python

That's a great idea, cc @aloctavodia @jwarmenhoven.

What will be the benefit of migrating these repositories? They are referenced in the PyMC3 documentation. http://docs.pymc.io/#learn-bayesian-statistics-with-a-book-together-with-pymc3

My port of the first edition is outdated, if we are going to move it here we should updated. Maybe we should focus all efforts on @JWarmenhoven port of the second edition.
When I suggested moving statistical rethinking code under pymcdevs I was thinking about long term maintenance and also I did not like to have a repository under my number with several other contributors, putting inside and organization sounds like a better choice.

@aloctavodia In that case you should probably add a link to @JWarmenhoven's version.

@twiecki good idea! Done it. I also just realize that I did a code update six month ago, not that bad! Will check if another one is necessary.

@rosgori Are you still interesting in adding this book? If not I think we can just close this issue.

You can close this issue.

authors: JWarmenhoven aloctavodia junpenglao rosgori twiecki

repo: pymc-devs/resources


text:
Unhandled Runtime Error TypeError: Type error

Describe the bug
A clear and concise description of what the bug is.
Im not sure this is M1 issue or not.
Have tested in intel, it work!!
Once use event, I always meet Unhandled Runtime Error TypeError: Type error.
To Reproduce
Steps to reproduce the behavior:

Code/Link to Repo:

import pynecone as pc

class WordCycleState(pc.State):
    # The words to cycle through.
    text = ["Welcome", "to", "Pynecone", "!"]

    # The index of the current word.
    index = 0

    def next_word(self):
        self.index = (self.index + 1) % len(self.text)

    @pc.var
    def get_text(self):
        return self.text[self.index]

def index():
    return pc.heading(
        WordCycleState.get_text,
        on_mouse_over=WordCycleState.next_word,
        color="green",
    )

app = pc.App(state=WordCycleState)
app.add_page(index)
app.compile()

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.

** Specifics (please complete the following information):**

Device: Apple M1
OS Darwin 21.6.0
Pynecone 0.1.30
Python 3.8.16
Node v20.2.0
Bun 0.6.1

Additional context
Add any other context about the problem here.

It can work after reinitialize.
related: https://stackoverflow.com/questions/72220659/react-next-js-and-type-script-unhandled-runtime-error-typeerror-cannot-read-pr

repo: pynecone-io/pynecone


text:
Upload component picking up image

Hello,
I am trying to implement an image upload component but seem to be getting the following error. I am implementing the exact same code as in the docs. Here is a screenshot of my error:

Cancel this... I just needed to upgrade the version and run pc init again.


text:
Enable ~= syntax when checking Py2 compatibility

This should fix #476.
I do not remember full PEP 440, so I am not sure whether there are other valid expressions to disallow Python 2 which would still be false-positive in flit.
Probably it would be better to use specifiers from packaging library so that flit does not need to implement own specifier parsing. Therefore this is just a quick fix for the issue at hand, which makes a small improvement but probably is not final.

Thanks! Can you add a case or two to the test for this:
https://github.com/pypa/flit/blob/f86127764bddb05cb6b5d9a1b6ae4346f786c94a/flit_core/flit_core/tests/test_common.py#L106-L122

@LEdoian flit cannot depend on packaging because packaging needs (or will need if everything goes as expected) flit to build.

@takluyver: Done.

@nanonyme: That seems unfortunate, but I understand this is not worth complicating the bootstrapping processes.

It wouldn't just complicate it but make it impossible. This package is at the very bottom of the food chain, it cannot have any external dependencies.

I do not think it would be outright impossible, but I do not have a simple and elegant solution.
The idea would be to try importing packaging. If it succeeds, use the full specifier matching, if not, fall back to the current regex one with its limitations. This way packaging could be built and everyone else can use full-fledged version (since once packaging is built, it does not need flit anymore).
But this would be needlessly complex and awful from software engineering standpoint, especially given that flit tries to be the simple option. So not worth it.

And yes, I agree with you that trying the import and falling back to a simpler implementation is not desirable - it's extra complexity, and people will still want to fix the bugs with the fallback option.
I think doing a subset of proper version specifier handling inside flit_core is feasible, because a) it only needs to handle Python's predictable X.Y.Z version numbers (I checked the code in pip, that ignores anything beyond the three numbers, so we can do the same), and b) we don't need the nice SpecifierSet API that packaging offers.

(To be clear, I'm not saying you need to do that in this PR, just thinking about the direction we might go in)

I've just merged #593, which solved the same thing.

authors: LEdoian nanonyme takluyver

repo: pypa/flit