text:
Idea: functionally-derived non-dimensional coordinates

@Cadair and I are from the solar and astrophysics communities, respectively (particularly SunPy and Astropy). In our fields, we have a concept of something called "World Coordinate Systems" (WCS) which basically are arbitrary mappings from pixel coordinates (which is often but not necessarily the same as the index) to physical coordinates. (For more on this and associated Python/Astropy APIs, see this document). If we are reading correctly, this concept maps roughly onto the xarray concept of "Non-dimension coordinates".
However, a critical difference is this: WCS are usually expressed as arbitrary complex mathematical functions, rather than coordinate arrays, as it is crucial to some of the science cases to carry sub-pixel or other coordinate-related metadata around along with the WCS.
So our question is: is it in-scope for xarray non-dimensional coordinates to be extended to be functional instead of having to be arrays? I.e., have the coordinate arrays be generated on-the-fly from some function instead of being realized as arrays at creation-time. We have thought about several ways this might be specified and are willing to do some trial implementations, but are first asking here if it is likely to be

Easy
Hard
Impossible
PR will immediately be rejected on philosophical grounds, regardless?

Thanks!

Thanks for reaching out Erik! We’d love to find a way to better support Astro data in xarray. Before digging deeper, I just want to ask a clarification question. When you say “arbitrary complex mathematical functions”: what are the arguments / inputs to these functions?
Presumably they have to be evaluated at some point, ie for plotting. Can you describe what happens when the time comes to turn the arbitrary functions to actual numbers?
Your answer will help us respond more accurately to your question.

On Dec 13, 2019, at 3:51 PM, Erik Tollerud notifications@github.com wrote:

@Cadair and I are from the solar and astrophysics communities, respectively (particularly SunPy and Astropy). In our fields, we have a concept of something called "World Coordinate Systems" (WCS) which basically are arbitrary mappings from pixel coordinates (which is often but not necessarily the same as the index) to physical coordinates. (For more on this and associated Python/Astropy APIs, see this document). If we are reading correctly, this concept maps roughly onto the xarray concept of "Non-dimension coordinates".
However, a critical difference is this: WCS are usually expressed as arbitrary complex mathematical functions, rather than coordinate arrays, as it is crucial to some of the science cases to carry sub-pixel or other coordinate-related metadata around along with the WCS.
So our question is: is it in-scope for xarray non-dimensional coordinates to be extended to be functional instead of having to be arrays? I.e., have the coordinate arrays be generated on-the-fly from some function instead of being realized as arrays at creation-time. We have thought about several ways this might be specified and are willing to do some trial implementations, but are first asking here if it is likely to be
Easy
Hard
Impossible
PR will immediately be rejected on philosophical grounds, regardless?
Thanks!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

It would also be good to hear about "sub-pixel metadata" → this seems to be the main reason why you want to carry around the analytic rather than the discrete evaluated form (which we basically support through dask). Is that right or am I missing something?

Thanks for the quick responses @rabernat and @dcherian!

When you say “arbitrary complex mathematical functions”: what are the arguments / inputs to these functions?

The short answer is: pixels. Which in astro is sometimes the same as "array index", but usually off-by-0.5 (i.e., the center of the pixel), and occasionally off-by-1 (for files generated by e.g. FORTRAN or other 1-based indexing schemes...).
The longer answer is: it depends on which direction you mean.  Most WCS' are by design invertible, so you can go world-to-pixel or pixel-to-world.  In the former case that means there's a bunch of possibly inputs - examples include wavelength (for a spectrum), energy (for a photon-counting experiment), or altitude and azimuth (celestial coordinates that are similar to lat/lon but on the celestial sphere instead of a reference geoid).
Additionally, the WCSs have parameters, which are usually not thought of as "inputs" to the WCS, but rather internal state or metadata (usually they are stored in file headers and interpreted by software).
If you want to see some concrete examples, you might check out Astropy's docs for our WCS interface, or the design document for the most recent iteration of that interface.  But those might be impenetrable for a non-astronomer, so I'm open to more follow-ups!

It would also be good to hear about "sub-pixel metadata"

Sorry I wasn't clear there - I meant there are cases where the sub-pixel structure of the coordinates matter - e.g. telescopes often have pretty intense optical distortion, and we sometimes interpolate in such a way that the sub-pixel information in the WCS is critical to getting the interpolation right.  And by "metadata" I meant both abstract parameters that encode that transformation (polynominal coefficients, information about which sky-projection is used, etc), and more physical metadata like "the temperature of the observatory, which slightly changes the shape of the distortion".

which we basically support through dask

Oh, tell me more about that - maybe this is the key for our needs?

I should have said "discrete lazily evaluated form (which we support through dask)". I think we already have what you want in principle (caveats at the end).
Here's an example:
import dask
import numpy as np
import xarray as xr

xr.set_options(display_style="html")

def arbitrary_function(dataset):
    return dataset["a"] * dataset["wavelength"] * dataset.attrs["wcs_param"]

ds = xr.Dataset()
# construct a dask array.
# In practice this could represent an on-disk dataset,
# with data reads only occurring when necessary
ds["a"] = xr.DataArray(dask.array.ones((10,)), dims=["wavelength"], coords={"wavelength": np.arange(10)})

# some coordinate system parameter
ds.attrs["wcs_param"] = 1.0

# complicated pixel to world function
# no compute happens since we are working with dask arrays
# so this is quite cheap.
ds.coords["azimuth"] = arbitrary_function(ds)
ds

So you can carry around your coordinate system parameters in the .attrs dictionary and the non-dimensional coordinate azimuth is only evaluated when needed e.g. when plotting
# Both 'a' and 'azimuth' are computed now, since actual values are required to plot
ds.a.plot(x="azimuth")

In practice, there are a few limitations. @djhoese and @snowman2 may have useful perspective here.

xarray tends to compute "non-dimensional coordinates" more than necessary. The more egregious examples have been fixed  (#3068, #3311, #3454, #3453) but there may still be some places where fixes are needed (#3588).
there's some discussion about carrying around Earth-specific coordinate system parameters here: https://github.com/pydata/xarray/issues/2288; https://github.com/pydata/xarray/issues/2996.

Additional info:

https://docs.dask.org/en/latest/array.html
https://xarray.pydata.org/en/stable/dask.html

PS: If it helps, I'd be happy to chat over skype for a half hour getting you oriented with how we do things.

For reference, here is how CRS information is handled in rioxarray: CRS management docs.

For reference, here is how CRS information is handled in rioxarray: CRS management docs.

Nice! I didn't know you had documented this.
Sorry this is going to get long. I'd like to describe the CRS stuff we deal with and the lessons learned and the decisions I've been fighting with in the geoxarray project (https://github.com/geoxarray/geoxarray). I'm looking at this from a meteorological satellite point of view. @snowman2 please correct me if I'm wrong about anything.

In our field(s) and software the CRS object that @snowman2 is talking about and has implemented in pyproj encapsulates our version of these "complex mathematical functions", although ours seem much simpler. The CRS object can hold things like the model of the Earth to use and other parameters defining the coordinate system like the reference longitude/latitude.
When it comes to computing the coordinates of data on a CRS, the coordinates are typically on a cartesian plane so we have an X and a Y and points in between can be linearly interpolated. These work well as 1D xarray coordinates.
These X and Y coordinates don't tell you all the information alone though so we need the CRS information. Xarray's current rasterio functionality adds this CRS definition as a string value in .attrs. The problem with using .attrs for this is most operations on the DataArray object will make this information disappear (ex. adding two DataArrays).
In geoxarray I was going to try putting a pyproj CRS object in a DataArray's coordinates (.coords). I figured this would be good because then if you tried to combine two DataArrays on different CRSes, xarray would fail. Turns out xarray will just ignore the difference and drop the crs coordinate so that was no longer my "perfect" option. Additionally, to access the crs object would have to be accessed by doing my_data_arr.coords['crs'].item() because xarray stores the object as a scalar array.
Xarray accessors, last time I checked, often have to be recreated when working with Dataset or DataArray objects. This has to do with how low-level xarray converts Variable objects to DataArrays. I didn't expect this when I started geoxarray and I'm not really sure how to continue now.

As for your use case(s), I'm wondering if an xarray accessor could work around some of the current limitations you're seeing. They could basically be set up like @dcherian described, but "arbitrary_function" could be accessed through x, y, z = my_data_arr.astro.world_coordinates(subpixels=4) or something. You could do my_data_arr.astro.wcs_parameters to get a dictionary of common WCS parameters stored in .attrs. The point being that the accessor could simplify the interface to doing these calculations and accessing these parameters (stored in .coords and .attrs) and maybe make changes to xarray core unnecessary.

I'm wondering if an xarray accessor could work around some of the current limitations you're seeing.

I think this sounds like a good idea.

This looks like a nice use case for the forthcoming Xarray's custom index feature.
How I see CRS/WCS-aware Xarray datasets with custom indexes:

A set of coordinate(s) and their attributes hold data or metadata relevant for public use and that could be easily (de)serialized

A custom index (CRSIndex or WCSIndex) provides CRS/WCS-aware implementations of common Xarray operations such as alignment (merge/concat) and data selection (sel), via Xarray.Index's equals, union, intersection and query methods added in #5102 and #5322 (not yet ready for use outside of Xarray). Such custom index may also be used to hold some data that is tricky to propagate by other means, e.g., some internal information like "functional" coordinate parameters or a crs object. Xarray indexes should definitely provide more flexibility than coordinate data or attributes or accessor attributes for propagating this kind of information.

Xarray accessors may be used to extend Dataset/DataArray public API. They could use the information stored in the CRSIndex/WCSIndex, e.g., add a crs read-only property that returns the crs object stored in CRSIndex, or add some some extract_crs_parameters method to extract the parameters and store them in Dataset/DataArray attributes similarly to what @djhoese suggests in his comment above.

For this use case a possible workflow would then be something like this:
# create or open an Xarray dataset with  x, y, z "pixel" (possibly lazy) coordinates
# and set a WCS index
dataset = (
    xr.Dataset(...)
    .set_index(['x', 'y', 'z'], WCSIndex, wcs_params={...})
)

# select data using pixel coordinates
dataset.sel(x=..., y=..., z=...)

# select data using world coordinates (via the "astro" accessor,
# which may access methods/attributes of the WCS index)
dataset.astro.sel_world(x=..., y=..., z=...)

# return a new dataset where the x,y,z "pixel" coordnates are replaced by the "world" coordinates
# (again using the WCS index, and propagating it to the returned dataset)
world_dataset = dataset.astro.pixel_to_world(['x', 'y', 'z'])

# select data using world coordinates
world_dataset.sel(x=..., y=..., z=...)

# select data using pixel coordinates (via the "astro" accessor)
world_dataset.astro.sel_pixel(x=..., y=..., z=...)

# this could be reverted
pixel_dataset = world_dataset.astro.world_to_pixel(['x', 'y', 'z'])
assert pixel_dataset.identical(dataset)

# depending on the implementation in WCSIndex, would either raise an error
# or implicitly convert to either pixel or world coordinates
xr.merge([world_dataset, another_pixel_dataset])

We could also imagine
# returns a new dataset with both pixel and world (possibly lazy) coordinates
>>> new_dataset = dataset.astro.append_world({'x': 'xw', 'y': 'yw', 'z': 'zw'})

# so that we can directly select data either using the pixel coordinates...
>>> new_dataset.sel(x=..., y=..., z=...)

# ...or using the world coordinates
>>> new_dataset.sel(xw=..., yw=..., zw=...)

# the WCS index would be attached to both pixel and word dataset coordinates
>>> new_dataset
<xarray.Dataset>
Dimensions:  (x: 100, y: 100, z: 100)
Coordinates:
  * x                     (x) float64 ...
  * xw                    (x) float64 ...
  * y                     (y) float64 ...
  * yw                    (y) float64 ...
  * z                     (z) float64 ...
  * zw                    (z) float64 ...
Data variables:
    field                 (x, y, z) float64 ....
Indexes:
    x, y, z, zw, yw, zw    WCSIndex

@benbovy Thanks. This looks really promising and is pretty inline with what I saw geoxarray's internals doing for a user. In your opinion will this type of CRSIndex/WCSIndex work need #5322? If so, will it also require (or benefit from) the additional internal xarray refactoring you mention in #5322?
I can really see this becoming super easy for CRS-based dataset users where libraries like geoxarray (or xoak) "know" the common types of schemes/structures that might exist in the scientific field and have a simple .geo.set_index that figures out most of the parameters for .set_index by default.

In your opinion will this type of CRSIndex/WCSIndex work need #5322? If so, will it also require (or benefit from) the additional internal xarray refactoring you mention in #5322?

Yes, CRSIndex/WCSIndex will need to provide an implementation for the query method added in #5322. However, this could be "as simple as" internally using PandasIndex for each 1-d coordinate in case of raster/grid data, maybe with an additional check that the values provided to .sel are in the same CRS (for example int the case of advanced indexing where xarray.DataArray or xarray.Variable objects are passed as arguments).
What will be probably more tricky is to find some common way to handle CRS for various indexes (e.g., regular gridded data vs. irregular data), probably via some class inheritance hierarchy or using mixins.

I can really see this becoming super easy for CRS-based dataset users where libraries like geoxarray (or xoak) "know" the common types of schemes/structures that might exist in the scientific field and have a simple .geo.set_index that figures out most of the parameters for .set_index by default.

In case we load such data from a file/store, thanks to the Xarray backend system, maybe we won't even need a .geo.set_index but we'll be able to build the right index(es) when opening the dataset!

@benbovy I'm reading over the changes in #5322. All of this is preparing for the future, right? Is it worth it to start playing with these base classes (Index) in geoxarray or will I not be able to use them for a CRSIndex until more changes are done to xarray core? For example, none of this set_index for Index classes stuff you showed above is actually implemented yet, right?

@djhoese you're right, I thought it was better to do all the internal refactoring first but we maybe shouldn't wait too long before updating set_index and the DataArray / Dataset constructors so that you and others can start playing with custom indexes.

@benbovy It's been a while since I've looked into xarray's flexible index work. What's the current state of this work (sorry if there is some issue or blog I should be watching for)? Is it possible for me as a user to create my own index classes that xarray will accept?

@djhoese not yet but hopefully soon! Most of the work on explicit indexes is currently happening in #5692, which once merged (probably after the next release) will provide all the infrastructure for custom indexes. This is quite a big internal refactoring (bigger than I initially thought) that we cannot avoid as we're changing Xarray's core data model. After that, we'll need to update some public API (Xarray object constructors, .set_index(), etc.) so that Xarray will accept custom index classes. This should take much less work than #5692, though.

authors: benbovy dcherian djhoese eteq rabernat snowman2

repo: pydata/xarray


text:
Unexpected behavior when opening Canadian global GEM 25km grib files using pynio engine

Hi all, I am experiencing interesting behavior when opening a GEM global 25km resolution grib files using pynio.
ds=xr.open_dataset('CMC_glb_TMP_ISBL_700_latlon.24x.24_2020050600_P198.grb',engine='pynio')
print(ds)

<xarray.Dataset>
Dimensions: (lat_0: 751, lon_0: 1500)
Coordinates:

lat_0 (lat_0) float32 -90.0 -89.76 -89.52 ... 89.52 89.76 90.0
lon_0 (lon_0) float32 180.0 180.24 180.48 ... 539.52 539.76
Data variables:
TMP_P0_L100_GLL0 (lat_0, lon_0) float32 ...
print(ds['lon_0'])
<xarray.DataArray 'lon_0' (lon_0: 1500)>
array([180. , 180.24, 180.48, ..., 539.28, 539.52, 539.76], dtype=float32)
Coordinates:
lon_0 (lon_0) float32 180.0 180.24 180.48 180.72 ... 539.28 539.52 539.76
Attributes:
long_name: longitude
grid_type: Latitude/Longitude
units: degrees_east
Dj: [0.24]
Di: [0.24000001]
Lo2: [179.76]
La2: [90.]
Lo1: [180.]
La1: [-90.]

Notice that the range for lon_0 is 180->539.76. When I open using cfgrib as the xarray engine:
ds=xr.open_dataset('CMC_glb_TMP_ISBL_700_latlon.24x.24_2020050600_P198.grb',engine='cfgrib')
print(ds['longitude'])

<xarray.DataArray 'longitude' (longitude: 1500)>
array([-180. , -179.76, -179.52, ..., 179.28, 179.52, 179.76])
Coordinates:
time datetime64[ns] ...
step timedelta64[ns] ...
isobaricInhPa int64 ...

longitude (longitude) float64 -180.0 -179.8 -179.5 ... 179.5 179.8
valid_time datetime64[ns] ...
Attributes:
units: degrees_east
standard_name: longitude
long_name: longitude

Therefore, it appears 'cgfrib' shows the longitude correctly while 'pynio' does not.
MCVE Code Sample

# Your code here

Expected Output
Problem Description

Versions
Output of xr.show_versions()

Can you check if the file is read properly using only pynio? i.e. remove xarray from the equation here.

Good question, actually it appears that this is occurring in pynio without the use of xarray:
ds=Nio.open_file('CMC_glb_TMP_ISBL_700_latlon.24x.24_2020050600_P198.grb')   ds.variables['lon_0'].get_value()
gives a result of:
array([180.  , 180.24, 180.48, ..., 539.28, 539.52, 539.76], dtype=float32)

Ah thanks for checking. I would raise an issue over at pynio then.

Will do, thanks!

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Setting datarrays with non-dimension coordinates errors

What happened?
I'm not sure if this is a bug or a feature but I was expecting this example to work since the new coord is just a slight rewrite of the original dimension coordinate:
import xarray as xr

ds = xr.tutorial.open_dataset("air_temperature")

# Change the first time value:
ds["air_new"] = ds.air.copy()
air_new_changed = ds.air_new[{"time": 0}] * 3
ds.air_new.loc[air_new_changed.coords] = air_new_changed  # Works! :)

# Add a another coord along time axis and change
# the first time value:
ds["air_new"] = ds.air.copy().assign_coords(
    {"time_float": ds.time.astype(float)}
)
air_new_changed = ds.air_new[{"time": 0}] * 4
ds.air_new.loc[air_new_changed.coords] = air_new_changed  # Error! :(

Traceback (most recent call last):

  Cell In[25], line 5
    ds.air_new.loc[air_new_changed.coords] = air_new_changed

  File ~\AppData\Local\mambaforge\envs\jw\lib\site-packages\xarray\core\dataarray.py:222 in __setitem__
    dim_indexers = map_index_queries(self.data_array, key).dim_indexers

  File ~\AppData\Local\mambaforge\envs\jw\lib\site-packages\xarray\core\indexing.py:182 in map_index_queries
    grouped_indexers = group_indexers_by_index(obj, indexers, options)

  File ~\AppData\Local\mambaforge\envs\jw\lib\site-packages\xarray\core\indexing.py:144 in group_indexers_by_index
    raise KeyError(f"no index found for coordinate {key!r}")

KeyError: "no index found for coordinate 'time_float'"

Despite being very similar to the original one, the new coord is not a dimension coordinate, which is thus not indexed by default so loc or other label-based indexing won't work with it.

That said, we could probably make the error message a bit more user-friendly and verbose than just saying that no index is found, e.g.,
No index found for coordinate 'x', which therefore cannot be used with `.loc` or `.sel` (label-based indexing). Set an index first for that coordinate using either `set_index` or `set_xindex`. Note that you can still use `.iloc` and `.isel` (integer-based indexing) when no index is set.

Would you mind showing the intended way for my example? Because I don't find this intuitive at all.
ds["air_new"] = ds.air.copy().assign_coords(
    {"time_float": ds.time.astype(float)}
)
# ds["air_new"] = ds["air_new"].set_xindex("time_float") # Doesn't work
a = ds["air_new"].set_xindex("time_float")
air_new_changed = a[{"time": 0}] * 4
a.loc[air_new_changed.coords] = air_new_changed

ValueError: Xarray does not support label-based selection with more than one index over the following dimension(s):
'time': 2 indexes involved
Suggestion: use a multi-index for each of those dimension(s).

Other tests:
ds.air_new.iloc[air_new_changed.coords] = air_new_changed  # Error! :(
AttributeError: 'DataArray' object has no attribute 'iloc'

ds.air_new.sel[air_new_changed.coords] = air_new_changed  # Error! :(
TypeError: 'method' object does not support item assignment

Haha yes agreed that is confusing. So improving the error messages will require more coordination across the codebase and thinking more about the possible cases. That is not an easy task.
Regarding your example, just setting a single index for "time_float" does not work: currently you cannot call loc or sel and pass labels at once for two or more independently indexed coordinates that share the same dimension (the error message mostly makes sense except the "dimension" term that is not accurate here). This because it is easier to raise now when such case occurs and maybe figure out later how we can merge results from different index queries along the same dimension.
(My bad, Xarray has no iloc indeed... I'm mostly using isel/sel).
I'm not sure what exactly you're trying to achieve in your example, actually.

item assignment on labels selected using both the "time" and "time_float" coordinates? Then set a multi-index from both of them and call loc. Or maybe chain two calls to loc each with one of these coordinates but I'm not sure at all that it will work.

item assignment on labels selected using only the "time" coordinate? Then be sure to not pass any mapping to loc that contains a key referring to an unindexed coordinate (i.e., do not include "time_float" there).

I simply want to modify a few values in an array. :) A similar example in numpy-land:
a = np.array([1, 1, 1, 1])
b = a[0] * 3
a[0] = b

Now in xarray land this ds["air_new"] happens to have a few helper coordinates. This is what I came up with and it took longer than I'd like to admit:
# Remove unneccessary coordinates without indexes:
ds["air_new"] = ds.air.copy().assign_coords({"time_float": ds.time.astype(float)})
air_new_changed = ds.air_new[{"time": 0}] * 3
ds.air_new.loc[
    {
        k: air_new_changed.coords[k].values
        for k in air_new_changed.coords.keys() & air_new_changed.indexes.keys()
    }
] = air_new_changed

Is this the most elegant way? How would you have done it?
Using masks doesn't have this issue for some reason:
# Use a mask instead...
ds["air_new"] = ds.air.copy().assign_coords({"time_float": ds.time.astype(float)})
mask = xr.ones_like(ds["air_new"], dtype=bool)
mask.loc[{"time": ds.time.isel(time=0)}] = False
ds["air_new"] = ds["air_new"].where(mask, 3 * ds["air_new"])

Why would I want to pass around helper coordinates?

Normalizing coordinates, for example between 0 and 1.
Converting absolute time to relative time: ds["relative_time"] = ds.time - ds.time[0]
Datetime can be a nightmare to code with so easier to just convert to floats: ds["time_float"] = ds.time.astype(float)

Yes helper coordinates are useful. However I don't really understand why do you want to pass labels to .loc for all of those coordinates (or all the indexed ones) at once?
Since your helper coordinates are all derived from the same original coordinate, couldn’t you just select data using the one that is the most convenient to you ? E.g.,
ds.air_new.loc[{"time": air_new_changed.time.values}] = air_new_changed

This should yield the same results (and it is much more elegant) than
ds.air_new.loc[
    {
        k: air_new_changed.coords[k].values
        for k in air_new_changed.xindexes.keys()
    }
] = air_new_changed

Both the "time" and "time_float" coordinates share the same dimension and should be subset / propagated the same way.
I doubt that you want to update values on a selection using arbitrary (possibly non-overlapping) labels for each of those coordinates, do you? E.g.,
ds.air_new.loc[{"time"="2013-01-01", "time_float"=1.42e18}] = …

We don’t support that (yet) because it is more complicated. How should we join the integer array indices selected on the "time" dimension from the given "time" and "time_float" labels? By union, intersection or difference? Or do we want an exact match and raise otherwise? It might get even more complicated when multiple indexes, coordinates and/or dimensions are involved. In theory, we could support it but it would represent quite a lot of work and the implementation would quickly get much more complicated.
I can imagine how just passing all coordinates like
ds.air_new.loc[air_new_changed.coords] = air_new_changed

would be convenient as (with carefully chosen default behavior) it removes some manual steps or thinking, similarly to Xarray auto-alignment. However, starting to consider more use cases we will quickly see that there is no easy generic solution, like for alignment (#7045). Note that we also don't support alignment when multiple (independent) indexes are found along the same dimension (example here) for the same reasons.

authors: Illviljan benbovy


text:
if TYPE_CHECKING considered harmful by runtime typecheckers

What happened?
This is a cross-post from the @beartype issue tracker, in which a downstream user tried to apply runtime type checking while using xarray... only to discover all the types, at runtime, were non-existent. The chief cause of this is spurious usage of if TYPE_CHECKING used throughout the xarray repo, which is undefined at runtime, like the types it provides.
https://github.com/beartype/beartype/issues/456
The below bug report is merely copied from that post.
The relevant issue (with respect to the MCVE below) seems to be the below example from the code base:
from __future__ import annotations
from typing import IO, TYPE_CHECKING, Any, Generic, Literal, cast, overload
...

if TYPE_CHECKING:
    ...
    from xarray.core.dataarray import DataArray # Crucially missing at runtime

...

class Dataset(
    DataWithCoords,
    DatasetAggregations,
    DatasetArithmetic,
    Mapping[Hashable, "DataArray"], # Terribly undefined at runtime
):

What did you expect to happen?
For the types to be visible at runtime, for runtime type checking.
From @leycec, the only real change needed should be:
class Dataset(
    DataWithCoords,
    DatasetAggregations,
    DatasetArithmetic,
    Mapping[Hashable, "xarray.core.dataarray.DataArray"],  # <-- this is all @beartype asks
):

Minimal Complete Verifiable Example
import xarray as xr
 from beartype import beartype

 @beartype
 def process_data(data: xr.Dataset):
     print(f"Processing data of type: {type(data)}")

 ds = xr.Dataset({"temperature": (["x", "y"], [[20, 25], [30, 35]])})
 process_data(ds)  # This fails

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[X] Complete example — the example is self-contained, including all data and the text of any traceback.
[X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.
[X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output
On the `# this fails` line:

BeartypeCallHintForwardRefException: Forward reference \"DataArray\" unimportable from module \"__main__\"."

Anything else we need to know?
MVCE Versions
beartype: 0.19.0
xarray: 2024.9.0
Python: 3.11.10
Environment
Omitted the environment, because not relevant.

Do you have suggestions for how to avoid this? Happy to take a PR which removes some of these if it the proposed code still works. Generally this is done to avoid circular or optional imports. (I don't think "spurious" is an accurate description)
(note #9561 as a case for more of this...)

...heh. @leycec has been summoned. Thanks so much to @JWCS for getting this typing ball rolling. You rock! :rock:
TYPE_CHECKING: After All These Years, Still You Plague @beartype
This is an incredibly fun discussion from my perspective – but probably an incredibly tedious and non-fun discussion from everybody else's. I acknowledge this. I shall aim for concision and fail. :face_exhaling:

Is there any direction from the python typing community on using TYPE_CHECKING — I think we would update our approach if the mypy folks said it was bad practice.

It's not so much that TYPE_CHECKING is unilaterally bad per say; it's more that TYPE_CHECKING is really hard to get right under real-world conditions and constraints. Since it's hard to get right, it's easy to get wrong. More formally, TYPE_CHECKING promotes fragility at runtime – especially desynchronization between static type-checkers like mypy and pyright on the one hand and hybrid runtime-static type-checkers like @beartype and typeguard on the other.
TYPE_CHECKING is thus the Python equivalent of Dijkstra's infamous "Go To Considered Harmful". Like TYPE_CHECKING, goto statements are hard to get right. @JWCS was omniscient when he chose this issue title.
Conclusion: mypy Is Right About Everything for Once!?
The way to get TYPE_CHECKING right is basically to:

Do exactly what mypy says.

Specifically, this is what mypy advises to resolve import cycles and circular imports:
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    import bar

def listify(arg: 'bar.BarClass') -> 'list[bar.BarClass]':  # <-- absolute forward references for victory!
    return [arg]

That's almost what xarray is already doing – but not quite. xarray currently prefers relative to absolute forward references. The xarray equivalent of the above code would look something like:
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from bar import BarClass

def listify(arg: 'BarClass') -> 'list[BarClass]':  # <-- relative forward references for great loss!
    return [arg]

Unfortunately for xarray, your seemingly sensible change to the stock mypy advice renders the resulting type hints unresolvable at runtime. Why? Because TYPE_CHECKING is False at runtime, which is when type-checkers like @beartype and typeguard run. The xarray equivalent of the above code thus reduces at runtime to simply:
from typing import TYPE_CHECKING

def listify(arg: 'BarClass') -> 'list[BarClass]':  # <-- relative forward references for great loss!
    return [arg]

We see the issue. BarClass is a relative forward reference. By definition, relative forward references are relative to the current submodule. @beartype thus expects to find BarClass in the current submodule. But BarClass doesn't live in the current submodule! BarClass lives in the external bar module. @beartype no longer knows where BarClass lives, because all contextual metadata required to find out where BarClass lives has already been destroyed by the Python interpreter at runtime. :grimacing:
Shockingly, mypy is actually right about everything for once. Absolute forward references are the way. They preserve the context that runtime type-checkers need to resolve circular imports.
Seriously? We Need to Refactor Our Entire Friggin' Codebase?
Yeah. I wouldn't want to do that either. Clearly, refactoring all circular imports across your entire codebase from relative to absolute forward references is a non-trivial, tedious, and boring endeavour. I'd run from that too. :runner: :dash:
The alternative is incompatibility with runtime type-checkers. That's... not great either. Both options kinda suck, huh?
But fear not! From @beartype's admittedly completely different perspective, this is sorta resolved. We've internally "blacklisted" both xarray and Pydantic, which deploy a similar if TYPE_CHECKING: strategy throughout their codebases.
By "blacklisted," I mean that @beartype now merely shallowly type-checks just the types of objects that originate from xarray and Pydantic. Previously, @beartype tried to deeply type-check the items in containers that originate from xarray and Pydantic  xarray.Dataset. For example, @beartype tried to type-check that xarray.Dataset containers map from Hashable objects to xarray.DataArray containers. Oh well.
It could be worse. @beartype now officially "supports" xarray and Pydantic by mostly ignoring everything from xarray and Pydantic. We'd looooove to fully support both xarray and Pydantic, though. If you feel like tackling this in a year or two, just ping us when the resolution lands and we'll immediately "unblacklist" xarray in @beartype.
I'm So Tired, Fam.
Yeah. Me too. Let's pass out, everybody. Open-source volunteerism is a done thing tonight. :sleeping:

OK — keeping this brief — do you see a reasonable way to convert the imports to work? We don't want xarray.core.dataarray.DataArray every time we use a DataArray, but we're probably OK with that in the type definitions...

I have the same question then @max-sixty.
How would you actually replace
if TYPE_CHECKING:
    ...
    from xarray.core.dataarray import DataArray

def x(da: DataArray) ...

Do you really have to do:
if TYPE_CHECKING:
    ...
    import xarray

def x(da: xarray.core.dataarray.DataArray) ...

?

Surely that last suggestion wouldn't work either, for the same reason? When TYPE_CHECKINGisFalse, xarray` doesn't get imported, so the reference won't be resolvable by the runtime type checker.
I have to say this seems like the sort of thing that should be resolved at the level of linter conventions and standards rather than by going around to individual codebases...

...heh. @headtr1ck has the right of it on both accounts. That's an @airbus scientist for you, huh? What won't those guys come up with next!? Basically, you have two sane choices:
tl;dr: This Is What I'm Saying
This suffices for @beartype. This should suffice for everyone else, too:
def x(da: 'xarray.DataArray'): ...  # <-- @beartype loves this

If you hate quotes, ...no shame this is also totally fine:
from __future__ import annotations
def x(da: xarray.DataArray): ...  # <-- @beartype even loves this

Note the distinct lack of TYPE_CHECKING anywhere. If you really want to use TYPE_CHECKING, you can. No worries. "You do you," as Gen Z says. But there's not much point in doing so, in this case. TYPE_CHECKING is usually a last resort when mypy or pyright just won't shut up about something – not logic you necessarily throw around in every submodule.
@headtr1ck Is Right: Just Use the Short Form
If you hate reiterating xarray.core.dataarray.DataArray everywhere, just prefer the short-form xarray.DataArray everywhere like above. That's fine. All runtime type-checkers (including @beartype) know how to dynamically import attributes from third-party modules. You don't need to tell runtime type-checkers how to import stuff. Trivial, right?
In fact, I'm unclear why you even need to tell static type-checkers how to import attributes. What's the point of redundant boilerplate like this, anyway?
if TYPE_CHECKING:
    import xarray

No idea, honestly. xarray.DataArray is already enough metadata to unambiguously resolve that forward reference. Importing xarray on top of that is just meaningless fluff, really. You shouldn't need TYPE_CHECKING at all to declare forward references. I shrug non-committally. </shrug>
But What If You Hate Even the Short Form?
If you hate even xarray.DataArray, define a new private submodule – say, xarray._forwardrefs – that collects all of your forward references together. Then import and reuse them throughout your codebase as type hints: e.g.,
# In "xarray._forwardrefs":
DataArray = 'xarray.DataArray'

# In every other "xarray" submodule:
from xarray._forwardrefs import DataArray
def x(da: DataArray): ...  # <-- @beartype actually even loves this

Literally anything other than what you're currently doing is something that @beartype loves. :smiling_face_with_tear:
Heuristics: A Great Way to Blow Up PyTorch

One idea — but maybe @beartype have already considered this — is for @beartype to do some heuristics.

...heh. Heuristics, huh? So, hacks. Actually, I usually adore hacks. @beartype is one giant hack resting on top of a pile of kludges cobbled together with vicious one-liners.

So for example, DataArray is exposed at the root of the package xarray.DataArray, so that could be a reasonable place to check.

Sure. In this specific case, ad-hoc heuristics like that definitely work. In the general case, they... don't. Runtime type-checking is really all about definitive knowledge. "Intuitively guessing" where ambiguous types reside inevitably breaks down in common edge cases, invites non-portable fragility, and basically destroys the Python ecosystem.
@beartype has been integrated into PyTorch. You know? I used to be cavalier about this sort of thing. Then I repeatedly broke PyTorch. Microsoft was understandably unhappy. Now, I exercise caution in all things. I grit my teeth. :grimacing:
Explicit is better than implicit. Encouraging ambiguity just makes other people's code blow up.
But Maybe It's Better to Just Give Up
Given that the plan to close label was just applied, it's unclear whether this is worth additional effort. I kinda assumed going in that this was a lost cause. Predictably, it's distinctly smelling like a lost cause. I hate wasting everyone's time – especially my own.
If "your type hints are busted at runtime, so your package fundamentally violates CPython semantics including upcoming standards like PEP 649" isn't a compelling argument, it's probably best I bow out and just hack on open @beartype issues.
I sigh. I sigh and collapse into bed. :face_exhaling:

This suffices for @beartype. This should suffice for everyone else, too:
def x(da: 'xarray.DataArray'): ...  # <-- @beartype loves this

Note the distinct lack of TYPE_CHECKING anywhere.

The reason we have the if TYPE_CHECKING is because of circular imports — does this solution get around that? When I try the change locally, it raises errors.

Given that the plan to close label was just applied, it's unclear whether this is worth additional effort.

FWIW I'm marking it plan-to-close because I'm not sure we're going to make progress on getting something that meets the three criteria above, and I also don't want to lose anyone's time, especially yours. If there is a way of making progress on those three criteria then let's make a plan and fix it. Do you think there is a way of meeting those criteria?
Is that reasonable?

By now I didn't have time to play around with that.
But mainly the issue is that for us xarray is not a third party package, we are xarray.
So we cannot simply import xarray.
The proposal of using forward refs as strings is not too much work to implement. Probably we can use our types module and replace the if TYPE_CHECKING imports with these forward refs.
But I agree that without some automated way of checking this it is not very useful.

Let's reopen when we find a way that meets the three criteria in https://github.com/pydata/xarray/issues/9581#issuecomment-2397604939.
@leycec we want to be good citizens in the python ecosystem, and we want to support projects like beartype — so please feel free to reopen / comment if there's some solution, thank you for commenting so far (& for building beartype!).

Also note that the TCH rules of ruff might not do the right thing, at least not by default:

Unused imports add a performance overhead at runtime, and risk creating import cycles. If an import is only used in typing-only contexts, it can instead be imported conditionally under an if TYPE_CHECKING: block to minimize runtime overhead.

@max-sixty Would adding pytest-beartype to xarray CI tests qualify as a kind of "automated way of checking"?

@max-sixty Would adding pytest-beartype to xarray CI tests qualify as a kind of "automated way of checking"?

Yes that would!

authors: DimitriPapadopoulos JWCS TomNicholas headtr1ck leycec max-sixty


text:
Add dataarray scatter

Splitting up #5622 as the scope of it has grown too large now.

Adds support for dataarray scatter plots and replaces the dataset version.
Scatter now has 3d support with the z argument.
Scatter now always returns a single pathcollection, earlier it could return a list of pathcollection when using categoricals
Better legend, handles now categoricals. Making continous/discrete options slightly redundant, still there though.
Facetgrid generalized slightly to handle 3d plots.

TODO:

sharex sharey test https://github.com/pydata/xarray/pull/7047: len(list(ax.get_shared_x_axes())) != 0

[x] Tests added
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst
[ ] New functions/methods are listed in api.rst

I'm merging this soon. Will hopefully go smoothly.

I'm merging this soon. Will hopefully go smoothly.

I will wait with merging https://github.com/pydata/xarray/pull/7052
and then try to fix the merge conflicts :)

Thanks @Illviljan !

Nice work!

source: gharchive/pull-request

authors: Illviljan dcherian headtr1ck mathause


text:
Use TypeVars rather than specific unions

Also from the chars of #8208 -- this uses the TypeVars we define, which hopefully sets a standard and removes ambiguity for how to type functions. It also allows subclasses.
Where possible, it uses T_Xarray, otherwise T_DataArrayOrSet where it's not possible for mypy to narrow the concrete type down.

The failure is coming from the docs — getting lots of timeouts; I think unrelated to the code changes

Yeah, fair.
Your call on whether it's an improvement.
I think if we just have DataArray | Dataset around the place, that's totally fine. The thing I was less keen on is a bunch of different ones that then mean we're digging through mypy errors to understand the cause of something. But actually that hasn't been an issue so far. So no objections to closing...

Closing — thanks for the review & useful feedback @headtr1ck , greatly appreciated!


text:
Real support

I'm opening this pull request to be able to easily add comments to your changes (we'll merge much later). will do a review in the diff now.

/cc @Jubo-Xu

excellent! I'll mark it as not draft, then the CI should run I think.

I don't know why the actions aren't being run. I opened #56 where it seems to work.

repo: pydrofoil/pydrofoil

license: MIT


text:
Derive DirectoryObject from Path?

Currently DirectoryObject is its own class. There are, however, functionalities that are given almost identically in Path from pathlib (which DirectoryObject anyway heavily uses). Furthermore there are functionalities that I would love to have but are not implemented in DirectoryObject yet, such as path_1 / path_2 etc. So I thought maybe it makes more sense to derive DirectoryObject from Path, so that people can straightforwardly use the same functionalities. What do you think? @liamhuber

💯 I am huge fan of this. Maybe it will require merging DirectoryObject and FileObject together.
The biggest catch I can think of will be to ensure that we exclusively extend rather than modify pathlib.Path, as I'm sure there's a billion places where people rely on the pattern if isinstance(thing, pathlib.Path): thing_that_depends_on_Pathlike_behavior(thing), and it would be nice if it was safe to simply plug our objects in under these conditions.

Somewhat related, there has been some activity in pyiron_base modifying the working directory behaviour, and it would be lovely if the snippets solution could be used across the pyiron-verse. To that end, it would be good to double check what changes were made and why, and see whether we can adapt what's done here to accomplish the same thing.
Similarly, the stuff with browsing zipped content might be a cool feature to add in.

Maybe it will require merging DirectoryObject and FileObject together.

Yeah that's actually my biggest concern. Right now I really like the fact that the files and directories are separated, because that's somewhat crucial that each node has its own working directory and file objects are derived from them, and all this is properly conceptualised.

It sounds like we're on the same page (albeit your page is more densely packed with knowledge). I guess a reasonable attack is to simply try moving forward with the re-parenting as see how it turns out.

authors: liamhuber samwaseda


text:
Add new extractor for Bluecoders.io (http://bluecoders.io/)

Also adds a dependency to parsedatetime lib in order to parse natural dates

Thanks for your contribution, CkuT

@CkuT that would be very nice if you add the "address" property (this is the content of  just after )
A trainee is finishing a dev task which integrates geoloc/geosearch of job opportunities on pyjobs. It is based on address, so it would be very nice to add this for bluecoders.io :-)

I just made a quick implementation. Does the city enough for you ?
See here : 1db9776

Yes, that's fine

authors: CkuT lebouquetin

repo: pyjobs/crawlers


text:
add submodule for sanantonio

PyLadies Pull Request
Please document the following. We will aim to be as timely as possible in processing your pull request, however as this is a volunteer staffed project anticipate at least a week for pull requests to start to be reviewed.
Does this PyLadies previously exist in the main PyLadies repo?

[ ] Yes
[x] No

If you marked yes, have you deleted it from the the main PyLadies repo?

[ ] Yes, and here is the pull request: [Insert pull request url here]
[ ] No

Is this pull requesting adding a PyLadies chapter website?

[x] Yes
[ ] No

If you marked yes, is this chapter registered in the PyLadies Chapter Directory?

[ ] Yes
[x] No

If you marked no, please register your chapter using the following form here. The form can only be accessed by being logged into your official PyLadies email account. Were you able to complete the registration?

[x] Yes
[ ] No, I could not because [add response below]:

Is this pull request updating a PyLadies website?

[ ] Yes
[x] No

Pull Request Checklist

[x] Fill out all sections above

/cc @pyladies/tech-and-infra-admins

Closing as we need to use the  repo  made in https://github.com/pyladies/chapter-websites/issues/36#issuecomment-790156008

authors: PyLadiesSA lorenanicole

repo: pyladies/chapter-websites


text:
Event loop support

Does pylast support asyncio?

Probably not, but pull requests are welcome.

Closing for now, can re-open as necessary.

authors: epifab hugovk

repo: pylast/pylast

license_source: bigquery


text:
Use scribble.png get bad result

i use scribble.png instead of trimap to compute alpha,the result is not good

what wrong with this?hope for your kindly reply@99991

Unfortunately, scribbles are not supported. The reason is that it is impossible to tell if a pixel is black because it is a scribble or because the image is black. It is pure coincidence that it works this well at all.
When converting the scribbles to a trimap, you can see that there is a black pixel in the trimap because the lemur has a black pixel.

from pymatting import *

# convert scribbles to trimap

scribbles = load_image("scribbles.png", "GRAY")

is_fg = scribbles == 1.0
is_bg = scribbles == 0.0

trimap = 0.5 + 0.5 * is_fg - 0.5 * is_bg

save_image("trimap.png", trimap)

Removing the stray black pixel in the trimap will produce this alpha matte, where the black part of the lemur is correct now:

In addition, you could use knn alpha matting to get rid of most of the errors around the lemur, but it does not work well for the eyes because their color is substantially different from the colors which are constrained with the trimap.

from pymatting import *

image = load_image("image.png", "RGB")
trimap = load_image("trimap.png", "GRAY")

alpha = estimate_alpha_knn(image, trimap, laplacian_kwargs=dict(
    n_neighbors=[10,10],
    distance_weights=[10.0, 0.01],
))

save_image("alpha.png", alpha)

But I'm afraid that you will need a better trimap to get perfect results.

Much thanks for your specific reply,looks like that just use scribbles is not enough to get perfect result.Is it exist which the work can perform well on scribbles?I tried serveral works on github but they all failed to perform well.@99991

You could try

information flow matting https://github.com/yaksoy/AffinityBasedMattingToolbox
shared matting https://www.inf.ufrgs.br/~eslgastal/SharedMatting/#Downloads
GCA Matting https://github.com/Yaoyi-Li/GCA-Matting

You could try

information flow matting https://github.com/yaksoy/AffinityBasedMattingToolbox
shared matting https://www.inf.ufrgs.br/~eslgastal/SharedMatting/#Downloads
GCA Matting https://github.com/Yaoyi-Li/GCA-Matting

Ok,i will try it.Thanks for your rapidly reply:)

repo: pymatting/pymatting


text:
Consolidate pipenv settings to use get_from_env

The issue
See #5407

some pipenv settings are insufficiently documented, especially in terms of how to disable them
implementation of pipenv settings detection is inconsistent, even though pipenv.environments.get_from_env exists as a pretty hardened utility

The fix

refactor all relevant settings to use get_from_env, which more explicitly allows settings to be disabled

add default kwarg to get_from_env

update docs to indicate how to explicitly disable a setting

Other changes

Fix rst in contributing guide
Refactor out is_env_truthy

The checklist

[x] Associated issue
[x] A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

@oz123 I'm getting this failed test case below. is it possible the test case is broken because a new vulnerability for wheel has been reported (51499)? a bit of a funky test case because it seems to assume that there will never be a new vulnerability in any of the installed packages. unless I am misinterpreting what is happening here
_________________________________________________________________________________________________________________________________________________________________________ test_pipenv_check __________________________________________________________________________________________________________________________________________________________________________

pipenv_instance_private_pypi = functools.partial(<class 'tests.integration.conftest._PipenvInstance'>, capfd=<_pytest.capture.CaptureFixture object at 0x1162d8190>)

    @pytest.mark.cli
    @pytest.mark.needs_internet(reason='required by check')
    @flaky
    def test_pipenv_check(pipenv_instance_private_pypi):
        with pipenv_instance_private_pypi() as p:
            c = p.pipenv('install pyyaml')
            assert c.returncode == 0
            c = p.pipenv('check')
            assert c.returncode != 0
            assert 'pyyaml' in c.stdout
            c = p.pipenv('uninstall pyyaml')
            assert c.returncode == 0
            c = p.pipenv('install six')
            assert c.returncode == 0
            c = p.pipenv('check --ignore 35015')
>           assert c.returncode == 0
E           assert 1 == 0
E            +  where 1 = <Result SystemExit(1)>.returncode

tests/integration/test_cli.py:156: AssertionError
------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
$ pipenv check --ignore 35015
Checking PEP 508 requirements...
Passed!
Checking installed package safety...
51499: wheel <0.38.0 resolved (0.37.1 installed)!
Wheel 0.38.0 fixes a potential DoS attack via the 'WHEEL_INFO_RE' regular expression.

Command failed...
------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Captured stderr call ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Notice: Ignoring CVE(s) 35015

Traceback (most recent call last):
  File "/Users/micahsmith/workspace/pipenv/.venv/lib/python3.10/site-packages/click/testing.py", line 408, in invoke
    return_value = cli.main(args=args or (), prog_name=prog_name, **extra)
  File "/Users/micahsmith/workspace/pipenv/pipenv/cli/options.py", line 57, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "/Users/micahsmith/workspace/pipenv/pipenv/vendor/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/Users/micahsmith/workspace/pipenv/pipenv/vendor/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/micahsmith/workspace/pipenv/pipenv/vendor/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/micahsmith/workspace/pipenv/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/micahsmith/workspace/pipenv/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/micahsmith/workspace/pipenv/pipenv/vendor/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/Users/micahsmith/workspace/pipenv/pipenv/cli/command.py", line 480, in check
    do_check(
  File "/Users/micahsmith/workspace/pipenv/pipenv/core.py", line 2921, in do_check
    sys.exit(1)
SystemExit: 1

went ahead and updated the test case to ignore the new vulnerability, tests pass locally now

@micahjsmith see my PR #5450 there is a fix for that wheel issue . Thank you for putting up the time for this PR!

repo: pypa/pipenv


text:
[FR] Allow preserving directory structure of headers

What's the problem this feature will solve?
The problem and a solution using distutils to override the install_headers command is detailed here: https://stackoverflow.com/a/50114715/1976323
TL;DR;
setup(headers=['header1.h`, 'path/to/header2.h'])

will strip off path/to/ and in the binary wheel, we end up with header1.h and header2.h in the same directory.
In my case, for technical reasons beyond my control, the directory structure must be preserved, so setuptools doesn't work out of the box or with a simple configuration option.
Describe the solution you'd like
Since distutils is deprecated, it would be nice to have a non-deprecated solution, like a simple flag to select preserving directory structure or not.
Bonus points if it can be done without a setup.py at all.
Alternative Solutions
The solution in the linked SO answer still works.
Additional context
No response
Code of Conduct

[X] I agree to follow the PSF Code of Conduct

Hmm... there was a recent discussion on deprecating headers so maybe this isn't the way to go. 🤷

Maybe it would be probably better to defer this FR? I believe that most of the code handling include dirs comes from pypa/distutils, which probably accepts bugfixes but not new features. But then monkeypatching parts of distutils to add new functionality is always a painful path.
Anyway, if you have a plan on how to implement this, I think we can discuss the details.
Otherwise you can follow numpy example and exposes a function that returns the include paths, and/or specify file paths inside your package directories (that can be easily retrieved by importlib.resources.files).

Otherwise you can follow numpy example

Yes, I am going to try this for now.

authors: abravalheri dlech

repo: pypa/setuptools

license: mit


text:
Combined 11 Dependabot Pull requests

Closes #11553
Closes #11552
Closes #11551
Closes #11550
Closes #11549
Closes #11544
Closes #11541
Closes #11539
Closes #11532
Closes #11531

Not sure what's going on with the RTD build here.

This looks similar to #11504 (blocked)

I removed the changes to docs.txt to see if that gets past it for now, and we can revisit docs later.

@di that seems to have done the trick!

authors: di miketheman

repo: pypa/warehouse


text:
Split openpype lib into 2 or more parts

Description
We have to split content of openpype lib into core lib and all other lib parts. Only core lib should be imported on start before any logic in OpenPype is used to be able do validations and have ability in real library parts to use all imports without wrappers.

also thing like otio (editorial.py) should be imported elsewhere, not in core or lib I think. That would solve some headaches from wrong builds of otio wheels and so on.

also thing like otio (editorial.py) should be imported elsewher

Agree, core should have real minimum that would not call or use on import. That would give us ability to run validations on start and catch them properly.

So I moved files here and there trying to find out what should be and what should not be in core. Will try to sum up what I think about it.
would split current lib into these parts

openpype core - with these parts you can use OpenPype as software

mongo connection
logging definition
settings

pipeline core - without these pipeline logic of OpenPype won't work

anatomy
applications
modifications of avalon and pyblish (currently in plugin_tools.py)
...

openpype core utils - handy functions to get information from or about core

pype info
execute commands
vendored binaries
...

pipeline utils - handy functions that can be used across multiple places in pipeline logic

profiles
avalon context functions (make sence to use it only in host where context is known and set in avalon.Session
...

specific pipeline utils - functions that care about one specific pipeline part

editorial
abstract implementations

Regarding the editorial and its otio import: I believe the otio import could be hidden in decoration function and passed into a function as function argument. So during importing of lib from anywhere otio will not be imported until the time you will need to use any of editorial.py functiuons. Just an idea..
def import_otio(func):
    def inner():
		import opentimelineio as otio
        func(otio=otio)
    return inner

@import_otio
def any_function(input_object, otio=None):
	if otio:
		otio_object = otio.create_object(input_object)
    	return otio_object

Regarding the editorial and its otio import: I believe the otio import could be hidden in decoration function and passed into a function as function argument.

That is actually different issue (and there is a better solution). The problem is that in openpype.lib is used a lot of crutial parts from inside of openpype.lib which should be validated before usage. For example PypeLogger is created on many places on import which may cause crash before it is possible to validate if mongo logging can and should be used. Same for avalon repository and many others...

authors: antirotor iLLiCiTiT jezscha

repo: pypeclub/OpenPype


text:
Can't register a nitrokey 3 security key as 2FA

Describe the bug
Trying to add a nitrokey 3 for 2 factors auth from account configuration raise a Unsupported credential public key type "OKP" error.
Tried with both nitrokey 3A and nitrokey 3 mini.
The same keys works with Github, Google and Microsoft for 2FA auth.
My Platform

Os: Arch
Browser: Firefox 117

This error comes from our dependency on py_webauthn: https://github.com/duo-labs/py_webauthn/blob/56acd35480b7fc355499e9458931dc31476ec08f/webauthn/helpers/decode_credential_public_key.py#L116
The key types that library supports are listed here, and there is some existing discussion about this issue here.
I've submitted a new issue about this here: https://github.com/duo-labs/py_webauthn/issues/175

I've tried from Chromium and I  am able to add the security keys.
Fun fact: after that, I am able to connect to pypi account using the keys from Firefox !

As far as I can tell this is a bug or issue in Firefox, here's where Chrome expects the kty field of the JWK to be OKP:
https://github.com/chromium/chromium/blob/638018f724c52a7b8bf1aca4fe76b6a18fffc542/components/webcrypto/algorithms/x25519.cc#L315-L326
I'm not sure if there's anything we can do here besides recommend using a different browser to register the security key in question.

Problem seems resolved in firefox. I can't reproduce with latest nightly.

I get the same error on test.pypi.org with NK3 + Firefox 115.10.0. Chrome 124.0.6367.118 worked however.

authors: APN-Pucky di fabien-michel

repo: pypi/warehouse


text:
Remove unused db field for ActiveState Trusted Publisher

This field never ended up being used so I'm removing it to declutter the warehouse code and DB a bit.

Thanks @di!


text:
Too many package dependencies

Hi there.
I just noticed when using your module that a lot of modules got pulled in.
In particular, the netifaces module is pulled in, which is a binary extension.  And others.
Browsing your code, I can't see that there are other direct dependencies than fastapi and python-socketio.  But the Pipfile includes a host of extensions, and so does setup.py.   (python-engineio is automatically added as part of python-socketio)
Not sure why the module needs the various "extras" in setup.py, since there is no extras code.

Hey, thanks for addressing the issue. I'll try tidy up the dependencies file in next couple of days 🙏🏻

Hey, thanks for addressing the issue. I'll try tidy up the dependencies file in next couple of days 🙏🏻

authors: kristjanvalur pyropy

repo: pyropy/fastapi-socketio


text:
How can I use gaussian correction in getints3c?

Hello PySCF Community,
I previously utilized Gaussian correction in the getints function successfully with an earlier version of PySCF. However, after updating to the most recent version, I encountered an error when attempting to use the same feature. The error message is as follows:
File "/home/.local/lib/python3.8/site-packages/pyscf/gto/moleintor.py", line 243, in getints
return getints3c(intor_name, atm, bas, env, shls_slice, comp,
File "/home/.local/lib/python3.8/site-packages/pyscf/gto/moleintor.py", line 560, in getints3c
cintopt = make_cintopt(atm, bas[:max(i1, j1)], env, intor_name)
File "/home/.local/lib/python3.8/site-packages/pyscf/gto/moleintor.py", line 813, in make_cintopt
foptinit = getattr(libcgto, intor+'_optimizer')
File "/usr/lib/python3.8/ctypes/init.py", line 386, in getattr
func = self.getitem(name)
File "/usr/lib/python3.8/ctypes/init.py", line 391, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/.local/lib/python3.8/site-packages/pyscf/lib/libcgto.so: undefined symbol: int3c2e_gtg_optimizer
Any suggestions for resolving this error would be greatly appreciated.
Best,
Nan

Hi,
I plan to calculate the gtg integrals for a large molecule and focus on calculating integrals for a specific segment. I saw libcint has been removed gtg integrals. Are there any alternative approaches or updates in the pyscf library that offer similar functionality?
from pyscf import lib,gto
from pyscf.gto.moleintor import getints
geom =  """
O                 -0.02159827    0.93952482    0.00000000
H                  0.93840173    0.93952482    0.00000000
H                 -0.34205286    1.84446066    0.00000000
"""
basis ="ccpvtz"
auxbasis = "ccpvtzjkfit"
mol_gauss = gto.M(atom=geom, basis= basis)
auxmol_gauss = gto.M(atom=geom, basis= auxbasis)
slice = (0, mol_gauss.nbas, 0, mol_gauss.nbas, mol_gauss.nbas, mol_gauss.nbas+ auxmol_gauss.nbas)
def slice_int2e3c_gc(shls_slice):
'''
Caclualte 2e3c - with gaussian
(alpha beta |X)_gaussian = (ab|X)_gaussian
'''
aosym = 's1'
comp = 1
out = None
hermi = 0
ao_loc = None
atm, bas, env = gto.mole.conc_env(mol_gauss._atm, mol_gauss._bas, mol_gauss._env,
                auxmol_gauss._atm, auxmol_gauss._bas, auxmol_gauss._env)
cintopt = None
intor = mol_gauss._add_suffix('int3c2e_gtg')

return getints(intor, atm, bas, env, shls_slice, comp, hermi, aosym,ao_loc, cintopt, out)

print(slice_int2e3c_gc(slice))
Thanks,
Nan

Is the integral you are requesting just the conventional 3c2e integrals?

Hello,
Not conventional 3c2e. I'd like to use Gaussian-type geminal integrals. Here is the formula for this:
$V_{gtg} = X_0 e^{-\gamma |r_{12}|^2}$

It seems these integrals have been removed in libcint 6.0.0 since they are erronous https://github.com/sunqm/libcint/blob/8d13863ff481cea27efea5e56c9e4d352cdb8f80/ChangeLog#L10

repo: pyscf/pyscf


text:
Feature / User Auth

Login and Signup system completed

@pysk0101 Conflict mevcut. Çözebilir misin?


text:
🐛 redirect_url ignored

Bug description
OAuth2Core doesnt use redirect_uri
https://github.com/pysnippet/fastapi-oauth2/blob/33b42dcebc098c0a1086880bd76ff8c110bb0da2/src/fastapi_oauth2/core.py#L79-L80
despite passing it in __init__
https://github.com/pysnippet/fastapi-oauth2/blob/33b42dcebc098c0a1086880bd76ff8c110bb0da2/src/fastapi_oauth2/core.py#L69
So i'd propose
return self.redirect_uri if self.redirect_uri else urljoin(str(request.base_url), "/oauth2/%s/token" % self.provider)

Reproduction URL
No response
Reproduction steps
Set a different redirect uri cause you overwrote the default oauth2_router, wonder why the redirect uri is ignored
Screenshots

Logs
No response
Browsers
Firefox
OS
Linux

@twsl, thanks for the report, but what you described above is a misunderstanding. The redirenct_uri of the OAuth2Client refers to a specific path where the user should be redirected when the authentication process is done (e.g., https://example.com/dashboard see the docs), if it isn't provided it redirects to the base_uri of your FastAPI application which is usually the / path. Meanwhile, the get_redirect_uri refers to the OpenID Connect token endpoint handler URL managed by fastapi-oauth2 itself.
Please feel free to ask questions about the project and configuration if you need help. For questions, please use discussions section of the repository in the future :)
P.S. Sorry for not getting back to you sooner.

authors: ArtyomVancyan twsl


text:
Consider rename?

I'm the maintainer of pysqlite3:

https://github.com/coleifer/pysqlite3
https://pypi.org/project/pysqlite3

I'm not sure quite why this fork exists, or how it differs from the above packages, but I think it would be considerate to rename this so that it is more clear:

Why this project exists separately from the above
How this project differs from the above
Could you rename this so that the differences are more obvious?

I think the main intention of this fork is publishing wheels for macOS and Windows: https://github.com/pysqlite3/pysqlite3/commit/b04ceb5fd11881b6843ee149d67ce72de273dee6

authors: coleifer edgarrmondragon

license: Zlib


text:
attr.asdict(recurse=True) now can handle nested lists

Right now if field contains nested lists it will not properly encoded to dict:
>>> @attr.s
... class Foo():
...     a = attr.ib()
...
>>> a = Foo(a=[[Foo(a=1)]])
>>> attr.asdict(a)
{'a': [[Foo(a=1)]]}

I added support for this case (only for lists and only for asdict function) and also updated testing strategy

Hi,
I see two problems with this solution.
The first is we're making a new function every asdict invocation, which slows things down. The second is, it doesn't solve the problem fully. Consider this:
>>> b = Foo([{1: Foo(1)}])
>>> asdict(b)
{'a': [{1: Foo(a=1)}]}

So if we're gonna solve it, let's solve it right. I've pushed a branch with what I think is a more correct solution: https://github.com/python-attrs/attrs/tree/feature/recursive-asdict. It makes asdict in general a little slower, but it's faster than this solution.
Before:
> pyperf timeit -g -s "from attr import asdict; from random import Random; from tests.utils import nested_classes, simple_classes; r = Random(1); C = simple_classes().example(r); c=C()" "asdict(c)"
Mean +- std dev: 2.43 us +- 0.09 us

After:
> pyperf timeit -g -s "from attr import asdict; from random import Random; from tests.utils import nested_classes, simple_classes; r = Random(1); C = simple_classes().example(r); c=C()" "asdict(c)"
Mean +- std dev: 2.51 us +- 0.10 us

This branch:
> pyperf timeit -g -s "from attr import asdict; from random import Random; from tests.utils import nested_classes, simple_classes; r = Random(1); C = simple_classes().example(r); c=C()" "asdict(c)"
Mean +- std dev: 2.95 us +- 0.10 us

Please add the tests to that branch and we can get this merged.

Ping? 😇

Fixed in #395 – thank you for the inspiration.

authors: Pr0Ger Tinche hynek

repo: python-attrs/attrs


text:
Addition of a new command !wiki

Hi
So I have a suggestion, not sure if here's where I should post. If it's the wrong place, let me know so I could relocate it
I was thinking, we could have a !wiki command
where !wiki 'xyz' would make the bot fetch the wikipedia article on xyz if it exists.
Just a thought

Good idea. Are you planning on working on it?

I can work on it if I'm assigned to do so.
I say assigned because I'm slow with these things now, so it would take me a little longer than it would take an experienced guy to code it up and then I'd hate to code it up just to see someone else having done it before me.
So if I can be given the job with some time to do it, and that no-one else is to do it meanwhile, then sure, I'd love to work on it.

repo: python-discord/bot


text:
Restore wheel building on Windows and Python 2.7

Can't build brotli 1.0.9 on Windows + Python 2.7 due to https://github.com/google/brotli/issues/848, will fix this once the linked issue is addressed.

Python 2.7 is EOL, closing this issue.

repo: python-hyper/brotlicffi


text:
Python-docx: Structure problem. One Paragraph contains two objects.

Hi, the situation is I want to print the text from the paragraph.
When I code
doc.paragraphs[i].text
It comes two lines. Like:
ABC
DEFG
Than I think it contains two lists, So code as blow to get the first list only:
doc.paragraphs[i].text[0]
However, the result is not like what I think, it becomes:
A
D
I have no idea why it could be that?
How can I do to get the first list "ABC" ??
btw. [i] is one of paragraphs from document

Please post support questions on StackOverflow using the python-docx tag.

@scanny hello, I have already posted my questions on StackOverflow.
But still no one know how to do.
The link is: https://stackoverflow.com/q/69039644/10870868

@Huiicat you're not providing the minimal reproducible code example and so you're not giving us enough to go on to help you. What we need is something like this:
>>> doc = Document("my-doc.docx")
>>> paragraph = doc.paragraphs[17]  # --- the problematic one ---
>>> paragraph.text
'ABC
DEFG'
... etc.

authors: Huiicat scanny

repo: python-openxml/python-docx


text:
Export fails due to urllib3 version mismatch

I'm using version 1.5.0 of this plugin.
$ poetry --version
Poetry (version 1.5.1)

$ poetry export -f requirements.txt --without-hashes -o requirements/prod.txt

The `urllib3` package has the following compatible candidates `[Package('urllib3', '1.26.17')]`;
but, the exporter dependency walker previously elected `urllib3 (2.0.6)` which is not compatible
with the dependency `urllib3 (>=1.25.4,<1.27)`. Please contribute to `poetry-plugin-export` to
solve this problem.

$ poetry self add urllib3
Using version ^2.0.6 for urllib3

Updating dependencies
Resolving dependencies... (0.0s)

Because poetry-instance depends on poetry (1.5.1) which depends on urllib3 (>=1.26.0,<2.0.0),
urllib3 is required. So, because poetry-instance depends on urllib3 (^2.0.6), version solving failed.

Not quite sure how I could contribute..

I've seen this across multiple versions of Poetry: 1.3.2, 1.4.2, 1.5.1, and 1.6.1.

It appears to be related to #183

I have a similar problem I guess. I'm using Poetry version of 1.4.2 which has been depending on poetry-plugin-export 1.5.0 till 2 days ago. Then I started to get the following error:
#9 12.84 virtualenv 20.24.6 requires platformdirs<4,>=3.9.1, but you'll have platformdirs 2.6.2 which is incompatible.
#9 12.84 poetry-plugin-export 1.6.0 requires poetry<2.0.0,>=1.6.0, but you'll have poetry 1.4.2 which is incompatible.
#9 12.84 poetry-plugin-export 1.6.0 requires poetry-core<2.0.0,>=1.7.0, but you'll have poetry-core 1.5.2 which is incompatible.
#9 12.84 Successfully installed SecretStorage-3.3.3 attrs-23.1.0 build-0.10.0 cachecontrol-0.12.14 cffi-1.16.0 cleo-2.1.0 crashtest-0.4.1 cryptography-41.0.5 distlib-0.3.7 dulwich-0.21.6 filelock-3.13.1 html5lib-1.1 importlib-metadata-6.8.0 importlib-resources-6.1.0 installer-0.7.0 jaraco.classes-3.3.0 jeepney-0.8.0 jsonschema-4.19.2 jsonschema-specifications-2023.7.1 keyring-23.13.1 lockfile-0.12.2 more-itertools-10.1.0 msgpack-1.0.7 packaging-23.2 pexpect-4.8.0 pkginfo-1.9.6 pkgutil-resolve-name-1.3.10 platformdirs-2.6.2 poetry-1.4.2 poetry-core-1.5.2 poetry-plugin-export-1.6.0 ptyprocess-0.7.0 pycparser-2.21 pyproject-hooks-1.0.0 rapidfuzz-3.5.1 referencing-0.30.2 requests-toolbelt-0.10.1 rpds-py-0.10.6 shellingham-1.5.4 tomli-2.0.1 tomlkit-0.12.1 trove-classifiers-2023.10.18 urllib3-1.26.18 virtualenv-20.24.6 webencodings-0.5.1 zipp-3.17.0
#9 13.01 WARNING: You are using pip version 20.2.4; however, version 23.3.1 is available.
#9 13.01 You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
#9 DONE 13.2s

#11 [4/8] COPY . /app
#11 DONE 0.0s

#12 [5/8] WORKDIR /app
#12 DONE 0.0s

#13 [6/8] RUN eval $(ssh-agent -s) && mkdir -p ~/.ssh && chmod 0600 ~/.ssh &...
#13 0.278 Agent pid 8
#13 0.304 # gitlab.trendyol.com:22 SSH-2.0-OpenSSH_7.4
#13 0.314 # gitlab.trendyol.com:22 SSH-2.0-OpenSSH_7.4
#13 0.322 # gitlab.trendyol.com:22 SSH-2.0-OpenSSH_7.4
#13 0.330 # gitlab.trendyol.com:22 SSH-2.0-OpenSSH_7.4
#13 0.337 # gitlab.trendyol.com:22 SSH-2.0-OpenSSH_7.4
#13 DONE 0.4s

#14 [7/8] RUN poetry export -f requirements.txt --without-hashes --without-u...
#14 1.298 /usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.18) or chardet (3.0.4) doesn't match a supported version!
#14 1.298   warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
#14 1.298
#14 1.298 to_pep_508() got an unexpected keyword argument 'resolved'
#14 ERROR: executor failed running [/bin/bash -c poetry export -f requirements.txt --without-hashes --without-urls --without dev --output requirements.txt]: runc did not terminate sucessfully
------
 > [7/8] RUN poetry export -f requirements.txt --without-hashes --without-urls --without dev --output requirements.txt:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/bash -c poetry export -f requirements.txt --without-hashes --without-urls --without dev --output requirements.txt]: runc did not terminate sucessfully

It seems that two days ago, a new version of poetry-plugin-export was released. Since Poetry depends poetry-plugin-export<2.0.0,>=1.3.0, it install  the new version of 1.6.0 that is not compatible with other dependencies.
I solved the problem by installing specifically poetry-plugin-export==1.5.0 before installing poetry==1.4.2

I see this same issue my my personal project.
$ poetry export -f requirements.txt --output requirements.txt --without-hashes

The `urllib3` package has the following compatible candidates `[Package('urllib3', '1.26.18')]`;  but, the exporter dependency walker previously elected `urllib3 (2.0.7)` which is not compatible with the dependency `urllib3 (>=1.25.4,<1.27)`. Please contribute to `poetry-plugin-export` to solve this problem.

$ poetry --version
Poetry (version 1.7.1)

poetry-plugin-export==1.5.0

This is not working for me.
Has anyone found a workaround for this?

The below solution worked for me:

Manually updated the version of urllib3 in the poetry.lock file

Running this in the terminal
$ poetry export -f requirements.txt --output requirements.txt --without-hashes

I've temporarily fixed this for myself by requiring urllib3 = "<2" in pyproject.toml

I've temporarily fixed this for myself by requiring urllib3 = "<2" in pyproject.toml

This worked for me. Thanks!

Got this error:
The `scipy` package has the following compatible candidates `[Package('scipy', '1.8.1', source_type='legacy', source_url='http://local.dc.local:8081/repository/pypigr/simple', source_reference='local.dc.local')]`;  but, the exporter dependency walker previously elected `scipy (1.10.1)` which is not compatible with the dependency `scipy (>=1.3,<1.9)`. Please contribute to `poetry-plugin-export` to solve this problem.

Not sure if that related, but some of the errors are the same
Any workaround or solution?

added the following to my pyproject.toml
urllib3 = ">=1.26,<2"

fixes it but it's deffo super ugly...

authors: agsimmons david-nano farukcankaya fmagno jmuddle-via loleg ltrojan-ths mikemadden42 pcha457 robertpro

repo: python-poetry/poetry-plugin-export


text:
poetry export doesn't include non-optional groups

The poetry export --help description mentions this:

--with=WITH            The optional dependency groups to include.

However, it must also be used to include non-optional groups.
e.g.:
Given the following example pyproject.toml: https://gist.github.com/jonapich/80b8090d610da02d5de5137ca96da253
I get the following trace:
> poetry export | grep tenacity
(empty)

> poetry export --with my | grep tenacity
tenacity==8.2.2 ; python_version >= "3.8" and python_version < "3.12" \

I'm on poetry 1.5.1 on Windows, but it was reproduced on Linux as well.

It seems like the default poetry export should match poetry install in terms of what gets exported.  Are breaking changes allowed at this point?

Are breaking changes allowed at this point?

I am not aware of anything else that would require a major version bump so we would do it only for this change. Not sure if we want to do this.
Apart from that, I do not have a strong opinion on this. (If we should change it at all.) I think there are pros and cons for the current behavior:

pro current behavior: I assume most users want to export only the main group because for development they use Poetry and do not need to export requirements.txt files; they need them only for production and there you normally want to install only main dependencies.
contra current behavior: obviously inconsistent with other Poetry commands

@Secrus Do you have an opinion on this topic?

authors: bperkins24 jonapich radoering


text:
Update JSON file during semantic release

Question
Hello, I want to know if it is possible to update a JSON file using the resulting tag of the semantic release before upload the release to GitHub.
I tried something, but my plugin.json file is not updated by the CI/CD, should I use the assets key in the TOML file to specify the path to my JSON file, or manually add commit and push in the CI/CD script ?
Configuration
I use GitHub Action to run semantic release, here is the CI/CD:
name: "Run Semantic Release"

on:
  push:
    branches: [main]

jobs:
  release:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.9"]
        poetry-version: ["1.8.3"]
    permissions:
      contents: write
      id-token: write
    concurrency: release
    steps:
      #----------------------------------------------
      # Check-out repository
      #----------------------------------------------
      - name: Check out code
        uses: actions/checkout@v4
      #----------------------------------------------
      # Setup Workflow
      #----------------------------------------------
      - name: Setup Workflow
        uses: ./.github/actions/setup
        with:
          python-version: ${{ matrix.python-version }}
          poetry-version: ${{ matrix.poetry-version }}
          load-cache: true
      #----------------------------------------------
      # Run Semantic Release
      #----------------------------------------------
      - name: Python Semantic Release
        id: release
        uses: python-semantic-release/python-semantic-release@v8.0.0
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
      #----------------------------------------------
      # Install JQ to parse JSON
      #----------------------------------------------
      - name: Install JQ
        run: sudo apt-get update && sudo apt-get install jq
        shell: bash
      #----------------------------------------------
      # Update Plugin Version
      #----------------------------------------------
      - name: Update Plugin Version
        env:
          tag: ${{ steps.release.outputs.tag }}
        run: |
          #!/bin/bash
          # Update the version in plugin.json directly
          jq --arg version "$tag" '.version = $version' plugin.json > tmp.json && mv tmp.json plugin.json
        shell: bash
      #----------------------------------------------
      # Publish
      #----------------------------------------------
      - name: Publish package ditributions to GitHub Releases
        uses: python-semantic-release/upload-to-gh-release@main
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}

Semantic Release Configuration:
[tool.semantic_release]
version_variables = ["python-lib/salesforecast_v2/__init__.py:__version__"]
version_toml = ["pyproject.toml:tool.poetry.version"]
assets = []
tag_format = "{version}"
commit_message = "conf: update plugin to {version} version"

[tool.semantic_release.branches.main]
match = "main"

[tool.semantic_release.commit_parser_options]
allowed_tags = [
    "add",
    "rm",
    "build",
    "conf",
    "config",
    "cfg",
    "ci",
    "docs",
    "feat",
    "fix",
    "perf",
    "style",
    "refacto",
    "test",
]
minor_tags = ["feat"]
patch_tags = ["fix", "perf"]

[tool.semantic_release.publish]
dist_glob_patterns = ["dist/*"]

Thanks in advance for your help !

Add plugin.json to the version_variables array. It will require a version to exist inside the quotation marks but it should work if the key value is in the json file. This has worked for yaml files but I can't recall directly for json but the regex is flexible in punctuation for most languages variable specification. If you have more than one version key, it likely will not work as you intended. If you can provide the example of the plugin file that would also be helpful.
In the case it doesn't work, you would need to write a script that is executed during the build_command. You can refer to PSR for this example as we update versions within the docs.
If you have to do the latter, it is important to consider if you want to manually do a git add in the next step or you want to add the file to the assets list. Adding to the assets list will ensure that the change does get added to the repository index however it comes with additional logic where your file will be individually uploaded to the assets section of your release on GitHub.

Thanks for the precisions!
The json file look like this:
{
    "id": "sales-forecast",
    "version": "1.4.4",
    "meta": {
        "label": "Sales forecast",
        "description": "...",
        "author": "....",
        "icon": "icon-money",
        "tags": [
            "Dataiku plugin"
        ],
        "url": "",
        "licenseInfo": "Apache Software License"
    }
}

I will try to simply add plugin.json:version in the version_variables array! Thanks for your answer.

Looking at the file contents, it should work just fine.
Just make sure your entry includes the path to the plugins file before the colon. I don't know your file structure so I made it generic in my recommendation.

Yep, it is at the root of the directory, so should be good ! Thank you very much. I will let you know if it works asap.

If you want to try it out locally without needing a release environment pipeline, use the following:
pip install python-semantic-release
semantic-release version --no-changelog --no-commit --no-tag

git status
git diff plugins.json

Hum unfortunately, it didn't work, but it updates correctly a __version__ in a Python file. So should be due to the JSON format.

Ok, I'll check on my side

@AdrienRiauxExt, I got it to work, it was the quotes that were not accounted for by default.  If you change the spec to a TOML raw string (using ''') and then wrap your version with quotes it works.
version_variables = [
  '''plugins.json:"version"''',
]

Oh thanks you're amazing ! That's more than enough :)
One last question haha, I tried the second option you mentioned (like in PSR for the docs), in the build_command what does the
python -m pip install -e .[build]
python -m build .

? And is it necessary ?

# [1] install project . (current directory) as an editable install, with the optional dependency set called 'build'
python -m pip install -e .[build]

# [2] run the module named 'build' on the current directory to create a sdist & wheel
python -m build .

The module build is the chosen build frontend that was recommended if you came from the setuptools ecosystem as it is their replacement after the PEP 517/518 rewrite.  There are many other build systems to choose instead.

And is it necessary ?

If you are using the GitHub Action, you must install the build tool into the environment because it is currently written as a Docker Action.  A Docker action will build a container at the beginning of the job and then mount your project into the environment. Unfortunately, this means that any dependencies you previously installed do not exist in the container (see #723) but PSR does.  The recommended fix for this (until #692 is released in v10) is to add the installation step into the build_command to ensure it works (ie step 1).  Secondly, it is also very important to manage your dependency version of the build tool just as you would any other dependency.  You could hard code a version into the build command specification (which is what we had done at first) but then since it is not located with all your other versions that means it never gets updated unless you happen to stumble upon it.  We have Dependabot configured for the repository so this automatically opens PRs when there are new versions available to notify us. Since the version was in the build command, Dependabot never notified us.  So instead in #921, we defined our build dependency as an optional dependency group and updated the version there and with step 1, we always ensure the correct build tool version is installed in the GitHub Action environment.
A variant of Step 2 is required as that is the command that actually performs the build of the python distribution artifacts and we have ensured it will work because of step 1.

Ok understand. Thanks a lot for your fast and complete answers!

You're welcome. I will close it with the new release that will include the slight modification to the regex.  At that point you will no longer need to specify the quotes in the variable name.

:tada: This resolution has been included in version 9.8.9 :tada:
The release is available on:

PyPi package
GitHub release

You will no longer need to specify the quotes in the variable name.


text:
Expert parallelism / MoE example would be awesome :)

I loved seeing the blog post with a simple, standalone implementation of many techniques used in production to speed up LLMs. Would love to see this extended to MoE like Mixtral, which at the moment seem fairly annoying to use and hack on. Curious how torch.compile can help with these, and possible issues that might arise like graph breaks due to gating.

@andersonbcdefg We have added the support of Mixtral-8x7B MoE, please check https://github.com/pytorch-labs/gpt-fast/pull/71. Feel free to try and share feedback.

authors: andersonbcdefg yanboliang

repo: pytorch-labs/gpt-fast


text:
Refactor unit tests in functional

The PR addresses these issues:

In autograd tests, the complex dtype is not considered. Add torch.cdouble support for the Tensors.
In batch_consistency tests, the assert_batch_consistency method only accepts single Tensor, this is not applicable to some methods, for example, lfilter and filtfilt requires three Tensors as the arguments, hence they don't use assert_batch_consistency in the tests. Refactor the test to accept a list of Tensors
Similar in torchscript_consistency tests, the func in each test method only accepts one tensor as the argument, for the other arguments for F.xxx method, they need to be defined inside the func. If there is no Tensor argument in the test, they use a dummy tensor which is not used anywhere. In this PR, we refactor _assert_consistency and _assert_consistency_complex to accept a list of inputs instead of just one tensor.

Updated the description.

Sure, will do.

Split the PR into three small PRs:

[ ] https://github.com/pytorch/audio/pull/2244
[ ] https://github.com/pytorch/audio/pull/2245
[ ] https://github.com/pytorch/audio/pull/2246

repo: pytorch/audio


text:
[WIP] Initial commit for gain

Skeleton for function to incorporate gain into torchaudio
 |  gain(self, gain_db=0.0, normalize=True, limiter=False, balance=None)
 |      Apply amplification or attenuation to the audio signal.
 |
 |      Parameters
 |      ----------
 |      gain_db : float, default=0.0
 |          Gain adjustment in decibels (dB).
 |      normalize : bool, default=True
 |          If True, audio is normalized to gain_db relative to full scale.
 |          If False, simply adjusts the audio power level by gain_db.
 |      limiter : bool, default=False
 |          If True, a simple limiter is invoked to prevent clipping.
 |      balance : str or None, default=None
 |          Balance gain across channels. Can be one of:
 |           * None applies no balancing (default)
 |           * 'e' applies gain to all channels other than that with the
 |              highest peak level, such that all channels attain the same
 |              peak level
 |           * 'B' applies gain to all channels other than that with the
 |              highest RMS level, such that all channels attain the same
 |              RMS level
 |           * 'b' applies gain with clipping protection to all channels other
 |              than that with the highest RMS level, such that all channels
 |              attain the same RMS level
 |          If normalize=True, 'B' and 'b' are equivalent.

See comment in #260 for VCTK.

TODO: Add filters to dither
Options:            Lipshitz - `L`
                    F-Weighted - `FW`
                    Modified-E-Weighted - `MEW`
                    Improved-E-Weighted - `IEW`
                    Gesemann - `G`
                    Shibata - `S`
                    Low-Shibata - `LS`
                    High-Shibata - `HS`

authors: CamiWilliams vincentqb


text:
Add a different bucketbatch DataPipe with max_token_num

🚀 The feature
Requested by audio domain to have a bucketbatch with max_token_num (maybe also with infinite buffer).
Motivation, pitch
A DataPipe to have similar logic like https://github.com/pytorch/audio/pull/2147
min_len and max_len can be achieved by a filterDataPipe.
Alternatives
We may be able to combine it with the existing bucketbacth. And, max_token_num is mutual exclusive with batch_num.
Additional context
No response
cc: @nateanl

cc @parmeet

I've been thinking about this and I'm interested to consider how this implementation will look. One challenge I envision is with materialization of the "lengths list" which is used to create the batches. Perhaps this is a feature that's only supported in MapDataPipes unless there's some clever way to handle 1) getting lengths of all examples, 2) using the indices corresponding to those lengths to retrieve examples from the underlying data.

Thanks @erip ,
I was talking to @nateanl earlier about the same concern. One of the reasons thatIterDataPipe is introduced is to handle huge dataset that doesn't fit into memory. A streaming way to load data would be ideal. But, it means getting lengths of all examples may not be possible in all cases.
I consider a buffer size can be introduced for this DataPipe. If we set it as unlimited, we can load lengths into memory and run the bucket batch for all data. But, if the memory is limited, a partial bucket sampling maybe needed.
And, to save the memory, you may want to only load filename and path as the input data. And, after the batching, the actual data can be loaded and decoded based on the filename.
It's more like my high-level idea. It may be wrong when we get into detail.

In torchaudio we have torchaudio.info method that reads the length of the audio without loading the whole file into memory. This could boost the speed of loading all the lengths into memory.

I think that's probably a fine tradeoff as compared to having the entire dataset in memory and would work well under a finite-buffer implementation.

Add a reference from TF: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#bucket_by_sequence_length

Closing as this is done by #283

authors: ejguan erip nateanl

repo: pytorch/data


text:
Dimensions for WAV2VEC for downstream tasks

Please help me with the output dimensions for wav2vec.
For 0.5 second of audio, the output dimensions are [1x512x48]
For 1 second of audio, the output dimensions are [1x512x98]
For 2 second of audio, the output dimensions are [1x512x198]
For 3 second of audio, the output dimensions are [1x512x298]
Am I correct to say that the stride length of features is approx. 10ms?
Can we increase the stride length? say, we get a [1x512] vector every second?

@myleott   how to use the wav2vec for downstream tasks, such as emotion recognition, speaker recognition?
Thank you very much!

repo: pytorch/fairseq


text:
[BUG] Compatibility of shapes TanhNormal vs Normal in log prob.

Describe the bug
Compatibility of shapes TanhNormal vs Normal in log prob.
To Reproduce
import torchrl
loc = torch.zeros(1)
scale = torch.ones(1)
normal = Normal(loc=loc, scale=scale)
tanhnormal = TanhNormal(loc=loc, scale=scale, min=-minmax, max=minmax)
x = tanhnormal.sample((1000,))
n = normal.log_prob(x)
t = tanh_normal.log_prob(x)
print(n.shape)
print(t.shape)

torch.Size([1000, 1])
torch.Size([1000])

Expected behavior
Same shape between the two tensors.
System info
import torchrl, numpy, sys
print(torchrl.__version__, numpy.__version__, sys.version, sys.platform)

0.4.0+2cfd9b6 1.24.4 3.10.12

Checklist

[X] I have checked that there is no similar issue in the repo (required)
[X] I have read the documentation (required)
[X] I have provided a minimal working example to reproduce the bug (required)

This is because TanhNormal builds an Independent(Normal) base distribution:
https://github.com/pytorch/rl/blob/d68a900ed95a81c32ecf41e7bcdbb8629d618d63/torchrl/modules/distributions/continuous.py#L407
We could make this optional via a kwarg in constructor (and document it better) if you think that could help!
The goal here was to use this for SAC and PPO without requiring the users to pass the dist within an Independent which can be annoying and repetitive.

cc @skandermoalla

TorchRL also has an IndependentNormal premade for you. This has the same shapes as TanhNormal

authors: andreamiele matteobettini vmoens

repo: pytorch/rl


text:
load_in_8bit for model quantization

Does torchtune allow load_in_8bit or load_in_4bit before performing SFT or DPO fine-tuning on models? If not, what modifications are required to run training on quantized models?

Hey @SUMEETRM , good question! @ebsmothers , do you have any insights on this one?

@SUMEETRM thanks for creating the issue. We currently do not support training in precision lower than 16-bit as it generally leads to poor performance. However, load_in_4bit (as used in Hugging Face) is also used for QLoRA-style usage of NF4, which we do support. If you can share a bit more about what you're trying to do I'd be happy to give some more pointers here.

authors: SUMEETRM ebsmothers felipemello1

repo: pytorch/torchtune