text:
Explicit indexes in xarray's data-model (Future of MultiIndex)

I think we can continue the discussion we have in #1426 about MultiIndex here.
In comment , @shoyer recommended to remove MultiIndex from public API.
I agree with this, as long as my codes work with this improvement.
I think if we could have a list of possible MultiIndex use cases here,
it would be easier to deeply discuss and arrive at a consensus of the future API.
Current limitations of MultiIndex are

It drops scalar coordinate after selection #1408, #1491
It does not support to serialize to NetCDF #1077
Stack/unstack behaviors are inconsistent #1431

I'm using MultiIndex a lot,
but I noticed that it is just a workaround to index along multiple kinds of coordinate.
Consider the following example,
In [1]: import numpy as np
   ...: import xarray as xr
   ...: da = xr.DataArray(np.arange(5), dims=['x'],
   ...:                   coords={'experiment': ('x', [0, 0, 0, 1, 1]),
   ...:                           'time': ('x', [0.0, 0.1, 0.2, 0.0, 0.15])})
   ...:

In [2]: da
Out[2]:
<xarray.DataArray (x: 5)>
array([0, 1, 2, 3, 4])
Coordinates:
    experiment  (x) int64 0 0 0 1 1
    time        (x) float64 0.0 0.1 0.2 0.0 0.15
Dimensions without coordinates: x

I want to do something like this
da.sel(experiment=0).sel(time=0.1)

but it cannot.
MultiIndexing enables this,
In [2]: da = da.set_index(exp_time=['experiment', 'time'])
   ...: da
   ...:
Out[2]:
<xarray.DataArray (x: 5)>
array([0, 1, 2, 3, 4])
Coordinates:
  * exp_time    (exp_time) MultiIndex
  - experiment  (exp_time) int64 0 0 0 1 1
  - time        (exp_time) float64 0.0 0.1 0.2 0.0 0.15
Dimensions without coordinates: x

If we could make a selection from a non-index coordinate,
MultiIndex is not necessary for this case.
I think there should be other important usecases of MultiIndex.
I would be happy if anyone could list them in this issue.

One API design challenge here is that I think we still want a explicit notation of "indexed" variables. We could possibly allow operations like .sel() on non-indexed variables, but they would be slower, because we would not want to create expensive hash-tables (i.e., pandas.Index) in a non-transparent fashion.

I sometimes find it helpful to think about what the right repr() looks right, and then work backwards from there to the right data model.
For example, we might imagine that "Indexes" are no longer coordinates, but instead a first-class notion on their own,
<xarray.Dataset (exp_time: 5)>
Coordinates:
  * experiment  (exp_time) int64 0 0 0 1 1
  * time        (exp_time) float64 0.0 0.1 0.2 0.0 0.15
Indexes:
    exp_time: pandas.MultiIndex[experiment, time]

"Indexes" might not even need to be part of the main Dataset.__repr__, but it would certainly be the repr for Dataset.indexes. Other entries could include:
    time: pandas.Datetime64Index[time]
    space: scipy.spatial.KDTree[latitude, longitude]

In this model:

We would promote "Indexes" to a first-class notion in the xarray data model:
(a) The levels of a MultiIndex would have corresponding Variable objects and be found in coords.
(b) In contrast, theMultiIndex would not have a corresponding Variable object or be part of coords, though it could still be returned upon __getitem__ access (computed on demand from .indexes).
(c) Dataset and DataArray would gain an indexes argument in their constructors, which could be used for passing indexes on to new xarray objects.
Coordinates marked with * are part of an index. They can't be modified, unless all corresponding indexes ares removed.
Indexes would still be propagated, like coordinates.

I think we currently assume variables[dim] is an Index.
Does your proposal means that Dataset will keep an additional attribute indexes, and indexes[dim] gives a pd.Index (or pd.MultiIndex, KDTree)?
It sounds a much cleaner data model.

Does your proposal means that Dataset will keep an additional attribute indexes, and indexes[dim] gives a pd.Index (or pd.MultiIndex, KDTree)?

Yes, exactly. We actually already have an attribute that works like this, but it's computed lazily, from either Dataset._variables or DataArray._coords.

CC @benbovy @fmaussion

I think that promoting "Indexes" to a first-class concept is indeed a very good idea, at both internal and public levels, even if at the latter level it would be another concept for users (it should be already familiar for pandas users, though). IMHO the "coordinate" and "index" concepts are different enough to consider them separately.
I like the proposed repr for Dataset.indexes. I wouldn't mind if it is not included in Dataset.__repr__, considering that multi-indexes, kdtree, etc. only represent a few use cases. In too many cases it could result in a long, uninformative list of simple pandas.Index.
I have to think a bit more about the details but I like the idea.

@shoyer, could you add more details of this idea?
I think I do not yet fully understand the practical difference between dim and index.

Use cases of the independent Index and dims
Would it general case where dimension and index are independent?
(It is the case only for MultiIndex and KDtree)?

MultiIndex implementation
In MultiIndex case, will a xarray object store a MultiIndex object and also the level variables as Variable objects (there will be some duplicates)?
If indexes[dim] returns multiple Variables, which realizes a  MultiIndex-like structure without pd.MultiIndex, indexes would be very different from dim,
because a single dimension can have multiple indexes.

Use cases of the independent Index and dims
Would it be general cases where dimension and index are independent?
(It is the case only for MultiIndex and KDtree)?

We would still assign default indexes (using a normal pandas.Index) when you assign a 1D coordinate with matching name and dimension. But in general, yes, it seems like you should be able to make an index even for variables that aren't dimensions, including for a 1D variable whose name does not match a dimension. The rule would be that any coordinates can be part of an index.
Another aspect to consider how to handle alignment when you have indexes along non-dimension coordinates. Probably the most elegant rule would again be to check all indexed variables for exact matches.
Directly assigning indexes rather than using this default or set_index() would be an advanced feature, not recommended for everyday use. The main use case is routines which create a new xarray object based on an existing one, and want to re-use old indexes.
For performance reasons, we probably do not want to actually check the values of manually assigned indexes, although we should verify that the shape matches. (We would have a clear disclaimer that if you manually assign an index with mismatched values the behavior is not well defined.)
In principle, this data model would allow for two mostly equivalent indexing schemes: MultiIndex[time, space] vs two indexes Index[time] and Index[space]. We would need to figure out how to propagate and compare indexes like this. (I suppose if the coordinate values match, the result could have the union of all indexes from input arguments.)

MultiIndex implementation
In MultiIndex case, will a xarray object store a MultiIndex object and also the level variables as Variable objects (there will be some duplicates)?

Yes, this is a little unfortunate. We could potentially make a custom wrapper for use in IndexVariable._data on the level variabless that lazily computes values from the MultiIndex (similar to our LazilyIndexedArray class), but I'm not certain yet that this is necessary.

If indexes[dim] returns multiple Variables, which realizes a MultiIndex-like structure without pd.MultiIndex, indexes would be very different from dim,
because a single dimension can have multiple indexes.

Every entry in indexes should be a single pandas.Index or subclass, including MultiIndex (possibly eventually allowing for index-like objects such as something based on a KDTree).

Thanks for the details.
(Sorry for my late responce. It took a long for me to understand what does it look like.)
I am wondering what the advantageous cases which are realized with this Index concept are.
As far as my understanding is correct,

It will enable more flexible indexing, e.g. more than one Indexes are associated with one dimension and we can select from these coordinate values very flexibly.
It will naturally integrate more advanced Indexes such as KDTree

Are they correct?

Probably the most elegant rule would again be to check all indexed variables for exact matches.

That sounds reasonable.

In principle, this data model would allow for two mostly equivalent indexing schemes: MultiIndex[time, space] vs two indexes Index[time] and Index[space].

I like the latter one, as it is easier to understand even for non-pandas users.
What does the actual implementation look like?
xr.Dataset.indexes will be an OrderedDict that maps from variable's name to its associated dimension?
Actual instance of Index will be one of xr.Dataset.variables?

I am wondering what the advantageous cases which are realized with this Index concept are.

The other advantage is that it solves many of the issues with the current MultiIndex implementation. Making MultiIndex levels their own variables considerably simplifies the data model, and means that many features (including serialization) should "just work".

In principle, this data model would allow for two mostly equivalent indexing schemes: MultiIndex[time, space] vs two indexes Index[time] and Index[space].
I like the latter one, as it is easier to understand even for non-pandas users.

I agree, but there are probably some advantages to using a MultiIndex internally. For example, it allows for looking up on multiple levels at the same time.

What does the actual implementation look like?
xr.Dataset.indexes will be an OrderedDict that maps from variable's name to its associated dimension?
Actual instance of Index will be one of xr.Dataset.variables?

I think we could get away with making indexes simply a dict, with keys given by index names and values given by a pandas.Index instance. We should enforce that Index.name or MultiIndex.names corresponds to coordinate variables.
For KDTree, this means we'll have to write our own wrapper KDTreeIndex that adds a names property, but we would probably need to add special methods like get_indexer anyways.

Just to say I'm interested in how MultiIndexes are handled also. In our use case, we have two variables conventionally named CHROM (chromosome) and POS (position) which together describe a location in a genome. I want to combine both variables into a multi-index so I can, e.g., select all data from some data variable for chromosome X between positions 100,000-200,000. For all our data variables, this genome location multi-index would be used to index the first dimension.

Will the new API preserve the order of the levels?  One of the features that's necessary for MultiIndex to be truly hierarchical is that there is a defined order to the levels.

@jjpr-mit can you explain your use case a little more? What sort of order dependent queries do you want to do? The one that comes to mind for me are range based queries, e.g, [('bar', 1) : ('foo', 9)].
I think it is still relatively easy to ensure a unique ordering between levels, based on the order of coordinate variables in the xarray dataset.
A bigger challenge is that for efficiency, these sorts of queries depend critically on having an actual MultiIndex. This means that if indexes for each of the levels arise from different arguments that were merged together, we might need to "merge" the separate indexes into a joint MultiIndex. This could potentially be slightly expensive.

I've been thinking about getting started on this. Here are my current thoughts on the right design approach.

Data model
Dataset.indexes and DataArray.indexes
My current thinking is that indexes should simply be a dictionary mapping from coordinate and/or dimension names to pandas.Index objects. Mapping from label-based to integer-based then becomes simply a matter of looking up the appropriate indexes for each coordinate/dimension (i.e., the keyword argument names in .sel()), and using the corresponding index(es) to transform label-based indexers into integer indexers.
If multiple coordinates are part of the same index, they should point to the same MultiIndex/KDTree object. The MultiIndex would be responsible for resolving the combined indexing operation along the coordinate dimension(s).
By default, indexes is populated with an Index/MultiIndex for each dimension of all indexes along that dimension. Additional indexes may be set manually, e.g., using set_index().
Indexes keyed by a dimension name are used for axis-positional indexing with .loc and for alignment with reindex/align. However, if the index is a MultiIndex with a level name matching a coordinate, then only that level will be used for indexing/alignment. In other words: the coordinate name corresponding to indexing request takes precedence, but if it isn't found, we use all indexes along the dimension.
Separate indexers without a MultiIndex should be prohibited
It should be impossible to express inconsistent and/or confusing states in xarray's data model. This sort of inconsistency (e.g., levels not being stored directly in Dataset.variables) is the major source of our issues with the current MultiIndex data model.
I'm particularly concerned about the clearly showing difference between coordinates that are part of a MultiIndex and coordinates that are separately indexed. I suspect we could make indexing operations nearly equivalent from a user perspective, but there would likely remain small differences that would be a source of confusion and bugs. Preserving indexes in the form in which they are created is not also not really an option, because there are lots of xarray operations that would probably normalize indexes into standard forms, such as groupby, stack/unstack and to/from_pandas.
The simplest option is to prohibit one of these cases entirely, either:

Always group repeated indexes along a dimension into a MultiIndex, or
Never use pandas.MultiIndex (keep separate indexes for each coordinate).

From xarray's perspective, it would certainly be cleaner to prohibit MultiIndex. The level order depend behavior of MultiIndex is not the best fit for xarray's data model, and could be challenging to keep in sync with coordinate order on xarray objects. We would need to ensure that coordinate/level order remains consistent in all operations, or at least ensure that coordinates are always printed in order of their appearence in MultiIndex levels. (We generally preserve coordinate order already, but well behaved programs using xarray currently don't need to rely on this behavior.)
That said, always using MultiIndexes for multiple indexes along the same dimension has it's own clear advantages. First, it's consistent with pandas, which makes it easier to transition data back and forth. Second, simultaneous indexing operations across MultiIndex levels would be difficult to express efficiently with a MultiIndex. This is probably the right choice for xarray.
We could potentially allow for non-consolidated indexes (not part of a MultiIndex) when using the advanced API (e.g., directly setting the indexes parameter). But we'll save this for later.
Functionality
Index variables
Every MultiIndex level must have a corresponding xarray.Variable object in coordinates on each Dataset/DataArray on which they appear. These objects may reference the same pandas.Index/pandas.MultiIndex object used for indexing, but must have immutable data (e.g., flag.writeable = False in NumPy). For now, I expect to reuse the existing IndexVariable class.
Now that levels are xarray.Variable objects, there will no longer be a Variable object in Dataset._variables/DataArray._coords corresponding to a pandas.MultiIndex. However, we will continue to create a "virtual variable" upon indexing consisting of an dtype=object array of MultiIndex values, as a fallback if there is no coordinate matching a dimension name.
Mapping indexes into pandas
Another concern is how to map all of the new possible indexing states into pandas:
# case 1 (one indexed variable, same name as dimension):
  * time  (time)

# case 2 (one indexed variable, different name from dimension):
  * year  (time)

# case 3 (multiple indexed variables, one has same name as dimension):
  * time  (time)
  * year  (time)

# case 4 (multiple indexed variables, all have different names from dimension):
  * year   (time)
  * month  (time)

For consistency with current behavior, case 1 should correspond to a standard pandas.Index and case 4 should correspond to a pandas.MultiIndex. But what about the intermediate cases 2 and 3, which are currently prohibited by xarray's data model?
I think we should use the rule that all indexed variables are consolidated into a single Index in pandas. If there are multiple indexed variables (case 3 or 4), this would be a MultiIndex; otherwise (cases 2 or 3), this would be a standard Index. This has a virtue of speed and simplicity: we can simply reuse the existing Index or MultiIndex object from indexes.
The other option would be prohibit cases 2 and 3 (like we currently do), because we will not be able to map them into pandas and back faithfully. I think this would be a mistake, because indexes on multiple levels would be useful for xarray, even if one level corresponds to the dimension name.
Indexes for unstack
With the introduction of more flexible and optional index levels, it may not always may sense to unstack() every index coordinate. We should support optionally specifying levels to unstack, possibly with an API mirroring stack(), e.g., perhaps .unstack(dim_name=['level_0', 'level_1']) to unstack  coordinates level_0 and level_1 from dimension dim_name.

@shoyer, thank you for detailing.
I am thinking how can we establish the following selecting-concatenating behavior with MultiIndex(-like) coordinate with our new Indexes machinary,
xr.concat([da.isel(x=i) for i in range(len(da['x':))], dim='x')

Personally, I think it would be nice if we could recover the original Index structue.
We may need to track Indexes object even when the corresponding dimension becomes one dimensional?
But one dimensional index sounds strange...
Or, we may give up to restore the original coordinate structure during the above action, but stil keep them as ordinary coodinates.

@fujiisoup Yes, we certainly could add a "N-dimensional index", even if it has no function other than a placeholder to mark a variable as an index. This would let us restore index state after selecting/concatenating along a dimension.
However, I'm not sure it would be a satisfactory solution. If we keep these indexes around like coordinates, we could end up with scalar coordinates from different dimensions. Then it's still not clear how they should stack up in the final result -- we would have the same issue we currently have with concatenating coordinates.
The other concern is that existence and behavior of scalar/N-dimensional indexes could be a surprising. What does it mean to index an N-dimensional index? This operations probably cannot be supported in a sensible way, or at least not without significant effort.

Overall, I agree with the proposed conclusion. And appreciate the level of thoughtfulness and clarity. I'm happy to help with some of the implementation if we can split this up.

I started thinking about how to do this incrementally, and it occurs to me that a good place to start would be to write some of the utility functions we'll need for this:

Normalizing and creating default indexes in the Dataset/DataArray constructor.
Combining indexes from all xarray objects that are inputs for an operations into indexes for the outputs.
Extracting MultiIndex objects from arguments into Dataset/DataArray and expanding them into multiple variables.

I drafted up docstrings for each of these functions and did a little bit of working starting to think through implementations in  https://github.com/pydata/xarray/pull/2195. So this would be a great place for others to help out. Each of these could be separate PRs.

I see this mentions an Index API, is that still to be decided?
On Tue, 29 May 2018, 05:28 Stephan Hoyer, notifications@github.com wrote:

I started thinking about how to do this incrementally, and it occurs to me
that a good place to start would be to write some of the utility functions
we'll need for this:

Normalizing and creating default indexes in the Dataset/DataArray
constructor.
Combining indexes from all xarray objects that are inputs for an
operations into indexes for the outputs.
Extracting MultiIndex objects from arguments into Dataset/DataArray
and expanding them into multiple variables.

I drafted up docstrings for each of these functions and did a little bit
of working starting to think through implementations in #2195
https://github.com/pydata/xarray/pull/2195. So this would be a great
place for others to help out. Each of these could be separate PRs.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/1603#issuecomment-392649605, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QvMauEPa6hfgorDoShZ2PwyYWk6Tks5t3M6AgaJpZM4PtACU
.

Yes, the index API still needs to be determined. But I think we want to
support something like that.
On Tue, May 29, 2018 at 1:20 AM Alistair Miles notifications@github.com
wrote:

I see this mentions an Index API, is that still to be decided?
On Tue, 29 May 2018, 05:28 Stephan Hoyer, notifications@github.com
wrote:

I started thinking about how to do this incrementally, and it occurs to
me
that a good place to start would be to write some of the utility
functions
we'll need for this:

Normalizing and creating default indexes in the Dataset/DataArray
constructor.
Combining indexes from all xarray objects that are inputs for an
operations into indexes for the outputs.
Extracting MultiIndex objects from arguments into Dataset/DataArray
and expanding them into multiple variables.

I drafted up docstrings for each of these functions and did a little bit
of working starting to think through implementations in #2195
https://github.com/pydata/xarray/pull/2195. So this would be a great
place for others to help out. Each of these could be separate PRs.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/1603#issuecomment-392649605,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AAq8QvMauEPa6hfgorDoShZ2PwyYWk6Tks5t3M6AgaJpZM4PtACU
.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/1603#issuecomment-392692996, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABKS1p8RjrupPM2z2d4_ylWX7826RQ0Rks5t3QTHgaJpZM4PtACU
.

Ok, cool. Was wondering if now was right time to revisit that, alongside
the work proposed in this PR. Happy to participate in that discussion,
still interested in implementing some alternative index classes.
On Tue, 29 May 2018, 15:45 Stephan Hoyer, notifications@github.com wrote:

Yes, the index API still needs to be determined. But I think we want to
support something like that.
On Tue, May 29, 2018 at 1:20 AM Alistair Miles notifications@github.com
wrote:

I see this mentions an Index API, is that still to be decided?
On Tue, 29 May 2018, 05:28 Stephan Hoyer, notifications@github.com
wrote:

I started thinking about how to do this incrementally, and it occurs to
me
that a good place to start would be to write some of the utility
functions
we'll need for this:

Normalizing and creating default indexes in the Dataset/DataArray
constructor.
Combining indexes from all xarray objects that are inputs for an
operations into indexes for the outputs.
Extracting MultiIndex objects from arguments into Dataset/DataArray
and expanding them into multiple variables.

I drafted up docstrings for each of these functions and did a little
bit
of working starting to think through implementations in #2195
https://github.com/pydata/xarray/pull/2195. So this would be a great
place for others to help out. Each of these could be separate PRs.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/1603#issuecomment-392649605,
or mute
the thread
<

https://github.com/notifications/unsubscribe-auth/AAq8QvMauEPa6hfgorDoShZ2PwyYWk6Tks5t3M6AgaJpZM4PtACU

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/1603#issuecomment-392692996,
or mute
the thread
<
https://github.com/notifications/unsubscribe-auth/ABKS1p8RjrupPM2z2d4_ylWX7826RQ0Rks5t3QTHgaJpZM4PtACU
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pydata/xarray/issues/1603#issuecomment-392803210, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QgygnzTX053NlGZ5A5j_tRkRxMj7ks5t3V79gaJpZM4PtACU
.

I've been thinking about this a little more in the context of starting on the implementation (in #2195).
In particular, I no longer agree with this "Separate indexers without a MultiIndex should be prohibited" from my original proposal. The problem is that the semantics of a MultiIndex are not quite the same as separate indexes, and I don't think all use-cases are well solved by always using a MultiIndex. For example, I don't think it's possible to do point-wise indexing along anything other than the first level of a MultiIndex.
Instead, I think we should make the model transparent by retaining an xarray variable for the MultiIndex, and provide APIs for explicitly converting index types.
e.g., for the repr with a MultiIndex:
Coordinates:
  * x        (x) MultiIndex[level_1, level_2]
  * level_1  (x) object 'a' 'a' 'b' 'b'
  * level_2  (x) int64 1 2 1 2

and without a MultiIndex:
Coordinates:
  * level_1  (x) object 'a' 'a' 'b' 'b'
  * level_2  (x) int64 1 2 1 2

The main way in which this could get confusing is if you explicitly mutate the Dataset to remove some but not all of the variables corresponding to the MultiIndex (e.g., x but not level_1 or vise-versa). We have a few potential options here:

Don't worry about it: if you mutate objects, you can potentially end up in slightly confusing internal states. If you care about whether level_1 uses a pandas.Index or pandas.MultiIndex, you can find out for sure by checking ds.indexes['level_1'].
Prohibit it in our data model: either (a) raise an error if you try to manually delete a single variable or (b) automatically delete all associated variables, too. Encourage using various explicit APIs that return new objects with a new index.
Use a different indicator than * for marking "indirect" indexes, so it's more obvious if some coordinates get removed, e.g.,

Coordinates:
  * x        (x) MultiIndex[level_1, level_2]
  + level_1  (x) object 'a' 'a' 'b' 'b'
  + level_2  (x) int64 1 2 1 2

The different indicator might make sense regardless but I am also partial to "Prohibit it in our data model." The main downside is that this adds a little more complexity to the logic for determining indexes resulting from an operation (namely, verifying that all MultiIndex levels still correspond to coordinates).

Potentially this is too much 'stepping back' now we're at the implementation stage - my perception is that @shoyer is leading this without much support, so weighting having some additional viewpoints, some questions:
Is a MultiIndex a feature of the schema or the implementation?
I had thought of an MI being an implementation detail in code, rather than in the data schema. We use it as a container for all the indexes along a dimension, rather than representing any properties about the data it contains.
One exception to that would be if we wanted multiple groups of indexes along the same dimension, for example:
Coordinates:
  * xa         (x) MultiIndex[level_a_1, level_a_2]
  * level_a_1  (x) object 'a' 'a' 'b' 'b'
  * level_a_2  (x) int64 1 2 1 2

  * xb         (x) MultiIndex[level_b_1, level_b_2]
  * level_b_1  (x) object 'a' 'a' 'b' 'b'
  * level_b_2  (x) int64 1 2 1 2

But is that common / required?
MultiIndex as an implementation detail
If it's an implementation detail, is there a benefit to investing in allowing both separate and MIs?
While it may not be possible to do pointwise indexing with the current implementation of MI, am I mistaken that it's not an API issue, assuming we pass in index names? e.g.:
[ins] In [22]: da = xr.DataArray(np.arange(12).reshape((3, 4)), dims=['x', 'y'],
    coords=dict(x=list('abc'), y=pd.MultiIndex.from_product([list('ab'),[1,2]])))

[ins] In [23]: da
Out[23]:
<xarray.DataArray (x: 3, y: 4)>
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
Coordinates:
  * x          (x) <U1 'a' 'b' 'c'
  * y          (y) MultiIndex
  - y_level_0  (y) object 'a' 'a' 'b' 'b'
  - y_level_1  (y) int64 1 2 1 2

[ins] In [26]: da.sel(x=xr.DataArray(['a','c'],dims=['z']),
    y_level_0=xr.DataArray(['a','b'],dims=['z'])
    y_level_1=xr.DataArray([1,1],dims=['z']))

Out[80]: # hypothetical
<xarray.DataArray (z: 3)>
array([ 0,  10])
Dimensions without coordinates: z

If that's the case, could we instead force all indexes along a dimension to be in a MI, tolerate the short-term constraints of the current MI implementation, and where needed build out additional features?
That would (ideally) leave us uncoupled to MIs - if we built a better in-memory data structure, we could transition. The contract would be around the cases above.
--
...and as mentioned above, these are intended as questions rather than high-confident views.

@max-sixty I like your schema vs. implementation breakdown. In general, I agree with you that it would be nice to have MultiIndex has an implementation detail rather than part of xarray's schema. But I'm not entirely sure that's feasible.
Let's try to list out the pros/cons. Consider a MultiIndex 'multi' with levels 'x' and 'y':

Advantages of MultiIndex as part of the data schema:

There is an explicit coordinate (of tuples) corresponding to MultiIndex values, which can be returned from ds.coords['multi']. This is inherently not that useful compared to the separable variables, but is a cleaner solution that creating ds.coords['multi'] as a "virtual" variable on the fly (which we would need for backwards compatibility).
We don't need to do full "normalization" when multiple indexes along the same dimension are encountered, e.g., in an operation that combines two different indexes, we would simply put both on the result instead of building a MultiIndex (which would require allocating a whole new array of integer codes).
The nature of the MultiIndex is more transparent as part of the data model. For example, if x and y are numeric, it could make sense to use either a MultiIndex or KDTree for indexing. Explicit APIs (e.g., set_multiindex and set_kdtree) would allow users a high level of control.
For advanced use-cases, it is potentially easier to work around the limitations of a MultiIndex, e.g., the way that some operations require lex-sorted-ness.

Advantages of MultiIndex as an implementation detail:

Simpler data model (for users). There are few good use cases for multiple indexes that aren't a MultiIndex.
Easier to do automatic alignment: we know that indexes will always have the same normalized form (in a MultiIndex). Otherwise, we would have to do this on the fly, or request that users explicitly setup compatible indexes.
More flexibility for xarray: we can potentially swap out indexing without changing the user-facing API. We might have something like a "hybrid" MultiIndex/KDTree that chooses the appropriate index based on the requested operation.
We don't need to create an explicit array of tuples for the MultiIndex variable (but we could still have a variable corresponding to a MultiIndex and only construct the .data array in a "lazy" fashion).
There's no need to name extraneous variables that only exist for the sake of a MultiIndex.
There's no need to support indexing like ds.sel(multi=list_of_pairs). Indexing like ds.sel(x=..., y=...) solves the same use case and looks nicer. That said, this would be a minor backwards compatibility break (this currently works in xarray).

P.S. I haven't made much progress on this yet so there's definitely still time to figure out the right decision -- thanks for your engagement on this!

For example, I don't think it's possible to do point-wise indexing along anything other than the first level of a MultiIndex.

This is clearly not true, since it works in pandas:
``python
import pandas as pd
index = pd.MultiIndex.from_product([list('ab'),[1,2]])
series = pd.Series(range(4), index)
print(series.loc[:, [1, 2]])

That said, I still don't know how to use public `MultiIndex` methods for this. Neither `index.get_loc_level([1, 2], level=1)` nor `index.get_loc((slice(None), [1, 2]))` work.

That said, I still don't know how to use public MultiIndex methods for this. Neither index.get_loc_level([1, 2], level=1) nor index.get_loc((slice(None), [1, 2])) work.

The answer is the index.get_locs() method: index.get_locs([slice(None), 1, 2]]) works. It's painfully slow for large numbers of points due to a Python loop over each point, but presumably that could be optimized if necessary.

Let me make a tentative proposal: we should model a MultiIndex in xarray as exactly equivalent to a sparse multi-dimensional array, except with missing elements modeled implicitly (by omission) instead of explicitly (with NaN).

💯- that very much resonates!
And it leaves the implementation flexible if we want to iterate.
I'll try to think of some dissenting cases to the proposal / helpful responses to the above.

we will definitely have to make some intentional deviations from the behavior of pandas

Looking at the reported issues related to multi-indexes in xarray, I have the same feeling. Simply reusing pandas.MultiIndex in xarray where slightly different semantics are generally expected has shown to be painful. It seems easier to have our own baked solution and deal with differences during xarray<-> pandas conversion if needed.
If we re-design indexes so that we allow 3rd-party indexes, maybe we could support both and let the user choose the one (xarray or pandas baked) that best suits his needs?
Regarding MultiIndex as part of the data schema vs an implementation detail, if we support extending indexes (and already given the different kinds of multi-coordinate indexes: MultiIndex, KDTree, etc.), then I think that it should be transparent to the user.
However, I don't really see why a multi-coordinate index should have its own variable (with tuples of values).  I don't want to speak for others, but IMHO ds.sel(multi=list_of_pairs) is rather a edge case and I'm not sure if we really need to support it. Using ds.sel(x=..., y=...) with DataArray objects is certainly more code to write, but this form of indexing is very powerful and it might not be a bad idea to encourage it.
If a variable for each multi-coordinate index is "just" for data schema consistency, then why not showing all those indexes in a separate section of the repr? For example:
Coordinates:
  * level_1  (x) object 'a' 'a' 'b' 'b'
  * level_2  (x) int64 1 2 1 2
Multi-indexes:
    pandas.MultiIndex [level_1, level_2]

It is equally transparent, not more verbose, and it is clear that multi-indexes are not part of the coordinates (in fact there is no need of "virtual" coordinates either, nor to name the index). I don't think single indexes should be shown here as it would results in duplicated, uninformative lines.
More generally, here is how I would see indexes handled in xarray (I might be missing important aspects, though):

Default behavior: all 1-dimensional coordinates each have their own, single index (pandas.Index), unless explicitly stated.
Explicit API is used for setting new, possibly multi-coordinate indexes. Note the absence of keyword argument below to specify the variables: This is actually more consistent with the pandas API but this would be a breaking change and I don't know how a smooth transition could look like.

set_index(['x', 'y'], kind='multiindex')   # xarray built-in index
set_index(['x', 'y'], kind='kdtree')     # xarray built-in index
set_index('x', kind=ASingleIndexWrapperClass)    # 3rd-party index

If a coordinate is removed from the Dataset or if its index is reset or changed:

If the coordinate had a single index, no problem
If the coordinate was part of a multi-coordinate index: a new index is built from all remaining coordinates that were also part of the original index, if it is supported. Otherwise, the original index is removed and the default behavior (single pandas.Index) is reset for all those remaining coordinates.

I am late for the party (but still only have time to write a short comment).
I am a big fan of MultiIndex and like @shoyer 's idea.
ds.sel(multi=list_of_pairs) can probably be replaced by ds.sel(x=..., y=...), but how about reindex along MultiIndex?
I have encountered its use cases several times.
I also think it would be nice to have MultiIndex as a variable.

I broadly agree with @benbovy 's proposal.
One question that I think is worth being clear on is what additional contracts do multiple indexes on a dimension have over individual indexes?
e.g. re:
Coordinates:
  * level_1  (x) object 'a' 'a' 'b' 'b'
  * level_2  (x) int64 1 2 1 2
Multi-indexes:
    pandas.MultiIndex [level_1, level_2]

Am I right in thinking the Multi-indexes is only a helpful note to users, rather than conveying anything about how data is accessed?
@fujiisoup 's poses a good case of this question:

ds.sel(multi=list_of_pairs) can probably be replaced by ds.sel(x=..., y=...), but how about reindex along MultiIndex?

(and separately, I think we can do much of this before adding the ability to set custom indexes, which would be cool but further from where we are, I think)

And broadening out further:

Default behavior: all 1-dimensional coordinates each have their own, single index (pandas.Index), unless explicitly stated.

This is basically how I think of indexes - as a performant lookup data structure, rather than a feature of the schema. An RDBMS in a good corollary there.
Now, maybe there's enough overlap between the data access and the data schema that we should let them couple - e.g. would you want to be able to run .sel on any coord, even 2D? While it's possible in concept, it could guide users to inefficient operations.
We probably don't need to answer this question to proceed, but I'd be interested whether others see indexes as a property of the schema / I'm missing something.

ds.sel(multi=list_of_pairs) can probably be replaced by ds.sel(x=..., y=...), but how about reindex along MultiIndex?

Indeed I haven't really thought about reindex and alignment in my suggestion above.
How do you currently reindex along a multi-index dimension?
Contrary to .sel, ds.reindex(multi=list_of_pairs) doesn't seem to work (the list of n-length tuples being interpreted as a n-dim array). The only way I've found to make it work is to pass another pandas.MultiIndex. Wouldn't be it rather confusing if we choose to go with our own implementation of MultiIndex for xarray instead of pandas.MultiIndex?
Wouldn't be possible to easily support ds.reindex(x=..., y=...) within the new data model proposed here?

Am I right in thinking the Multi-indexes is only a helpful note to users, rather than conveying anything about how data is accessed?

This is a good question.
A related question: apart from ds.sel(multi=list_of_pairs) and ds.reindex(multi=list_of_pairs) use cases discussed so far, is there other reasons of having a variable for a multi-index?

I think we can do much of this before adding the ability to set custom indexes, which would be cool but further from where we are, I think.

I agree, although whether or not we will eventually support custom indexes might influence the design choices that we have to do now, IMO.

Looking at the reported issues related to multi-indexes in xarray, I have the same feeling. Simply reusing pandas.MultiIndex in xarray where slightly different semantics are generally expected has shown to be painful. It seems easier to have our own baked solution and deal with differences during xarray<-> pandas conversion if needed.

I think the pandas.MultiIndex is a pretty solid data structure on a fundamental level, it just has some weird semantics for some indexing edge cases. Whether or not we write xarray.MultiIndex structure, we can achieve most of what we want with a thin layer over pandas.MultiIndex.

If a variable for each multi-coordinate index is "just" for data schema consistency, then why not showing all those indexes in a separate section of the repr?

Yes, I like this! Generally I like @benbovy's entire proposal :).
@fujiisoup can you clarity the use-cases you have for a MultiIndex as a variable?

Am I right in thinking the Multi-indexes is only a helpful note to users, rather than conveying anything about how data is accessed?

From a data perspective, the only thing having an Index and/or MultiIndex should change is that the data is immutable.
But by necessity the nature of the index will determine which indexing operations are possible/efficient. For example, if you want to do nearest-neighbor indexing with multiple coordinates you'll need a KDTree. We should not be afraid to raise errors if an indexing operation can't be done efficiently.

With regards to reindexing: I don't think this needs any special handling versus normal indexing (sel()). The rules basically fall out of those for normal indexing, except we handle missing values differently (by filling with NaN).
Another issue: how do automatic alignment with multiple indexes? Let me suggest a straw-man proposal: We always align indexed coordinates. If a coordinate is used in different types of indexes (e.g., a base Index in one argument and a MultiIndex level in another), we can either:

create a MultiIndex with the variable on the fly (this could be slightly expensive), or
fall back to only supporting "exact" indexing

It occurs to me that for the case of "multiple single indexes" along the same dimension there is no good way to use them simultaneously for indexing/reindexing at the same time. We should explicitly raise if you try to do this.
I guess we have a few options for automatic alignment with multiple single indexes, too:

We could only support "exact" indexing
We could require that aligning each index separately gives the same result

(2) seems least restrictive and is probably the right choice.

One advantage of not having MultiIndex objects as variables is that the serialization story gets simpler. The rule becomes "multi-indexes don't get saved".

What should the default behavior of set_index(['x', 'y']) without an explicit kind argument be?

Should this mean individual indexes or a combined MultiIndex? The later might be more surprising but is arguably more useful. It would make sense if the model is that set_index() always creates a single index object.
We could potentially automatically pick an index type using simple heuristics. For example, if the arguments are 1D, you get get a MultiIndex by default. If the arguments have two or more dimensions, you get a KDTree.

I wonder if we should also change the default value of the append argument in set_index() to append=None, which means something like "append if creating a MultiIndex". For most users, keeping a single MultiIndex is the most usable way to use multiple indexes along a dimension, and our default behavior should reflect that.

A couple of thoughts:
If nothing useful can be done in the case of "multiple single indexes", would it make sense to discourage users explicitly creating multiple single indexes along a dimension? "Multiple single indexes" would be just a default situation when nothing specific as been defined yet or resulting from a failback.
For example, why not requiring that set_index(['x', 'y']) (with a list as argument) should always result in a multi-index regardless of the kind argument, i.e., raise if a single index is given? This is close to the current behavior, I think. This would require calling set_index for each single index that we want to (re)define, but I don't think setting a lot of single indexes at the same time is something that often happens.
Hence, would it be possible to avoid append=None and instead change the default to append=True?

How should dimension names interact with index names - i.e. the "Mapping indexes into pandas" in @shoyer 's comment
I'd suggest that option (3) should be invalid, and that da[dim_name] should return all the indexes on that dimension

It occurs to me that for the case of "multiple single indexes" along the same dimension there is no good way to use them simultaneously for indexing/reindexing at the same time.

Sorry for maybe asking this again but I'm a bit confused now: is there any good reason of supporting "multiple single indexes" along the same dimension?
After all, perhaps better defaults would be to set indexes (pandas.Index) only for 1-d coordinates matching dimension names, like it is the case now.
If you want a different behavior, then you need to use .set_index(), which would raise if it results in multiple single indexes along a dimension. We could also add a new indexes argument to the Dataset / DataArray constructors to save some typing (and avoid the creation of in-memory pandas.Index for very long coordinates if an out-of-core alternative is later supported).

da[dim_name] should return all the indexes on that dimension

I think that one big source of confusion has been so far mixing coordinates/variables and indexes. These are really two separate concepts, and the indexes refactoring should address that IMHO.
For example, I think that da[some_name] should never return indexes but only coordinates (and/or data variables for Dataset). That would be much simpler.
Take for example
>>> da = xr.DataArray(np.random.rand(2, 2),
...                   dims=('one', 'two'),
...                   coords={'one_labels': ('one', ['a', 'b'])})
>>> da
<xarray.DataArray (one: 2, two: 2)>
array([[ 0.536028,  0.291895],
       [ 0.682108,  0.926003]])
Coordinates:
    one_labels  (one) <U1 'a' 'b'
Dimensions without coordinates: one, two

I find it so weird being able to do this:
>>> da['one']
<xarray.DataArray 'one' (one: 2)>
array([0, 1])
Coordinates:
    one_labels  (one) <U1 'a' 'b'
Dimensions without coordinates: one

Where does come from array([0, 1])? I wouldn't have been surprised if a KeyError was raised instead. Perhaps this specific case was initially for backward compatibility when the "dimensions without indexes" feature has been introduced, but it was a long time ago and I'm not sure this is still necessary.
I might be a good thing explicitly requiring da.set_index('one_labels') to enable indexing/alignment along dimension one in the example above.

I think that one big source of confusion has been so far mixing
coordinates/variables and indexes. These are really two separate concepts,
and the indexes refactoring should address that IMHO.
For example, I think that da[some_name] should never return indexes but
only coordinates (and/or data variables for Dataset). That would be much
simpler.

Can't claim to be following every detail here, but this sounds very
sensible to me FWIW.

I guess the error is probably the best idea.

Agreed. It seems very strict indeed, but it will be easier to relax this later than the other way. There is also a (very rare?) case where the two indexed coordinates have the same labels but are named differently in the two datasets (e.g., station_name and sname). In that case an error is probably better too. It would be a sort of indication that the most useful thing to do for future operations is to rename one of those coordinates first.

I'm starting to make these changes incrementally -- the first step is in https://github.com/pydata/xarray/pull/2639.

There's now a good few dozen issues that reference this PR.
Wondering if there's any particular help needed (in the form of coding, discussion, or any other fashion), so as to try and speed it up and unblock those issues?
(I'm personally interested in resolving problems like #934 myself - allowing selection on non-dim coords, which seems to be a major hassle for a lot of use cases.)

After spending a few hours on the issue tracker yesterday, it became clear to me that the issue--more flexible indexes--is a major blocker on many high-priority features going forward.
In #2639, @shoyer started to address this. In that now merged-PR, he outlined the following steps, each of which needs its own PR:

[x] In #2639 Indexes are recreated from coordinates every time a new DataArray or Dataset is created.
[ ] Refactor indexes to be propagated explicitly in xarray operations. This will facilitate future API changes, when indexes will no longer only be associated with dimensions. I will probably add some testing decorator that can be used to mark part of a test as including no creation of default indexes.
[ ] Add explicit entries into indexes for MultiIndex levels that are checked instead of MultiIndex variables. Still no public API changes (aside from adding more entries to .indexes).
[ ] Support arbitrary coordinates in indexes.

So the best way to make progress on all manner of higher-level xarray feature requests is to start working through the next three items in this list.

Explicitly propagating indexes requires going through most of xarray's source code and auditing each time we create a Dataset or DataArray object with low-level operations. We have some pretty decent testing functions for this in the form of xarray.testing._assert_internal_invariants, so this is now a pretty mechanical process -- you know it's working if you're now setting indexes explicitly and xarray's test suite passes.
Here's our current progress:

[x] most of dataset.py
[x] alignment.py
[ ] merge.py (#3234)
[ ] concat.py
[ ] dataarray.py
[ ] computation.py
[ ] groupby.py
[ ] resample.py
[ ] rolling.py
[ ] everything else!

@shoyer I was thinking of starting on one of the listed files. Do you have any tips? Are you working on any of those at present? What might be the easiest one to begin?

I'm not working on any of these right now. You might start with a few of the dataarray.py methods (no need to do them all at once) to get a sense of what piping these arguments around looks like. I suspect you could get quite a few of these working just by handling indexes in _to_temp_dataset/_from_temp_dataset.

I've noticed that basically all my current troubles with xarray lead to this issue (lack of MultiIndex support). I use xarray for machine learning/data science/econometrics. My current problem requires a semi-hierarchical indexing on one of the dimensions, and slicing/aggregation along some levels of those dimensions.
My first attempt was to just assume each dimension was orthogonal, which resulted in out-of-memory errors. I ended up using a MultiIndex for the hierarchy dimension to have a "dense" representation of a sparse subspace. Unfortunately, currently .sel() and such will cut out MultiIndex dimensions, and I've had to do boolean masking to keep all the dimensions I need.
Multidimensional groupby, especially within the MultiIndex, is a headache as it currently stands. I had to resort to making auxilliary dimensions with one-hot encoded levels (dummy variables) and doing multiply-aggregate operations by hand.
xarray is really beautiful and should be used more by data scientists, but it's really difficult to recommend it to colleagues when not all the familiar pandas-style operations are supported.

Thanks @NowanIlfideme for your feedback.
Could you perhaps share a gist of code related to your use case?

My first attempt was to just assume each dimension was orthogonal, which resulted in out-of-memory errors

We have experimental support for https://sparse.pydata.org/en/latest/index.html that may help but no documentation unfortunately. There are some details here: https://github.com/pydata/xarray/issues/3213 and https://github.com/pydata/xarray/issues/3484

Thanks @NowanIlfideme for your feedback.
Could you perhaps share a gist of code related to your use case?

The first example in this comment is similar to my use case: https://github.com/pydata/xarray/issues/3213#issuecomment-520741706 . There are several "core" dimensions, but some part of the coordinates may be hierarchical or cross-defined (e.g. country > province > city > building, but also country > province > voting district > building). We might have a full or nearly-full panel in the MultiIndex representation, but have a huge cross product (even if we keep strictly hierarchical dimensions out).
Meanwhile using a true COO sparse representation (as I understand it) will likely end up with slower operations overall, since nearly all machine learning models (think: linear regression) require a dense array input anyways.
I'll make an example of this when I find some free time, along with a contrasting one in Pandas. :)

I'll make an example of this when I find some free time, along with a contrasting one in Pandas. :)

👍

Many array types do have implicit indices.
For example, sparse arrays do have their coordinates / CSR representation as primary index (.sel()) while dense array's primary index is the position (.isel()).
Every labeled dimension is therefore just a separate mapping of a string to the index position in the array.
Going one step further, one could have continuous dimensions where positional indexing (.isel()) does not really make sense.
Looking at TileDB's dimensions provides an example for this.
=> Having explicit and implicit indices on arrays would be awesome!

Well, maybe we can consider the coordinates in a more generic way.
Coordinate is an array in data set cause co-indexed when we index its data set. It means that:

If A1,A2,A3 are in a same data set S, we index S[ {'A1':I} ] will return a new data set which not only have indexed A1, but they also been Indexed that the A2 A3 which have dims shared with A1. This behavior I call it co-index.
Use dims to determined the way how other array of the data set will be co-indexed.
If all dims of A1(as coordinate) are also in A2(as regular array co-indexed), obviously the behavior can simply follow the old behavior, just change at the same dim and contain others.
If A1 has a dim which not in A2, we should broadcast A2 at the dim, because the older behavior is to consider None dim as broadcast-able dim during other operation so co-index should follow it.
Some compatibility issues:
maybe need a New Type like DataArray but only have dims instead of both dims and coordinate
just define how Dataset to deal with index, maybe DataArray is simlar.

Hi @weipeng1999,
I'm not sure to fully understand your suggestion, would you mind sharing some illustrative examples?
It is useful to have two distinct coordinate variable vs data variable concepts. Although both are data arrays, the former is used to locate data in the dimensional space(s) defined by all dimensions in the dataset while the latter is used to store field data.
It also helps to have a clear separation between the coordinate variable and index concepts. An index is a specific data structure or object that allows efficient data extraction or alignment based one or more coordinate labels. Sometimes an index object may be handled like a data array (like pandas indexes) but this is not always the case (e.g., a KD-Tree).
Currently in Xarray the index concept is hidden behind "dimension" coordinate variables. The goal of the explicit index refactor is to bring it to the light and make it available to any coordinate (and also open it to custom index structures, not only pandas indexes).
It looks like what you suggest is some kind of implicit (co-)indexes hidden behind any dataset variable(s)?  We actually took the opposite direction, trying to make everything explicit.

Should we close this issue and continue the discussion in #6293?
For anyone who wants to track the progress on this topic: https://github.com/pydata/xarray/projects/1

authors: Hoeze NowanIlfideme aldanor alimanfoo benbovy dcherian fujiisoup jjpr-mit max-sixty maxim-lian rabernat shoyer weipeng1999

repo: pydata/xarray


text:
Integrate has undefined name 'dim'

https://github.com/pydata/xarray/blob/master/xarray/core/dataset.py#L4085
Should that be called coord or dim? Currently there's a variable that's undefined:
            raise ValueError('Coordinate {} does not exist.'.format(dim))

I would have made a quick fix but not sure the correct name

I think it should be coord. Also we should add a unit test :)

Closed by #3010

source: gharchive/issue

authors: max-sixty shoyer

license_type: permissive

license_source: github-api


text:
Revert addition of fastpath causing incompatibilities with cftime

See discussion in #9138
Fixes #9138
This commit and pull request mostly serves as a staging group for a potential fix.
Test with:
pytest xarray/tests/test_cftimeindex.py::test_cftime_noleap_with_str

[x] Closes #xxxx
[x] Tests added
[x] User visible changes (including notable bug fixes) are documented in whats-new.rst
[x] New functions/methods are listed in api.rst

The comment https://github.com/pydata/xarray/pull/9002/files#r1591214299 had the right intuition. The fastpath was the source of problems.
Empirically, i tested to see if it had any effects on the internal benchmarks that brought me to introducing it and it seems it does not (we avoid pandas quite heavily due to slowdowns it introduces, our data access is quite regular).

ps. i have visitors this week, so feel free to push to adjust anything you feel the need to change.

asv continuous main calendar_noleap --bench indexing

| Change   | Before [42ed6d30] <main>   | After [822c0b8e] <calendar_noleap>   |   Ratio | Benchmark (Parameter)                                      |
|----------|----------------------------|--------------------------------------|---------|------------------------------------------------------------|
| +        | 337±1μs                    | 618±1μs                              |    1.83 | indexing.AssignmentOptimized.time_assign_identical_indexes |
| +        | 67.4±0.6μs                 | 77.7±0.5μs                           |    1.15 | indexing.HugeAxisSmallSliceIndexing.time_indexing          |
| +        | 139±0.7μs                  | 159±0.5μs                            |    1.14 | indexing.Assignment.time_assignment_basic('1slice')        |

@pydata/xarray who is the best person to review this?

I believe the solution in https://github.com/pydata/xarray/issues/9138#issuecomment-2198145554 is a better path

Thanks for starting this @hmaarrfk—I went ahead and switched this to the approach suggested in https://github.com/pydata/xarray/issues/9138#issuecomment-2198145554 and moved the test to test_calendar_ops.py.
@aulemahal it would be great if you could take a look at this to ensure the test coverage is adequate.  Thanks!

Thank you @spencerkclark glad this is moving forward!

Just reporting:
BENCHMARKS NOT SIGNIFICANTLY CHANGED.

@spencerkclark This change and the tests look good to me!

Thanks all!

source: gharchive/pull-request

authors: aulemahal dcherian hmaarrfk max-sixty spencerkclark


text:
Still see system /usr/bin/python after activation

I am learning pyenv.
After reading existing open/closed issue. I finally got pyenv installed on RockLinux 8.5.
But when I try to activate 3.9.8 for log4j project, the which python still return /usr/bin/python.

Please advise where I did wrong. from following log

[me@rocky8t01 ~]$ which python
/usr/bin/python
[me@rocky8t01 ~]$ pyenv --version
pyenv 2.2.3
[me@rocky8t01 ~]$ pyenv activate log4j
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(log4j) [me@rocky8t01 ~]$ which python
/usr/bin/python
(log4j) [me@rocky8t01 ~]$ pyenv versions
  system
  3.9.8
  3.9.8/envs/log4j
* log4j (set by PYENV_VERSION environment variable)
(log4j) [me@rocky8t01 ~]$ /usr/bin/python -V
Python 3.6.8
(log4j) [me@rocky8t01 ~]$ echo $PATH
/home/me/.pyenv/bin:/home/me/.cargo/bin:/home/me/.local/bin:/home/me/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
(log4j) [me@rocky8t01 ~]$

Not sure why activate didn't insert "/home/me/.pyenv/versions/3.9.8/bin/" path into $PATH variable ?

(log4j) [me@rocky8t01 ~]$ export PATH=/home/me/.pyenv/versions/3.9.8/bin:$PATH
(log4j) [me@rocky8t01 ~]$ which python
~/.pyenv/versions/3.9.8/bin/python
(log4j) [me@rocky8t01 ~]$

I have the same issue, but on MacOS 12.0.1.
Solved with adding /.pyenv/shims to PATH (not only /.pyenv/bin). This doc about shims helped me.
export PATH="$HOME/.pyenv/shims:$HOME/.pyenv/bin:$PATH"

I have the same issue, but on MacOS 12.0.1.
Solved with adding /.pyenv/shims to PATH (not only /.pyenv/bin). This doc about shims helped me.
export PATH="$HOME/.pyenv/shims:$HOME/.pyenv/bin:$PATH"

Thanks! That solved my issue on Arch Linux!

Thanks - I had the same issue on Ubuntu 20.04 and the solution above worked.

repo: pyenv/pyenv-installer

license: MIT


text:
Allow algebras with only 1 or even 0 basis vectors

Support for just one basis is needed for dual numbers, cc @tBuLi.

CI failure is https://github.com/computationalmodelling/nbval/issues/139#issuecomment-665383149

Thanks for the quick PR!

CI failure looks like an accidental upstream sympy change of my own making...

Why are there much more failure in this PR for sympy master?

Because it needs rebasing on top of the other PR you just merged

@tBuLi: If you want to use this before we next release (which could be quite a while), you can pip install git+https://github.com/pygae/galgebra.git

authors: eric-wieser tBuLi utensil

repo: pygae/galgebra


text:
Represent Spaces In Generated HTML Using   Rather Than ' ' Characters

(Original issue 1319 created by bhreinb on 2017-01-25T11:23:33.626660+00:00)
Hi There,
I have a question about the following project. I use the tool set (pygments) through the following portal
http://hilite.me/
which essentially uses this project to generate the HTML which can be rendered to a HTML page etc. My use case would be to convert Gherkin source code to HTML. I invoke the button using style 'emacs'.
I render the HTML on a confluence page. Generally this works OK in most cases but the formatting sometimes gets lost when ' ' (space) character is used especially in the case of gherkin tables. To solve this the formatting is kept when space is represented as ' '. Is their a way to pass this as an input parameter or is this a feature that could be requested?
Many thanks in advance as to any help anyone provides on this

(Original issue was assigned to alex_gaynor)

repo: pygments/pygments-migration-test


text:
[patch] Add space for extra information

@JNmpi, per our discussion today, I added a private and very generic place for nodes to store extra information that might be useful to users of the node class: the class method Node._extra_info().
Function and macros expose the source code for their node_function and graph_creator, respectively. Dataclass nodes defined in __main__ are a little prickly with inspect.getsource for reasons I don't understand. Instead of having a lengthy fight with it for this private trial feature, I just directly leveraged the underlying dataclass's __doc__, which at least has the standard signature info.
Since this is all private methods for now, we can iterate from here freely in whatever direction is most useful for the GUI without worrying about backwards compatibility.

Pull Request Test Coverage Report for Build 10694175521
Details

0 of 0   changed or added relevant lines in 0 files are covered.
23 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-0.09%) to 91.248%

Files with Coverage Reduction
New Missed Lines
%

nodes/function.py
1
98.39%

nodes/transform.py
2
98.8%

nodes/macro.py
7
92.14%

node.py
13
90.68%

Totals

Change from base Build 10693688049:
-0.09%

Covered Lines:
2867

Relevant Lines:
3142

💛  - Coveralls

authors: coveralls liamhuber


text:
Nodes for the atomistics package

Example:
from pyiron_workflow import Workflow
from pyiron_workflow.atomistics_library.calculatornodes import get_emt, calc_with_calculator
from pyiron_workflow.atomistics_library.tasknodes import get_evcurve_task_generator, analyse_structures, generate_structures, get_bulk

wf = Workflow("evcurve")
wf.get_structure = get_bulk(element="Al")
wf.get_task_generator = get_evcurve_task_generator(structure=wf.get_structure)
wf.generate_structures = generate_structures(instance=wf.get_task_generator)
wf.get_calculator = get_emt()
wf.calc_with_calculator = calc_with_calculator(task_dict=wf.generate_structures, calculator=wf.get_calculator)
wf.fit = analyse_structures(instance=wf.get_task_generator, output_dict=wf.calc_with_calculator)

wf.draw()

wf.run()

As the workflows in the atomistics package all follow the same pattern of (0) initializing the generator, (1) generating atomic structures, (2) evaluate the atomic structures with a simulation code and (3) analyse the results, the integration was rather straight forward.
I recently updated the atomistics package and its documentation https://atomistics.readthedocs.io/en/latest/ so this might be helpful to better understand the classes that I wrapped here.

Pull Request Test Coverage Report for Build 6883431668

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-1.4%) to 86.667%

Totals

Change from base Build 6836834588:
-1.4%

Covered Lines:
3328

Relevant Lines:
3840

💛  - Coveralls

Please move atomistics_library to node_library/atomistics.

The issue with moving is that the module is named atomistics and the package I try to link is named atomistics as well, which leads to conflicts. In addition, I was wondering if there are any plans to store specific nodes separately, for example in this case the calculators which evaluate energy and forces for a given atomistic structure and the tasks which define what to do.

Then "nodesincalculatornodesandtasknodescan all dropped, since these live innode_library/atomistics/*anyhow. (In the future you'll be able to register sub-domains with the creator class so that you can do something likeWorkflow.register("atomistics", "pyiron_workflow.node_library.atomistics"); Workflow.create.atomistics.task.Foo`, but in the meantime at least it keeps all the specific nodes isolated from the raw infrastructure in a special folder)

I guess this is where I am confused as there is already an atomistics module in the node_library.

And atomistics (+version pin) can be added to the extras_require["node_library" ] in setup.py. (I'll add it to the notebooks env in a later PR as I work these into the educational material)

Does this mean we have one node library for everything? I thought it would be nice to have a node library for the atomistics package as in future this might become a standalone node library.

Nit: I would actually favour dropping output_labels="..." in the decorator and just pass the label without any keyword. IMO it's a little cleaner, and would make it more compatible with my intended change in the signature of the wrapper from wrapper(output_labels: Optional[list[str] | str) to wrapper(*output_labels: str).

Sure, that should be quick.
Some more questions from my side:

I would like to have the calculator_nodes be inputs for macros, but I was not exactly sure how I specify the node to be an input as it needs to connected to the evaluation node.
In addition, I would like to provide a set of macros one for each workflow like calculated_energy_volume_curve(), calculate_elastic_matrix() and so on. Would you store the macros in the same node library, or should the macros be moved to a different file, just to make it more clear for the users that by default I do not expect them to use individual nodes but rather recommend using the macros directly.

I added a series of macro nodes:
wf = Workflow("evcurve")
wf.get_structure = get_bulk()
wf.get_calculator = get_emt()
wf.evcurve = evcurve(structure=wf.get_structure, calculator=wf.get_calculator)

wf.draw()
wf.run()

The issue with moving is that the module is named atomistics and the package I try to link is named atomistics as well, which leads to conflicts.

Ahh, you are 110% right. Sorry. TBH, I care so little for the nodes currently in atomistics that I forgot what I called them...
Ok, so just forget moving them then. I still want to do it, and I think what is currently atomistics should be pyiron_atomistics so that this can be atomistics, but that will screw up the demo notebooks. I'll be responsible for making a PR later that moves this where I want it.
(Assuming, of course, that we deign to even keep the nodes currently in atomistics -- I rather suspect that between these node's and Joerg's nodes (#33), they are obsolete.)

In addition, I was wondering if there are any plans to store specific nodes separately, for example in this case the calculators which evaluate energy and forces for a given atomistic structure and the tasks which define what to do.

I'm not sure I understand the contrast in your example, but there is definitely a plan to boot all the actual nodes out of this repo and into one or more node package repos -- except perhaps for the standard package that relies only on the standard library and provides basic things like an If node.

I guess this is where I am confused as there is already an atomistics module in the node_library.

I guess this is resolved since the answer is just I was being dumb.

And atomistics (+version pin) can be added to the extras_require["node_library" ] in setup.py. (I'll add it to the notebooks env in a later PR as I work these into the educational material)

Does this mean we have one node library for everything? I thought it would be nice to have a node library for the atomistics package as in future this might become a standalone node library.

I 100% agree. In the medium-term (January? February?) I want to boot out (almost (see standard comment)) everything that is in node_library into one or more repos. IMO all the nodes you add here should be grouped together, but, as you say, it may be wise to have them separate from other nodes.
One solution I can see is to have atomistics/nodes/*.py right in the atomistics package, and put pyiron_workflow as an optional dependency there. Another is to have atomistics_nodes, or pyiron_nodes/atomistics where pyiron_nodes has a variety of independent sub-modules. Honestly, I'm open to a number of things, but I want to keep the stuff here as bare-bones as possible in the long run, and that means kicking out domain-specific (or other-package-dependent) node definitions.
For the time being, I think it's ok to just keep dumping specific nodes into this repo until we figure out what we want.

Sure, that should be quick.

👍

Some more questions from my side:

I would like to have the calculator_nodes be inputs for macros, but I was not exactly sure how I specify the node to be an input as it needs to connected to the evaluation node.

I'm not sure I follow you here. In general, node instances are not valid IO to other instantiated nodes -- and I feel rather strongly about keeping it that way, as functional and idempotent behaviour goes straight to hell when you travel down that path.
However, there are times when you do need variable node class as part of a macro definition. I've been calling these "meta"-nodes (cf. pyiron_workflow.meta), and the idea is that you dynamically define a new node class that takes other node classes as input, but when you actually instantiate it, it is building a particular macro with a particular class involved. #72 is good reading for current thoughts on this topic, particularly some syntactic sugar for skipping the middle class definition.
I both hope and expect you have the tools already available to accomplish what you want, but I humbly recognize that the syntax is still in desparate need of some attention.

In addition, I would like to provide a set of macros one for each workflow like calculated_energy_volume_curve(), calculate_elastic_matrix() and so on. Would you store the macros in the same node library, or should the macros be moved to a different file, just to make it more clear for the users that by default I do not expect them to use individual nodes but rather recommend using the macros directly.

For now I would put them in a separate file, but as long as these macros only depend on the standard package and other nodes defined here, I would definitely expect these to all live in the same repo long-term. This is a good example of the benefit of sub-domains on the creator -- the first good one we've had so far I think. I.e. wf.create.atomistics.calculator/task/macro exactly as you're laying out the filenames now. (Or science or material_properties instead of macro or whatever, but you get the idea and I'm fine with macro for now if you are)

Nice, LGTM. Good to see that it was so straightforward to wrap these, as that's absolutely the intent.

Wrapping functions as single value nodes was definitely easy, but I feel the creation of macros is still rather complex. As an example:
@macro_node()
def get_elastic_matrix(wf: Macro) -> None:
    wf.get_task_generator = get_elastic_matrix_task_generator()
    wf.internal = internal_macro(instance=wf.get_task_generator)
    wf.inputs_map = {
        "get_task_generator__structure": "structure",
        "get_task_generator__num_of_point": "num_of_point",
        "get_task_generator__eps_range": "eps_range",
        "get_task_generator__sqrt_eta": "sqrt_eta",
        "get_task_generator__fit_order": "fit_order",
        "internal__calculator": "calculator",
    }
    wf.outputs_map = {"internal__fit_dict": "fit_dict"}

In practice I would rather do:
@single_value_node("fit_dict")
def get_elastic_matrix(structure, calculator, num_of_point=5, eps_range=0.05, sqrt_eta=True, fit_order=2):
    return internal_macro(instance=get_elastic_matrix_task_generator(
        structure=structure,
        calculator=calculator,
        num_of_point=num_of_point,
        eps_range=eps_range,
        sqrt_eta=sqrt_eta,
        fit_order=fit_order
    ))

The obvious limitation of the second variant is that I loose the ability to modify the internals of a function, but it is much easier to write. So as a consequence it would be interesting to modify the syntax of the macro definition in a way that it becomes more similar to the definition of a single_value_node.

Wrapping functions as single value nodes was definitely easy, but I feel the creation of macros is still rather complex. As an example:

I want to be careful to draw a distinction between "complex" and "verbose" here... do you really feel that this is actually complex? To me it's just a two-node workflow, which is pretty straightforward.

In practice I would rather do:
...
The obvious limitation of the second variant is that I loose the ability to modify the internals of a function, but it is much easier to write. So as a consequence it would be interesting to modify the syntax of the macro definition in a way that it becomes more similar to the definition of a single_value_node.

Yes, this is indeed the limitation. Not only do you lose the ability to modify them, you also lose the ability to query them post-facto, e.g. to see intermediate output.
Function nodes and Composite nodes are really, fundamentally different in my view. The advantages of a Composite structure are that they can be flexible (exposing/hiding new IO, replacing nodes to change behaviour) and transparent (intermediate values are available for investigation, flow can be easily drawn as a graph, etc). The disadvantage is that they are slower because they come with the overhead of everything we do in pyiron_workflow to get stuff running as a node.
There is no limit to the internal complexity of Function nodes and, in principle, any macro that consists only of Function nodes can be trivially re-written as a Function node -- in your case where you start with the imported functions this is obvious, but Function.node_function is actually a class level attribute, so these functions are available to glue together even for people who only have the nodes. And while it might be a bit of work for nested macros, because you'd need to go through and "functionalize" each layer of macros, this can thus be done for any macro. This is intentional design, so that if you have some workflow that you absolutely love and know is working perfectly, you're free to construct super-optimized and super-rigid versions of it.
But, ultimately, I don't want to make macro definitions look like function node definitions -- I want them to look like workflow definitions, and indeed intend that most of them are just copy-pastes from people's workflows in a notebook. I do agree that the definitions of IO label maps is un-sexy, and am open to ideas for more sugary syntax there, but I still want all the macro.foo = SomeNode(); ... lines to look just like that.
I'll make a stacked PR demonstrating my currently preferred syntax for avoiding code duplication here.

There's the macro PR.
The only other thing I really see missing is a node for converting the returned dictionary into independent channels. Honestly though, that's a shortcoming of pyiron_workflow and not the fault of atomistics. I won't get it done today, but I'll try to think this week of how to cleanly implement such a dict-to-many-channels node and add it to the standard package. No need for that to hold up merging here, it just will make it more useful in the future.

authors: coveralls jan-janssen liamhuber


text:
Support of latgen-faster-mapped

Are there any plan for PyKaldi to support latgen-faster-mapped function? It is under bin package.

I guess you are asking if we are planning to add a Python script to PyKaldi that behaves similar to latgen-faster-mapped executable in Kaldi. We don't intend to replicate the behavior of existing Kaldi executables since we don't see the benefit apart from showing how it can be done using PyKaldi. I mean why not use the existing latgen-faster-mapped binary if that is the desired functionality? If you don't mind me asking, what is the use case you have in mind that is not answered by the existing Kaldi binary?
There is an example here which shows how gmm-decode-faster can be implemented using PyKaldi. It should be fairly straightforward to implement something similar for latgen-faster-mapped. I believe all of the wrappers necessary for implementing such a script already exist in PyKaldi.

I hope to fully understand how latgen-faster-mapped works. I wonder if there is any resource to assist me to understand the source code. So the purpose of PyKaldi is to use Kaldi in Python. Am I correct?

I hope to fully understand how latgen-faster-mapped works. I wonder if there is any resource to assist me to understand the source code.

I don't know of any resource other than kaldi documentation and source code that might help you understand how latgen-faster-mapped works. I am not sure why you need/want to fully understand how it works though. My guess is you want to use this decoder by providing precomputed likelihoods and you probably don't care how Kaldi does decoding. If this is the intended goal, you need to construct a DecodableMatrixScaledMapped object from the matrix of likelihoods (separate for each utterance) and call the decode method of the decoder with that decodable object as the argument. Then you can use the get_best_path method of the decoder to retrieve the highest scoring hypothesis.

So the purpose of PyKaldi is to use Kaldi in Python. Am I correct?

Yes, you are correct. The purpose is to expose Kaldi C++ library API to Python code. You can call Kaldi functions and construct Kaldi objects in Python. That being said, PyKaldi does not provide scripts that are analogues of Kaldi executables. PyKaldi provides the wrappers for the functions/classes needed for implementing such scripts but not the scripts themselves.

I understand. Thanks for your clarification!

repo: pykaldi/pykaldi


text:
GridPlan order issue

from #149
Small problem I see is for this sequence:
>>> seq = MDASequence(axis_order="gtc",time_plan={"interval": 1, "loops": 2},channels=[{"config": "DAPI", "exposure": 1}],grid_plan={"rows": 2, "columns": 2})
>>> for event in seq:
...    print(event)
...
index=mappingproxy({'t': 0, 'g': 0, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=0.0 x_pos=-0.5 y_pos=0.5
index=mappingproxy({'t': 1, 'g': 0, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=1.0 x_pos=-0.5 y_pos=0.5
index=mappingproxy({'t': 0, 'g': 1, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=0.0 x_pos=0.5 y_pos=0.5
index=mappingproxy({'t': 1, 'g': 1, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=1.0 x_pos=0.5 y_pos=0.5
index=mappingproxy({'t': 0, 'g': 2, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=0.0 x_pos=0.5 y_pos=-0.5
index=mappingproxy({'t': 1, 'g': 2, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=1.0 x_pos=0.5 y_pos=-0.5
index=mappingproxy({'t': 0, 'g': 3, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=0.0 x_pos=-0.5 y_pos=-0.5
index=mappingproxy({'t': 1, 'g': 3, 'c': 0}) channel=Channel(config='DAPI') exposure=1.0 min_start_time=1.0 x_pos=-0.5 y_pos=-0.5

Might no be a very typical setup with axis order "gtc, but note how the min_start_time resets for every g index. This leads to the first time series at position 0 respecting the interval, while all later positions are acquired at max speed.

@tlambert03 Just to confirm... I think this should be fixed as well with #188

Thanks!

authors: fdrgsp tlambert03 wl-stepp

repo: pymmcore-plus/useq-schema


text:
Hard code bun version

Currently we always download the latest bun, which could lead to issues. We should have a fixed version.

in case of manually installing bun use the following command:
RUN curl -fsSL https://bun.sh/install | bash -s -- bun-v0.5.5

Incompatible with windows, won't be able to use this command for windows in the actual repo

Incompatible with windows, won't be able to use this command for windows in the actual repo

We use Node.js instead of bun on Windows.

Incompatible with windows, won't be able to use this command for windows in the actual repo

We use Node.js instead of bun on Windows.

Makes sense, disregard my comment then :)

authors: Cheesy-Brik FHU-yezi PrerithSubramanya picklelo

repo: pynecone-io/pynecone


text:
Shouldn't have to pass self when chaining event handlers

Describe the bug
When chaining EventHandlers, pynecone-0.1.27 requires self to be passed explicitly, even though the method appears to be bound to self already. This is unusual and a behavioral difference from 0.1.24
To Reproduce
import pynecone as pc

class State(pc.State):
    val: str = "unset"

    def c1a(self):
        return self.c2

    def c1b(self):
        return self.c2("foo")

    def c1b_workaround(self):
        return self.c2(self, "foo")

    def c2(self, val="bar"):
        self.val = val

def index() -> pc.Component:
    return pc.center(
        pc.vstack(
            pc.heading(State.val, font_size="2em"),
            pc.hstack(
                pc.button("c1a: 'bar'", on_click=State.c1a),
                pc.button("c1b: 'foo'", on_click=State.c1b),
                pc.button("c1b-workaround", on_click=State.c1b_workaround),
            ),
        ),
        padding_top="10%",
    )

# Add state and page to the app.
app = pc.App(state=State)
app.add_page(index)
app.compile()

Expected behavior
Expect c1b handler to ultimately set the val to "foo". Instead it raises an exception on the backend.
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/pynecone/state.py", line 655, in _process_event
    events = fn(**payload)
             ^^^^^^^^^^^^^
  File "/Users/masen/code/pynecone-io/repro-event-chaining-self/repro_event_chaining_self/repro_event_chaining_self.py", line 11, in c1b
    return self.c2("foo")
           ^^^^^^^^^^^^^^
  File "/Users/masen/code/pynecone-io/repro-event-chaining-self/repro_event_chaining_self/repro_event_chaining_self.py", line 17, in c2
    self.val = val
    ^^^^^^^^
AttributeError: 'str' object has no attribute 'val'

** Specifics (please complete the following information):**

Python Version: 3.11.2
Pynecone Version: 0.1.27
OS: macOS 13.2.1
Browser (Optional):

Additional context
Haven't done any further investigation, but I did make a comment about this over here
Issue doesn't repro on main, instead we hit https://github.com/pynecone-io/pcweb/issues/62 on c1a and c2b works fine (c2b_workaround blows up with a TypeError: State.c2() takes from 1 to 2 positional arguments but 3 were given)

Dupe of #896


text:
DistutilsPlatformError in mingw built Python 2 (msys2)

Trying to do many things using setuptools in a mingw built Python 2 (For example msys2's mingw-w64-x86_64-python2 fails with this exception:
$ python setup.py build_ext
Traceback (most recent call last):
  File "setup.py", line 1, in <module>
    from setuptools import setup, Extension
  File "mingw64/lib/python2.7/site-packages/setuptools/__init__.py", line 160, in <module>
    monkey.patch_all()
  File "mingw64/lib/python2.7/site-packages/setuptools/monkey.py", line 104, in patch_all
    patch_for_msvc_specialized_compiler()
  File "mingw64/lib/python2.7/site-packages/setuptools/monkey.py", line 156, in patch_for_msvc_specialized_compiler
    msvc = import_module('setuptools.msvc')
  File "mingw64/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "mingw64/lib/python2.7/site-packages/setuptools/msvc.py", line 49, in <module>
    from distutils.msvc9compiler import Reg
  File "mingw64/lib/python2.7/distutils/msvc9compiler.py", line 306, in <module>
    raise DistutilsPlatformError("VC %0.1f is not supported by this module" % VERSION)
distutils.errors.DistutilsPlatformError: VC 6.0 is not supported by this module

distutils.msvc9compiler doesn't like being imported on mingw built Python, and this hack isn't necessary there.
P.S. I'm going to send a PR.

Hi @segevfiner, @jaraco:
I was comforted with similar issue using previous version of setuptools. Seeing it fixed in #1119, I have upgraded my installation of setuptools to the latest version 36.7.0. Unfortunately the issue persists.
krzysztof@Pkrzysztof MINGW32 ~/esp/esp-idf/docs
$ pip install sphinxcontrib.blockdiag
Collecting sphinxcontrib.blockdiag
  Using cached sphinxcontrib_blockdiag-1.5.5-py2.py3-none-any.whl
Requirement already satisfied: Sphinx>=0.6 in c:/msys32/mingw32/lib/python2.7/site-packages (from sphinxcontrib.blockdiag)
Collecting blockdiag>=1.5.0 (from sphinxcontrib.blockdiag)
  Using cached blockdiag-1.5.3-py2.py3-none-any.whl
Requirement already satisfied: alabaster<0.8,>=0.7 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: Jinja2>=2.3 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: snowballstemmer>=1.1 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: babel!=2.0,>=1.3 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: imagesize in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: setuptools in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: Pygments>=2.0 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: typing; python_version < "3.5" in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: colorama>=0.3.5; sys_platform == "win32" in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: docutils>=0.11 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: requests>=2.0.0 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: six>=1.5 in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: sphinxcontrib-websupport in c:/msys32/mingw32/lib/python2.7/site-packages (from Sphinx>=0.6->sphinxcontrib.blockdiag)
Requirement already satisfied: funcparserlib in c:/msys32/mingw32/lib/python2.7/site-packages (from blockdiag>=1.5.0->sphinxcontrib.blockdiag)
Requirement already satisfied: webcolors in c:/msys32/mingw32/lib/python2.7/site-packages (from blockdiag>=1.5.0->sphinxcontrib.blockdiag)
Collecting Pillow (from blockdiag>=1.5.0->sphinxcontrib.blockdiag)
  Using cached Pillow-4.3.0.tar.gz
Exception:
Traceback (most recent call last):
  File "C:/msys32/mingw32/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "C:/msys32/mingw32/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:/msys32/mingw32/lib/python2.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "C:/msys32/mingw32/lib/python2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "C:/msys32/mingw32/lib/python2.7/site-packages/pip/req/req_install.py", line 412, in run_egg_info
    self.setup_py, self.name,
  File "C:/msys32/mingw32/lib/python2.7/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "C:/msys32/mingw32/lib/python2.7/site-packages/setuptools/__init__.py", line 180, in <module>
    monkey.patch_all()
  File "C:/msys32/mingw32/lib/python2.7/site-packages/setuptools/monkey.py", line 104, in patch_all
    patch_for_msvc_specialized_compiler()
  File "C:/msys32/mingw32/lib/python2.7/site-packages/setuptools/monkey.py", line 182, in patch_for_msvc_specialized_compiler
    patch_func(*msvc9('find_vcvarsall'))
  File "C:/msys32/mingw32/lib/python2.7/site-packages/setuptools/monkey.py", line 169, in patch_params
    mod = import_module(mod_name)
  File "C:/msys32/mingw32/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "C:/msys32/mingw32/lib/python2.7/distutils/msvc9compiler.py", line 306, in <module>
    raise DistutilsPlatformError("VC %0.1f is not supported by this module" % VERSION)
DistutilsPlatformError: VC 6.0 is not supported by this module

Please give me some guidance ow to troubleshoot it.

@krzychb A quick workaround would be to allow the function to continue, even without the 'required' compiler.
In your case, in the file C:/msys32/mingw32/lib/python2.7/distutils/msvc9compiler.py remark line 306, and add the pass statement, so that the code looks more or less like the following:
+---[C:/msys32/mingw32/lib/python2.7/distutils/msvc9compiler.py:309]
|
| # More globals
| VERSION = get_build_version()
| if VERSION < 8.0:
|     pass
| #    raise DistutilsPlatformError("VC %0.1f is not supported by this module" % VERSION)
| # MACROS = MacroExpander(VERSION)
|
+---

This probably should do the trick..

Hi @PeterMosmans,
Thank you for looking into this issue!
In the meantime I have managed to bypass this issue by installing latest pillow with pacman - https://github.com/Alexpux/MINGW-packages/pull/3138, and then running pip install sphinxcontrib.blockdiag again.
This way there was no need to compile anything within the pillow source package.

Thank you @PeterMosmans , the trick works for me

Somehow, over a year later, this is still an issue.
In my case not related to setuptools though - I used python3-pip from mingw64 to install bcrypt.
Same error, same remedy. Thank you @PeterMosmans.

We've patched distutils in MSYS2 to work around this issue in the meantime: https://github.com/msys2/MINGW-packages/commit/8a162525a7d6f4a0ac2724db2e21c96eae1ba33f (importing msvc9compiler doesn't error out anymore now, only using the included code)

authors: JosefrBeltran PeterMosmans krzychb lazka mefistotelis segevfiner

repo: pypa/setuptools

license: mit

license_source: bigquery


text:
Add workaround for tox errors during env setup

Summary of changes
This is a workaround for #4031.

Add a cap for pyproject-api<1.6.0' in Github Actions.

Closes
Pull Request Checklist

[ ] Changes have tests
[ ] News fragment added in newsfragments/.
(See documentation for details)

This might no longer be necessary with tox==4.11.


text:
Limetime management for OIDC JWKS

The first step in verifying the OIDC JWTs is verifying their signatures, which means checking them against the OIDC provider's signing key set.
There are a few approaches we could take to acquiring and maintaining the updatedness of those keys:

Bake them into the Warehouse codebase. We could pull the current JWKS blob from the URI referenced in the provider's openid-configuration.

Pros: Simple.
Cons: More tedious to update in the event of key updates/rotations; requires a PR each time any provider does so. Gaps between the rotation and updates means that Warehouse would probably reject authentic JWTs.

Fetch the JWKS for each provider on Warehouse startup/initialization. Restarts always re-fetch the JWKS.

Pros: Simple.
Cons: Most of the same cons as (1).

All of (2), but we also include a periodic job that checks each OIDC provider's JWKS on a schedule.

Pros: No rotation downtime.
Cons: Requires the most code.

(3) is probably the right approach. At minimum, we're going to need:

A registry/configuration list of supported OpenID connectors (just GitHub, for the time being), tied to their openid-configuration URLs
A scheduled periodic task (similar to the ones already in Warehouse) for checking the openid-configuration + JWKS URL
A verification service for incoming JWTs

WIP branch is here: https://github.com/trailofbits/warehouse/tree/tob-jwk-management
Some outstanding design questions:

How should we store "live" JWKs? They're JSON blobs and they're not sensitive and they rotate, so it doesn't make sense to cram them into the DB. Perhaps Redis?

What should we do for our development environment? Does it make sense to have a DevelopmentJWKService that uses a local RSA keypair + JWK that we generate as part of make build, or should we just run the "full" JWKService?

How should we store "live" JWKs? They're JSON blobs and they're not sensitive and they rotate, so it doesn't make sense to cram them into the DB. Perhaps Redis?

I went ahead and did this on the branch -- we cache JWKs under the /warehouse/oidc/jwks/{provider} key for each provider.
You can smoke-test this with:
docker-compose run --rm web python -m warehouse oidc update-oidc-jwks
docker-compose run --rm web python -m warehouse oidc list-jwks --provider github

All of (2), but we also include a periodic job that checks each OIDC provider's JWKS on a schedule.

There's the potential for a brief window here where a new public key has been published to the JWKS but we haven't fetched it yet, and as a result signature verification will fail with legitimate signatures. Perhaps it be better to fetch the JWKS whenever we attempt to verify a JWT that has a kid (key ID) that we haven't encountered before?
This behavior works with cold-starts (the first verification with no JWK in cache causes us to reach out and get the entire JWKS) and this ensures we always have the latest keys before failing. To avoid these requests becoming abusive we could only repeat a request for a previously nonexistent kid outside of some window.

There's the potential for a brief window here where a new public key has been published to the JWKS but we haven't fetched it yet, and as a result signature verification will fail with legitimate signatures. Perhaps it be better to fetch the JWKS whenever we attempt to verify a JWT that has a kid (key ID) that we haven't encountered before?

That sounds good! And yeah, that provides nice self-starting behavior (much better than having to jumpstart things with a CLI command on each startup.)
We'll want to make sure that we preserve provider-first identification with these changes -- a previously unknown kid shouldn't enable a cache poisoning attack where an attacker tricks us into trusting a different provider for a matching kid.

To avoid these requests becoming abusive we could only repeat a request for a previously nonexistent kid outside of some window.

Sounds good. In that case we'll probably want an additional self-expiring KV in Redis for the cooldown window. A cooldown of even just 60s is probably more than sufficient.

authors: di woodruffw

repo: pypa/warehouse