text:
Flexible indexes refactoring notes

As a preliminary step before I take on the refactoring and implementation of flexible indexes in Xarray for the next few months, I reviewed the status of https://github.com/pydata/xarray/projects/1 and started compiling partially implemented or planned changes, thoughts, etc. into a single document that may serve as a basis for further discussion and implementation work.
It's still very much work in progress (I will update it regularly in the forthcoming days) and it is very open to discussion (we can use this PR for that)!
I'm not sure if Xarray's root folder is a good place for this document, though. We could move this into a new repository in xarray-contrib (that could also host other enhancement proposals) if that's necessary.
I'm looking forward to getting started on this and to getting your thoughts/feedback!

Thanks for your comments @keewis and @shoyer!
I think it's better for now to keep having this discussion and these notes in the Xarray repository, for more visibility. We could still move this elsewhere later if this PR becomes too cluttered, as there are potentially many aspects we can discuss about.

One thing I'm missing is duckarray support, though. Not sure if this is realistic, but I'm hoping to reduce the maintenance burden on duckarray support libraries (such as pint-xarray) as much as possible: subclassing every new index class (or having the index provider explicitly add duckarray support) seems a bit too much work.

I haven't looked much at pint-xarray yet, so I'm not sure to understand. Why would you need to subclass every new index class?
If you are referring to the issue that you describe in your comment https://github.com/pydata/xarray/issues/525#issuecomment-514805244, the refactoring should decouple indexes from the coordinates, leaving the latter "just" as if they were regular variables (thus with duckarray support). What is currently possible with non-index coordinates should be possible with all coordinates. Actually, I'm not sure that we'll need to keep IndexVariable after the refactoring.
Or maybe you're referring to unit-aware indexing (what @shoyer mentioned in https://github.com/pydata/xarray/issues/525#issuecomment-514880353)? In this case I'm not sure how we could do that without having specific index classes for that purpose. Maybe some pre/post indexing hooks in Xarray that could be used, e.g., to convert indexer units into the coordinate units?

I was referring to both, but I agree that the first point would be fixed by the decoupling. The second point is just as important, though. We're currently adding wrapper methods to pint-xarray which handle the unit-awareness, but I was hoping we could change the API so those would not be necessary. For the implementation: hooks would probably work. Other options might be decorator functions or context managers?

It's great to be able to follow along with the discussion here! I'm definitely interested in seeing where the duck array index support ends up.
One use-case motivated question: the flexible indexes refactoring has also been pointed to as the resolution to https://github.com/pydata/xarray/issues/2233, where multidimensional coordinates have the same name as one of their dimensions. I wasn't quite able to tell through the narrative here if that has been addressed along the way yet or not ("A. only 1D coordinates with a name matching their dimension name" for implicit index creation does seem to get close though). So, would it be worth directly addressing https://github.com/pydata/xarray/issues/2233 here, or should that wait?

For the implementation: hooks would probably work. Other options might be decorator functions or context managers?

Or similarly to _repr_inline_:
class MyDuckArray:
    ...

    def _sel_(self, indexer):
        """Prepare the label-based indexer to conform to this coordinate array."""
        ...
        return new_indexer

    ...

One use-case motivated question: the flexible indexes refactoring has also been pointed to as the resolution to #2233, where multidimensional coordinates have the same name as one of their dimensions. I wasn't quite able to tell through the narrative here if that has been addressed along the way yet or not ("A. only 1D coordinates with a name matching their dimension name" for implicit index creation does seem to get close though). So, would it be worth directly addressing #2233 here, or should that wait?

I think #2233 will be addressed by the index refactoring here. I don't see any issue with multidimensional coordinates having the same name as one of their dimensions once indexes are decoupled from dimensions/coordinates. I might still be missing something, though.

One thing I did not see discussed is alignment (or did I miss this?). Currently alignment is based on the "indexes" or well I guess pd.Index(dimension-coords). I assume not all potential indexes are useful for this? Alignment will still be based on 1D dimension-coords?
One (potential) edge case are MultiIndex which allow to align several non-dimension coordinates.

import xarray as xr
da1 = xr.DataArray([1, 2, 3], dims="x", coords=dict(time=("x", [1, 2, 3]), exp=("x", ["a", "a", "b"])))
da2 = xr.DataArray([2, 3, 4], dims="x", coords=dict(time=("x", [1, 2, 3]), exp=("x", ["a", "a", "a"])))
da1 = da1.set_index(x=("time", "exp"))
da2 = da2.set_index(x=("time", "exp"))

da1 + da2

<xarray.DataArray (x: 2)>
array([3, 5])
Coordinates:
  * x        (x) MultiIndex
  - time     (x) int64 1 2
  - exp      (x) object 'a' 'a'

Alignment hasn't been discussed yet here, but it should! Some quick thoughts:

support for alignment should probably be optional for an Xarray index wrapper.

like pandas.Index, the index wrapper classes that support it should implement .equals(), .union() and/or .intersection()
support might be partial if that makes sense (outer, inner, left, right, exact...).
index equality might involve more than just the labels, like the CRSIndex proposed in #2996
some indexes might implement inexact alignment, like in #4489 or a KDTree index that selects nearest-neighbors within a given tolerance
alignment may be "multi-dimensional", i.e., the KDTree example above vs. dimensions aligned independently of each other

we need to decide what to do when one dimension has more than one index that supports alignment

we should probably raise unless the user explicitly specify which index to use for the alignment

we need to decide what to do when one dimension has one or more index(es) but none support alignment

either we raise or we fail back (silently) to alignment based on dimension size

for inexact alignment, the tolerance might be given when building the index and/or when performing the alignment
are there cases when we want a specific index to perform alignment and another index to perform selection? It would be tricky to support that unless we allow multiple indexes per coordinate. "Meta" indexes (https://github.com/pydata/xarray/pull/4979#discussion_r585203065) would help but then I'm worried about the possible explosion of index wrapper classes.

There are also high-level methods that could use indexes in non-trivial ways.  These methods become complicated when considering nD versions of IntervalIndex (or CellIndex that represents the appropriate bounds variable, https://github.com/pydata/xarray/issues/1475). I'm raising these so we can define the scope of the API.

resample (CFTimeIndex and a DatetimeIntervalIndex)
DatetimeAccessor & TimedeltaAccessor properties (CFTimeIndex and a DatetimeIntervalIndex)
interp & interpolate_na,

with IntervalIndex, these become regridding operations. Should we support hooks for these operations?

differentiate, integrate, polyfit

raise an error if not a "simple" 1D index?

pad
sortby
plotting

plot.pcolormesh "infers" interval breaks along axes, which are really inferred bounds for the appropriate indexes.
plot.step again uses bounds. In fact, we may even want step to be the default 1D plotting function if the axis has bounds attached.

Another perhaps-unintended use-case is that various accessors will try to use .indexes to cache state (https://github.com/pydata/xarray/issues/3268#issuecomment-539463105).
An example is CRSIndex, where the crs applies to all variables in a dataset but isn't associated with a dimension on any of the other variables (Maybe this is a good definition for a PropertyIndex). This would be used by rioxarray, salem and friends. Also consider a possible XgcmIndex that basically contains an Xgcm.Grid object (also satisfies the PropertyIndex definition). We would want to propagate such PropertyIndexes in all operations, and also optionally check them during alignment (?).

There are also high-level methods that could use indexes in non-trivial ways.

Thanks @dcherian for listing those methods here, that's something worth to keep in mind! I think that for now it would be reasonable to restrict those methods to the indexes that are currently available in Xarray instead of trying to extend the API of Xarray index wrappers in order to support those special cases. I guess it's ok for "default" or "common" xarray indexes to provide extra functionality that could not be implemented in 3rd party indexes, as well as it would be ok for 3rd-party indexes to provide non-standard, extra functionality that would be reused for methods implemented in DataArray/Dataset accessors.

Maybe this is a good definition for a PropertyIndex

Are the CRSIndex and XgcmIndex examples really independent of any coordinate in the DataArray/Dataset? Looks like in #2996 a CRSIndex could be bound to x and y coordinates and a XgcmIndex could be bound to x, y, x_c, y_c, face, etc. coordinates?

Are the CRSIndex and XgcmIndex examples really independent of any coordinate in the DataArray/Dataset? Looks like in #2996 a CRSIndex could be bound to x and y coordinates and a XgcmIndex could be bound to x, y, x_c, y_c, face, etc. coordinates?

@rabernat @jbusecke (xgcm) @snowman2 @fmaussion @djhoese (crs) it would be interesting to have your thoughts here.
What would be the pros and cons of:

Refactoring xgcm.Grid into a XgcmGridIndex? The index would typically be assigned to the Dataset coordinates that are also specified in the coords argument of xgcm.Grid and all other arguments (except the Dataset itself) would become index options.  xgcm.Grid methods would then be accessible via Dataset accessor(s) (or eventually just replaced by xarray's corresponding methods).

Refactoring a crs attribute (either a "public" Dataset/DataArray attribute or hidden behind an accessor) into some CRSIndex that would typically be assigned to x/y or lat/lon coordinates?

A major advantage is that using a custom index, there's no need to encapsulate a Dataset/DataArray into a higher level structure (e.g., xgcm.Grid) and there would be more control on how it is propagated from one xarray object to another compared to an attribute or via a "stateful" accessor (e.g., crs). Another advantage is that Xarray selection and/or alignment can be customized. But that can be also a downside: unless we allow multiple indexes per coordinate, such XgcmGridIndex and CRSIndex would then have the responsibility of handling selection and alignment for all their corresponding coordinates. That may not be a big deal, though: XgcmGridIndex and CRSIndex could simply encapsulate pandas.Index instances for all (or a subset) of their coordinates.
Are there any other challenges and/or opportunities?

For reference for how rioxarray does things: https://corteva.github.io/rioxarray/stable/getting_started/crs_management.html
>>> import xarray, rioxarray
>>> xda = xarray.DataArray(1)
>>> xda.rio.write_crs(4326, inplace=True)
<xarray.DataArray ()>
array(1)
Coordinates:
    spatial_ref  int64 0
Attributes:
    grid_mapping:  spatial_ref
>>> xda.spatial_ref
<xarray.DataArray 'spatial_ref' ()>
array(0)
Coordinates:
    spatial_ref  int64 0
Attributes:
    crs_wkt:                      GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["...
    semi_major_axis:              6378137.0
    semi_minor_axis:              6356752.314245179
    inverse_flattening:           298.257223563
    reference_ellipsoid_name:     WGS 84
    longitude_of_prime_meridian:  0.0
    prime_meridian_name:          Greenwich
    geographic_crs_name:          WGS 84
    grid_mapping_name:            latitude_longitude
    spatial_ref:                  GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["...

A major advantage is that using a custom index, there's no need to encapsulate a Dataset/DataArray into a higher level structure (e.g., xgcm.Grid) and there would be more control on how it is propagated from one xarray object to another compared to an attribute or via a "stateful" accessor (e.g., crs)

I agree, this would be really nice.
One challenge is that often it is not advisable to explicitly build the coordinate arrays that correspond to such grids For example, consider a satellite image: 2D lat/lon arrays could be as expensive to store as the image itself, even though the values can be computed on the fly with very cheap arithmetic.
To fill this gap, we need first-class support for custom lazy arrays in xarray. If you read the documentation for the backend refactor (https://github.com/pydata/xarray/pull/4810), you'll see that we do have a minimal version of this internally in the form of core.indexing.LazilyIndexedArray. Ideally we would not only expose this functionality publicly, but would even factor it out into a separate lazy "duck array" library that could be used independently of xarray.

Ah yes, making it more easily reusable would be welcome indeed. I guess that such lazy arrays will be already needed for the creation of coordinates from the levels of an existing pandas.MultiIndex.

2D lat/lon arrays could be as expensive to store as the image itself, even though the values can be computed on the fly with very cheap arithmetic.

Just wanted to mention in case it comes up later, this is true for some datasets and for others the lon/lats are not uniformly spaced so they can't be calculated (just based on the way the satellite instrument works). They have to be loaded from the original dataset (on-disk file). For a while in the Satpy library we were storing 2D dask arrays for the lon/lat coordinates until we realized xarray was sometimes computing them and we didn't want that.

I think that for now it would be reasonable to restrict those methods to the indexes that are currently available in Xarray instead of trying to extend the API of Xarray index wrappers in order to support those special cases.

Fully agree. We should raise nice error messages when possible. I just wanted to raise awareness about this issue (i.e. methods that use indexes in non-trivial ways).

Are the CRSIndex and XgcmIndex examples really independent of any coordinate in the DataArray/Dataset?

Good point! I hadn't thought of it that way.

A major advantage is that using a custom index, there's no need to encapsulate a Dataset/DataArray into a higher level structure (e.g., xgcm.Grid) and there would be more control on how it is propagated from one xarray object to another compared to an attribute or via a "stateful" accessor (e.g., crs)

Yes! I'm v. happy to see this discussion is happening :)

But that can be also a downside: unless we allow multiple indexes per coordinate, such XgcmGridIndex and CRSIndex would then have the responsibility of handling selection and alignment for all their corresponding coordinates. That may not be a big deal, though: XgcmGridIndex and CRSIndex could simply encapsulate pandas.Index instances for all (or a subset) of their coordinates.

For XgcmIndex this is even desirable: https://github.com/xgcm/xgcm/issues/200, Re CRSIndex I think you're correct, it could just forward to whatever Index class it's wrapping

For reference for how rioxarray does things: https://corteva.github.io/rioxarray/stable/getting_started/crs_management.html

That's good to know, thanks! Like it may create specific indexes for time coordinates, I could imagine Xarray's decode_cf(decode_coords=True) to eventually return some kind of CRSIndex from any variable referred to in grid_mapping attribute.

Thanks everyone for your comments so far!! They have been really helpful in improving the notes!
This is now ready for another round of review. I've tried to include all the points raised in the discussion above. I also marked all the conversations as resolved even though it's still open for discussion! (it's just a way to "reset" them for more clarity). I'll move the notes into a design_notes folder just before merging this PR.
With the last commits, I think that the notes now cover most of the aspects regarding the use of indexes in Xarray. The goal with these notes is not to settle every detail of the refactoring (decisions can be made while iterating on the implementation), but rather describe the big picture and outline the main opportunities and challenges. Referring to the notes will help throughout the implementation. Hopefully it will allow more Xarray users and devs sharing their point of views to make sure we're not missing anything important here.
One thing that is not in the notes: to which acceptable extent this refactoring may introduce breaking changes? I think that it will be hard to avoid any breaking change. That said, as the index refactoring would rather bring internal data structures to the light I don't expect many things to break. Hardest part will probably be to ensure a smooth transition while updating the API that is too specific to pandas.MultiIndex into something that is more index-agnostic...

One thing that is not in the notes: to which acceptable extent this refactoring may introduce breaking changes? I think that it will be hard to avoid any breaking change. That said, as the index refactoring would rather bring internal data structures to the light I don't expect many things to break (at least, not the things that 90% of Xarray users often rely on). Hardest part will probably be to ensure a smooth transition while updating the API that is too specific to pandas.MultiIndex into something that is more index-agnostic...

My opinion is that breaking changes are OK if done with care. The main thing to keep in mind is that users dealing with change typically will not directly benefit from it.
So any breaking changes need to be intentional, and happen all at once in a major release. We also need to issue warnings when possible.
Depending on the magnitude of these changes, this could be a good reason to declare "xarray 1.0"

Thanks for your reviews @shoyer and @rabernat! I've updated the notes according to your comments.
From my point of view this is ready, but we can leave this PR open for another few days in case anyone else wants to add some comments (@pydata/xarray).
Next week I'll start with the implementation (I'll take on the open PR in https://github.com/pydata/xarray/projects/1 and will do only internal refactoring making sure that all tests are passing).

shall we merge?

shall we merge?

Yes! I wanted to wait for the bi-weekly dev meeting but I've just missed it (PST/PDT -> :facepalm: -> sorry).
Let's merge this and continue the discussion in follow-up issues/PRs.

authors: benbovy dcherian djhoese jthielen keewis mathause shoyer snowman2


text:
Explode polygons into rings: get_rings

Related to the comment from @brendan-ward at https://github.com/pygeos/pygeos/pull/318#pullrequestreview-615323018 about the indices returned by get_coordinates, which is fine for Multi-part geometries (through get_parts), but harder if you also want to know the indices of rings:

One approach (future PR, but good to consider how that impacts the API in this PR) would be to add an additional parameter return_ring_index that if True would add another integer array to the output tuple.  Would need to have a little validation: return_index must also be True and behavior for non-polygon types in a mixed collection is undefined (0 seems incorrect, -1 could produce bad indexing behavior later, etc).

An alternative to returning additional ring indices from get_coordinates, would be to have something similar as get_parts but then for the rings. This PR does a quick draft of such a function.
The main downside is that if you only need to indices, exploding into rings creates a lot of intermediate objects (and memory usage) just for that. Although the same can be said about get_parts for this use case.

This looks good! Is it conceivable to actually integrate the polygons into get_parts? Or is it too different in your opinion?
For just returning the indexes, I think we can try to also add a collection_index to get_coordinates. With that, we will have a complete representation of the geometries (except for the types and the mixed geometrycollections). That will be very useful for serialization.

Is it conceivable to actually integrate the polygons into get_parts? Or is it too different in your opinion?

Do you mean that get_parts would also automatically explode a polygon into its rings, like a MultiPolygon gets exploded in Polygons?
If we add it to get_parts, I think that should certainly be behind a flag (to enable with a keyword), because it is typically not want you want. For example when you have a mixture of Polygons/MultiPolygons, you want to get all as Polygons, and get_parts handles that perfectly.
Also, an important difference with rings is that those no longer are a simple composition. A MultiPolygon is semantically equal to its set of Polygons, but a Polygon is not just a combination of its rings, i.e. the different rings have a different interpretation (exterior/shell vs holes). I think that's an important distinction compared to what get_parts does for all types of geometries, and therefore probably worth to be a separate function?

If we would like such a get_rings as a separate function, I can certainly clean-up / finalize this PR.

I thought it might give us less code to maintain. Semantically there should indeed be a difference, I agree.
An option is to add explode_polygons=True/False to get_parts, but if it is easier implementation-wise to stick to the get_rings, please go for it.

Even with different public functions, the implementation in cython could still be shared.
Now, given that for rings, you need to get exterior ring and interior rings separately (it's not a single inner loop with GEOSGetGeometryN_r), it won't necessarily result in more readable code by putting that inside get_parts behind a keyword, I think (although it would certainly would reduce some duplication and thus less lines of code)

Now, given that for rings, you need to get exterior ring and interior rings separately (it's not a single inner loop with GEOSGetGeometryN_r), it won't necessarily result in more readable code by putting that inside get_parts behind a keyword, I think (although it would certainly would reduce some duplication and thus less lines of code)

I think we could do something on these lines:
cdef GetRingN(handle, polygon, n):
    if n == 0:
        return GEOSGetExteriorRing_r(handle, polygon)
    else:
        return GEOSGetInteriorRingN_r(handle, polygon, n - 1)

Then it looks much more like GEOSGetGeometryN_r. I think the extra overhead of the if-then will not be noticable.

I think we could do something on these lines:

Good idea, that's indeed a clean way to combine them (and reduce the cython code). Doing some timings now to ensure it doesn't give any slowdown (but I also don't expect that)

As expected, no noticeable speed difference by combining both.
Further updated the PR to include docs and tests. Should be ready for review now.

This seems like a case where it would be useful to also know the index or some other role identifier within the original geometry,

Yes. It will probably be possible to calculate most things from the current return value. Eg a mask for exterior vs interior rings could be created with np.insert(np.diff(index), 0, 1) == 0 for holes. And for example in https://github.com/pygeos/pygeos/pull/93#issuecomment-821103456 I needed the output in yet another form, and did a np.insert(np.bincount(index).cumsum(), 0, 0) to get offsets.
There might be several possible outputs people might need, which might be a reason to keep the function more simple (and document some examples, as those numpy code snippets I gave are not that trivial to found yourself). But of course, if there is a typical output you often need, it would be a good convenience to provide this.

also return inner index within each polygon, where 0 always indicates the shell and 1...N always indicates holes

This one would be similar as the "cumulative count" we did in the geopandas' explode implementation:
run_start = np.r_[True, outer_idx[:-1] != outer_idx[1:]]
counts = np.diff(np.r_[np.nonzero(run_start)[0], len(outer_idx)])
inner_index = (~run_start).cumsum()
inner_index -= np.repeat(inner_index[run_start], counts)

@brendan-ward are you OK with going forward with this PR as is? Your comments above (https://github.com/pygeos/pygeos/pull/342#pullrequestreview-644160823) are certainly heard, but I think the current PR gives functionality that is consistent with the API of get_parts (and is compatible with doing a reverse operation with polygons after https://github.com/pygeos/pygeos/pull/346/). And further enhancements / options in what kind of information is returned (or examples to achieve this) can be left as a follow-up I think.

source: gharchive/pull-request

authors: caspervdw jorisvandenbossche

repo: pygeos/pygeos

license_type: permissive

license_source: github-api


text:
Add support for setting and getting arrays in a Uniform via wrapper.

Allow setting arrays in uniforms in GLSL: uniform vec3 positions[9].
When accessing the shader via program["positions"] it will return this:
UniformArray(uniform=positions, data=[(128.0, 128.0, 128.0), (57.0, 123.0, 149.0), (36.0, 64.0, 76.0), (255.0, 255.0, 255.0), (255.0, 0.0, 255.0), (52.0, 85.0, 100.0), (151.0, 151.0, 151.0), (0.0, 0.0, 0.0), (178.0, 184.0, 216.0)])

Can be used as a list for the most part, you can iterate over it and do most of the things a list can do. It does not support sub-list assignment. (for example you can't set a program["positions"][0][1] = 20.0 you would have to pass the full data type type via program["positions"][0] = (12.0, 10.0, 20.0))
Set data in full via program["positions"] = new_full_array
Supports index getting: program["positions"][0] and setting: program["positions"][0] = (10.0, 20.0, 30.0) or:
array = program["positions"]
array[0] =  (10.0, 20.0, 30.0)

Also supports slicing.
Add checks for multidimensional arrays, as they can't be properly inspected. Not sure if I should raise an exception for them, warn, or just ignore, but went exception route for now. Can change if needed.
Looking for input on a change, when shader debugging is on, to either ignore or warn about getting/setting missing uniforms. When debugging or adjusting shaders, it gets quite annoying to have the program put out an exception when I have temporarily commented out some uniforms or ones that were optimized out. Would be nice if we can have an option to bypass this or at the least change to a warning.

This looks like a reasonable approach.  I also prefer the exception instead of warning, since it might cause confusion if the warning is missed.
I made some code comments, but the design overall looks good to me.

Made the changes you outlined with the exception of making everything gl_type * length * size as 1 length values would end up becoming arrays of arrays, which would mean single values would return as tuples and setting data they would have to set fully as [(1,), (2,), (3,)] or [[1], [2], [3]] which is not very user friendly.
Added warnings if debugging is on instead of exceptions for uniform setting and getting, still keeps the exceptions normally. In the future I can expand this to attributes as well, but not sure of the ramifications of bypassing that, at the moment.

repo: pyglet/pyglet


text:
CLexer doesn't understand float syntax

I can think of 4 different syntaxes for specifying floats and doubles in C:

0.123
0.123f
12.3e-2
12.3e-2f

The first 3 syntaxes highlighted correctly. The last one doesn't highlight f with the same color as the rest of the float.
My Pygments version: 2.6.1

123ff also gets highlighted as a float, even though it's not actually a float

@Akuli what should 123ff be highlighted as?

123f is a syntax error and should be highlighted as a syntax error
I would expect 123.ff to highlight as the float 123.f followed by f

maybe 123.ff should also be highlighted as an error? it gives a similar compiler error message as 123f

Thanks for this info, @Akuli! I'm fleshing out extensive unit tests for numeric literals. Looks like 1L is currently tokenized as a float so I'm diving into all numeric literals, not just floats.

Just a friendly reminder: Pygments is a highlighter, not a compiler frontend. Not every syntax error needs to be recognized as an error, and semantic errors even less so.

Georg, thanks for the reminder. I 100% agree. My goal won't be to introduce syntax checking, only to ensure that every valid numeric literal is tokenized correctly.
On September 7, 2020 5:07:20 AM UTC, Georg Brandl notifications@github.com wrote:

Just a friendly reminder: Pygments is a highlighter, not a compiler frontend. Not every syntax error needs to be recognized as an error, and semantic errors even less so.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/pygments/pygments/issues/1468#issuecomment-688038433

source: gharchive/issue

authors: Akuli birkenfeld kurtmckee

repo: pygments/pygments

license_source: bigquery


text:
How to use pykaldi to get phone posterior?

I want to calculate GOP by pykaldi ，and now I need phone-alignment and phone posterior ，I can get the former by aligner，however I don‘t know how to get the latter.

repeat

repo: pykaldi/pykaldi


text:
different version for windows and linux in one package

AVOID POSTING ISSUES UNDER THIS CATEGORY.
I want configurate one package in windows platform to install version==3.1.13 , in linux platform to install version==3.1.12

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.
If you're on macOS, run the following:
$ pipenv --support | pbcopy

If you're on Windows, run the following:
> pipenv --support | clip

If you're on Linux, run the following:
$ pipenv --support | xclip

This is essentially the same problem as #2171. (And you know this.)
I have proposed a workaround.
There is not much interest between core developers to implement native support.
We do not exclude the possibility, but someone needs to take charge, design a syntax that makes sense, and implement it. E.g. you?

repo: pypa/pipenv

license: MIT


text:
Add support for dependencies (such as six)

Originally reported by: jaraco (Bitbucket: jaraco, GitHub: jaraco)

I'd like to use the six library for Python 2 compatibility.
As it currently stands, setuptools does not have any non-extra dependencies, so introduction of any dependency has the potentially to be disruptive, so this ticket is created primarily around that discussion.
Here's what I have in mind:

Vendor six in the sdist to be used when setuptools installs itself (but never install from that verison).
Declare those dependencies as proper dependencies in install_requires.
Have the code reference those libraries naturally.

This change would require bootstrap installers to either:

Have the dependencies installed in advance.
Have network access to install the dependencies from PyPI.
Configure an index-url from which to install the dependencies.

What I want to avoid is for setuptools to vendor the dependencies for any purpose other than bootstrapping. I'd also like to avoid having conditional imports in the code. I especially want to avoid creating complicated vendoring scenarios that violate the expectations of platform package maintainers (namely Debian and Fedora).

Bitbucket: https://bitbucket.org/pypa/setuptools/issue/229

In the aforementioned "those changes", here's an updated verison of those changes in Github.

authors: ghost jaraco

repo: pypa/setuptools

license: mit


text:
fix: cannot set root package properly

According to https://docs.python.org/3/distutils/setupscript.html#listing-whole-packages, the root package can be set through package_dir by two ways:

'': <root-path>
'<root-pkg>': <the-dirpath-of-its-init-file>

However, the case is it only get changed if the key '' in package_dir. Otherwise, it just use a default value: the distribution root (os.cwd()) .
This make errors for pkg without a standard source directory layout.
E.g. when a pkg installed on develop mode, and with package_dir set not with a key :. Then the installed pkg wouldn't find itself with right root path.

I'd like to see a couple of things here:

More explanation about the flaw that's encountered. Do you have a sample project that fails under the current behavior? How does it fail?
Tests. Since this change doesn't introduce any tests to capture the desired behavior, the behavior that's introduced here could be removed and tests would still pass. Also, the tests could provide links back to the justification for the behavior (the explanation).

I recommend to start with the explanation, because it may be that there is a better recommendation that avoids this undesirable behavior altogether, and maybe the solution is to disallow the usage that led to the undesirable behavior.

Feel free to revisit when you have time to address the concerns. We can either re-open this PR or create a new one.

authors: jaraco songmeixu


text:
Public BigQuery table for distribution metadata - Bulk Load Script

This PR extends from #8093. It contains the script that can be used to populate the BigQuery tables  from the existing releases on PyPI. This PR doesn't need to be merged as the script just needs to be used once for migrating metadata.
Once #8093 is merged and data starts populating on the BigQuery table, note the oldest timestamp and other config parameters into the script to make sure all the metadata in the PostgreSQL database is migrated to BigQuery.
This script was tested on the local dev database which is similar to the TestPyPI database and the results as follows:
# of releases: 131,392
Total time: 34m 15s
Batch size: 1000               (how many releases were queried and uploaded at once)
Avg time / branch: 15.57s

Assuming that the prod PyPI is approx. 6 times TestPyPI, an estimated runtime of the migration would be around 3h 25min. This migration can occur concurrently to other PyPI processes and doesn't block any additions or updates to releases. These runtimes occurred with the PostgreSQL server running on i7 3Ghz with 4GB of container ram, so the true runtime could be faster than the estimated amount.

The change made in aa387b6cd8ce4154641766e206cc3deab317b9e4 under #8093 now contains an implementation of a sync task which will not only act as a fail safe in case of a failure during file upload but also be used as a migration script to move existing release files from the current database to BigQuery. Thus, we longer have a use for an independent migration script such as the one proposed in this PR.

repo: pypa/warehouse


text:
Nuke fails to publish with two subsets while creating review files for second subset

Describe the bug
Extract burnins fails to make review files with burnin for second subset.
To Reproduce

In Nuke, create main write render, and write render with new subset. Make sure review checkbox is ON.
Locally Publish shot with both outputs
Main subset is correctly rendered, with mov files and burnins
Second subset fails during the publish on creating burnins for second subset. Main subset burnins are created. Looks like burnins are trying to use main subset movs instead of second subset movs as an input.

RuntimeError: Failed to run: "ffprobe" -v quiet -print_format json -show_format -show_streams"//dps150/zback/GFX_storage/T009_pypetestAA/shots/trs001/work/compositing/renders/nuke/renderCompositingMain/renderCompositingMain.baked_h264.mp4"
Expected behavior
Pype should be able to make review files (with burn-ins) for more then one output.
Desktop (please complete the following information):

OS: Windows 10
Host: Nuke 11.3

Additional context
Publish was also failing on log messages in pype/hosts/nuke/lib.py:
https://github.com/pypeclub/pype/blob/a17fa74d4626a5c6d37475fc0c5a84318ea4f0b3/pype/hosts/nuke/lib.py#L1356
https://github.com/pypeclub/pype/blob/a17fa74d4626a5c6d37475fc0c5a84318ea4f0b3/pype/hosts/nuke/lib.py#L1365
https://github.com/pypeclub/pype/blob/a17fa74d4626a5c6d37475fc0c5a84318ea4f0b3/pype/hosts/nuke/lib.py#L1393
https://github.com/pypeclub/pype/blob/a17fa74d4626a5c6d37475fc0c5a84318ea4f0b3/pype/hosts/nuke/lib.py#L1417
Plus there was a try to delete non-existent Nuke nodes:
https://github.com/pypeclub/pype/blob/a17fa74d4626a5c6d37475fc0c5a84318ea4f0b3/pype/hosts/nuke/lib.py#L1170
[cuID:PYPE-1144]

Sending to Deadline has a similar issue with multiple subsets.

This is an old issue - has this been resolved since?

I checked local and farm (Deadline) publish. Both works. It has been resolved.

authors: BigRoy jrsndl tokejepsen

repo: pypeclub/OpenPype


text:
Deleting an organization should add a warning

When deleting an organization, a warning should let the owner know to download invoices/receipts from Stripe. It should also indicate what will happen to the subscription plan - whether the plan will be cancelled automatically or if the user has to cancel the plan before deleting the organization.

Seems we have two chances to communicate this, one in the "danger box"

and one in the confirm modal

I'm not sure how hard it is to add text to the modal, but if its possible I think this would be good to do for both.

Priority - high

resolved in #13395

authors: ewdurbin s-mm

repo: pypi/warehouse


text:
refactor: make captcha a real service

Best reviewed commit by commit, since there's some that are not too interesting (example: mass rename).
Paves the way for a future captcha service via interface.

Looks great, my only concern was CSP but it appears we are ahead of that!

authors: ewdurbin miketheman


text:
wB97m-D3BJ

Hello,
is it possible to use the functional wB97m-D3BJ in gpu4pyscf?
It is wB97m-V with the VV10 replaced with D3(BJ). https://pubs.acs.org/doi/abs/10.1021/acs.jctc.8b00842
It is available in PSI4 but I would like to use PySCF for the GPU acceleration

Yes, it is possible.  There will some script to work around the xc functional names. In libXC, there is only one functional for wb97m-v, which includes VV10. But the name wb97m-v is not available in dftd3. Here is the script to replace VV10 with D3(BJ).
import numpy as np
import cupy
import pyscf
from pyscf import lib
from gpu4pyscf.dft import rks, libxc
lib.num_threads(8)

atom = '''
O       0.0000000000    -0.0000000000     0.1174000000
H      -0.7570000000    -0.0000000000    -0.4696000000
H       0.7570000000     0.0000000000    -0.4696000000
'''

xc = 'wb97m-v'
bas = 'def2-tzvpp'
scf_tol = 1e-10
max_scf_cycles = 50
grids_level = 5

mol = pyscf.M(atom=atom, basis=bas)

mol.verbose = 3
mf = rks.RKS(mol, xc=xc).density_fit()

# turn off nonlocal correction
mf.nlc = None
mf._numint.libxc.is_nlc = lambda x: False
mf.grids.level = grids_level
mf.grids.atom_grid = (99,590)
mf.conv_tol = scf_tol
mf.max_cycle = max_scf_cycles
e_dft = mf.kernel()

# calculate dispersion correction
from gpu4pyscf.lib import dftd3
dftd3_model = dftd3.DFTD3Dispersion(mol, xc='wb97m', version='d3bj')
e_disp = res = dftd3_model.get_dispersion()['energy']
e_tot = e_dft + e_disp

print(f"total energy = {e_tot}")

thank you very much for the clear example! I will try it out

Hello. I only now got around to trying this. I have used the new syntax I found in the example: https://github.com/pyscf/gpu4pyscf/blob/master/examples/01-h2o_with_dispersion.py
I get an error when computing the gradient. How do I also compute the gradient with the D3bj?
here is the error:
Traceback (most recent call last):
  File "run_test.py", line 36, in <module>
    g_dft = g.kernel()
            ^^^^^^^^^^
  File "/scratch/users/sfarr/miniconda3/envs/gpu4pyscf_new/lib/python3.11/site-packages/pyscf/grad/rhf.py", line 415, in kernel
    de = self.grad_elec(mo_energy, mo_coeff, mo_occ, atmlst)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/users/sfarr/miniconda3/envs/gpu4pyscf_new/lib/python3.11/site-packages/gpu4pyscf/grad/rhf.py", line 523, in grad_elec
    dvhf = mf_grad.get_veff(mol, dm0)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/users/sfarr/miniconda3/envs/gpu4pyscf_new/lib/python3.11/site-packages/gpu4pyscf/df/grad/rks.py", line 55, in get_veff
    raise NotImplementedError
NotImplementedError

and here is my script:
import numpy as np
import pyscf
from gpu4pyscf.dft import rks

atom ='''
O       0.0000000000    -0.0000000000     0.1174000000
H      -0.7570000000    -0.0000000000    -0.4696000000
H       0.7570000000     0.0000000000    -0.4696000000
'''

mol = pyscf.M(
    atom=atom,                         # water molecule
    basis='def2-tzvppd',                # basis set
    verbose=1                          # control the level of print info
    )

mf_GPU = rks.RKS(                      # restricted Kohn-Sham DFT
    mol,                               # pyscf.gto.object
    xc='wB97m-V'                         # xc funtionals, such as pbe0, wb97m-v, tpss,
    ).density_fit()                    # density fitting

# turn off nlc
mf_GPU.nlc = None
mf_GPU._numint.libxc.is_nlc = lambda x: False

mf_GPU.disp = 'd3bj'

# Compute Energy
e_dft = mf_GPU.kernel()
print(f"total energy = {e_dft}")

# Compute Gradient
g = mf_GPU.nuc_grad_method()
g_dft = g.kernel()
print(g_dft)

Thank you

@sef43 Thank you for raising the issue here. PySCF DFT APIs have been improved for these special xc functionals. https://github.com/pyscf/pyscf/releases/tag/v2.6.0
Now you can simply do the following. Please see those examples for detailed usage.
import numpy as np
import pyscf
from gpu4pyscf.dft import rks

atom ='''
O       0.0000000000    -0.0000000000     0.1174000000
H      -0.7570000000    -0.0000000000    -0.4696000000
H       0.7570000000     0.0000000000    -0.4696000000
'''

mol = pyscf.M(
    atom=atom,                         # water molecule
    basis='def2-tzvppd',                # basis set
    verbose=1                          # control the level of print info
    )

mf_GPU = rks.RKS(                      # restricted Kohn-Sham DFT
    mol,                               # pyscf.gto.object
    xc='wB97m-d3bj'                         # xc funtionals, such as pbe0, wb97m-v, tpss,
    ).density_fit()                    # density fitting

# Compute Energy
e_dft = mf_GPU.kernel()
print(f"total energy = {e_dft}")

# Compute Gradient
g = mf_GPU.nuc_grad_method()
g_dft = g.kernel()
print(g_dft)

@sef43 Yes, PySCF v2.6 and GPU4PySCF v0.8 support RKS(mol, xc='wb97m-d3bj'). I deleted the comment before because I am planning to do more tests. I will add more unit tests for the new features.

https://github.com/pyscf/gpu4pyscf/pull/173
The PR mentioned above.


text:
DFT-D3/D4 gradient tests fail on CI (except aarch64 and macos)

linux-build (ubuntu-20.04, 3.8) and linux-build (ubuntu-20.04, 3.12) fail:
FAILED pyscf/grad/test/test_rks.py::KnownValues::test_finite_diff_df_rks_d3_grad - RuntimeError: No entry for 'lda,vwn' present
FAILED pyscf/grad/test/test_rks.py::KnownValues::test_finite_diff_df_rks_d4_grad - RuntimeError: Functional 'lda,vwn' not known
FAILED pyscf/grad/test/test_uks.py::KnownValues::test_fnite_diff_uks_d3_grad - RuntimeError: No entry for 'lda,vwn' present
FAILED pyscf/grad/test/test_uks.py::KnownValues::test_fnite_diff_uks_d4_grad - RuntimeError: Functional 'lda,vwn' not known

See e.g. https://github.com/pyscf/pyscf/actions/runs/8912712957/job/24476712155?pr=2195
At the same time, linux-build-aarch64 (ubuntu-20.04, cp39-cp39) and macos-build run fine.

fixing in https://github.com/pyscf/pyscf/pull/2198

repo: pyscf/pyscf


text:
2.0 is not on PyPI

https://pypi.org/project/pyscf/

Hi all,
Is the PySCF v2.0.0a version (libxc 5.1.2) still not on pypi?
https://pypi.org/project/pyscf/#history
Could it be a naming problem,
may cause by a misspelled release version (2.0a instead of 2.0.0a)?
https://github.com/pyscf/pyscf/commit/b76d2ceae0011d06b279f937bb36ce3c137dc57b
I found a pypi version called 2.0a0
https://github.com/pyscf/pyscf/runs/2281914728?check_suite_focus=true#logs
https://test.pypi.org/project/pyscf/2.0a0/
which installs.
However, it seems still not support for instance r2SCAN ('497,498')?
Failed to load library libtblis_einsum
XC functional 497 not found

A PySCF version properly installing from pip with support of newer libxc versions would be very nice to have.
Best,
Sebastian

Hi there,
I was wondering about the same thing.
There seems to be a new 2.0 version, but via pypi I do not get it.
Also, having the most current GitLab version of LibXC as a standard would
be nice for the future.
Best,
Kai

Hi @yurivict , has this been resolved?

authors: kaitrepte theonov13 verena-neufeld yurivict


text:
Fix tutorials' display on TopoModelX doc's website

What?
The Tutorials Tab of TopoModelX's documentation website does not display the tutorials properly.
https://pyt-team.github.io/topomodelx/tutorials/index.html
Specifically, the headers of every tutorials shows up which clutters the page without providing any useful information:
Tutorials
[Train a Cellular Attention Network (CAN)](https://pyt-team.github.io/topomodelx/notebooks/cell/can_train.html)
[Set-up](https://pyt-team.github.io/topomodelx/notebooks/cell/can_train.html#Set-up)
[Pre-processing](https://pyt-team.github.io/topomodelx/notebooks/cell/can_train.html#Pre-processing)
[Create the Neural Network](https://pyt-team.github.io/topomodelx/notebooks/cell/can_train.html#Create-the-Neural-Network)
[Train the Neural Network](https://pyt-team.github.io/topomodelx/notebooks/cell/can_train.html#Train-the-Neural-Network)
[Train the Neural Network with Attention](https://pyt-team.github.io/topomodelx/notebooks/cell/can_train.html#Train-the-Neural-Network-with-Attention)
[Train a Convolutional Cell Complex Network (CCXN)](https://pyt-team.github.io/topomodelx/notebooks/cell/ccxn_train.html)
[Set-up](https://pyt-team.github.io/topomodelx/notebooks/cell/ccxn_train.html#Set-up)

Etc...
We need to prevent these headers to appear, so that only the thumbnails of the tutorials are shown.
Why?
Because displaying headers of tutorials does not provide any useful information and clutters the page.
Where?
On TopoModelX, modify the documentation website's "tutorials" tab by modifying:
https://github.com/pyt-team/TopoModelX/tree/main/docs/tutorials
How?
Maybe the :maxdepth: 1 parameter can be tuned?

@ninamiolane
The tutorial website for me opens up like this:

@ninamiolane
Would you like me to remove the Section Navigation on the left?

@devendragovil indeed, the display of the tutorials has changed (and improved) since this task was done.
Tasks that remain:

[ ] FIX: Make the titles consistents
Many tutorials have the title "Train a Hypergraph Neural Network". That is not precise enough.
Let's modify the tutorials titles so that they are of the form "Train a [name of the model] ([abbreviation of the model])".
For example: "Train a Cell Attention Network (CAN)".
[ ] If needed, clean the Navigation section on the left.
By updating the titles, the Navigation section might become more readable already.
Clean it such that only the titles of the tutorials appear, but not their sections' titles.
No need to remove it entirely.

Thanks!

authors: devendragovil ninamiolane

repo: pyt-team/TopoModelX


text:
add reference gain design pattern to create_statefbk_iosystem

This PR adds a second design pattern to the create_statefbk_iosystem function, allowing controllers that use a static gain on the reference input instead of the desired state and input.  As described in the user documentation:

The ~control.create_statefbk_iosystem function can be used to create an I/O system consisting of a state feedback gain (with optional integral action and gain scheduling) and an estimator.  A basic state feedback controller of the form
$u = u_d - K (x - x_\text{d})$
can be created with the command::
ctrl, clsys = ct.create_statefbk_iosystem(sys, K)
where sys is the process dynamics and K is the state feedback gain (e.g., from LQR).  The function returns the controller ctrl and the closed loop systems clsys, both as I/O systems.  The input to the controller is the vector of desired states $x_\text{d}$, desired inputs $u_\text{d}$, and system states $x$.
The above design pattern is referred to as the "trajectory generation" ('trajgen') pattern, since it assumes that the input to the controller is a feasible trajectory $(x_\text{d}, u_\text{d})$.  Alternatively, a controller using the "reference gain" pattern can be created, which implements a state feedback controller of the form
$u = k_\text{f}, r - K x$,
where $r$ is the reference input and $k_\text{f}$ is the feedforward gain (normally chosen so that the steady state output $y_\text{ss}$ will be equal to $r$).
A reference gain controller can be created with the command::
ctrl, clsys = ct.create_statefbk_iosystem(sys, K, kf, feedfwd_pattern='refgain')

The reference gain design pattern is described in more detail in Section 7.2 of FBS2e (Stabilization by State Feedback) and the trajectory generation design pattern is described in Section 8.5 (State Space Controller Design).

In addition, this PR includes some changes that make the error messages for setting up a state space system more informative, but giving the expected size of matrices that do not have the right dimensions.  (This could go in a separate PR, but it is here because I got frustrated with the error messages while debugging some examples.)

coverage: 94.678% (-0.02%) from 94.697%
when pulling 222b08d4d8931b44f5cf46cbf789db19240e5903 on murrayrm:create_statefbk-29Nov2024
into 57b530742bae451fc566f3fb00acf18435c052b1 on python-control:main.

I will review this today.

authors: coveralls murrayrm slivingston

repo: python-control/python-control


text:
Time response plots

This PR implements time response plots for the various simulation routines in python-control via a new function time_response_plot() and via the plot() method on the TimeResponseData class.  This is aligned with the new plotting paradigm in #645.
Examples (from the user documentation):
Input/output time responses are produced one of several python-control functions: forced_response, impulse_response, initial_response(), input_output_response(), and step_response(). Each of these return a TimeResponseData object, which contains the time, input, state, and output vectors associated with the simulation. Time response data can be plotted with the time_response_plot() function, which is also available as the plot() method. For example, the step response for a two-input, two-output can be plotted using the commands:
sys_mimo = ct.tf2ss(
    [[[1], [0.1]], [[0.2], [1]]],
    [[[1, 0.6, 1], [1, 1, 1]], [[1, 0.4, 1], [1, 2, 1]]], name="MIMO")
response = step_response(sys)
response.plot()

which produces the following plot:

The TimeResponseData object can also be used to access the data from the simulation:
time, outputs, inputs = response.time, response.outputs, response.inputs
fig, axs = plt.subplots(2, 2)
for i in range(2):
    for j in range(2):
        axs[i, j].plot(time, outputs[i, j])

A number of options are available in the plot method to customize the appearance of input output data. For data produced by the impulse_response() and step_response() commands, the inputs are not shown. This behavior can be changed using the plot_inputs keyword. It is also possible to combine multiple lines onto a single graph, using either the overlay_signals keyword (which puts all outputs out a single graph and all inputs on a single graph) or the overlay_traces keyword, which puts different traces (e.g., corresponding to step inputs in different channels) on the same graph, with appropriate labeling via a legend on selected axes.
For example, using plot_input=True and overlay_signals=True yields the following plot:
ct.step_response(sys_mimo).plot(
  plot_inputs=True, overlay_signals=True,
  title="Step response for 2x2 MIMO system " +
  "[plot_inputs, overlay_signals]")

Input/output response plots created with either the forced_response() or the input_output_response() functions include the input signals by default. These can be plotted on separate axes, but also “overlaid” on the output axes (useful when the input and output signals are being compared to each other). The following plot shows the use of plot_inputs=’overlay’ as well as the ability to reposition the legends using the legend_map keyword:
timepts = np.linspace(0, 10, 100)
U = np.vstack([np.sin(timepts), np.cos(2*timepts)])
ct.input_output_response(sys_mimo, timepts, U).plot(
    plot_inputs='overlay',
    legend_map=np.array([['lower right'], ['lower right']]),
    title="I/O response for 2x2 MIMO system " +
    "[plot_inputs='overlay', legend_map]")

Another option that is available is to use the transpose keyword so that instead of plotting the outputs on the top and inputs on the bottom, the inputs are plotted on the left and outputs on the right, as shown in the following figure:
U1 = np.vstack([np.sin(timepts), np.cos(2*timepts)])
resp1 = ct.input_output_response(sys_mimo, timepts, U1)

U2 = np.vstack([np.cos(2*timepts), np.sin(timepts)])
resp2 = ct.input_output_response(sys_mimo, timepts, U2)

ct.combine_traces(
    [resp1, resp2], trace_labels=["Scenario #1", "Scenario #2"]).plot(
        transpose=True,
        title="I/O responses for 2x2 MIMO system, multiple traces "
        "[transpose]")

This figure also illustrates the ability to create “multi-trace” plots using the combine_traces() function. The line properties that are used when combining signals and traces are set by the input_props, output_props and trace_props parameters for time_response_plot.
Additional customization is possible using the input_props, output_props, and trace_props keywords to set complementary line colors and styles for various signals and traces:
out = ct.step_response(sys_mimo).plot(
    plot_inputs='overlay', overlay_signals=True, overlay_traces=True,
    output_props=[{'color': c} for c in ['blue', 'orange']],
    input_props=[{'color': c} for c in ['red', 'green']],
    trace_props=[{'linestyle': s} for s in ['-', '--']])

coverage: 94.967% (+0.1%) from 94.849% when pulling dd6983ef83e5cf20e6a9e7ed278d377e5594f1ff on murrayrm:time_plots-20Jun2023 into d1530357f1d57837756ea7adcb0ac592d473eb93 on python-control:main.

Looks pretty comprehensive and pretty good to me. I particuarly like that this makes it much quicker to plot the inputs and outputs of a mimo system.

authors: coveralls murrayrm sawyerbfuller


text:
Raspberry PI Locator

Script Title - Raspberry PI Locator
Brief - A simple Python script to check popular Raspberry PI-certified shops for availability of Model 4s.
Instructions

Create a new folder for your script and file/folder name should be appropriate.
Create a README.md (using this template) in your folder for program Instructions
add requirements.txt if needed

Programming Language

[x] Python

Happy Coding

Hello, I would like to be assigned to this issue please.

repo: python-geeks/Automation-scripts


text:
How can I access manipulate text that is placed in a shape in a document.

Someone has drawn a shape in a document that I need to manipulate. I iterated over all the inline_shapes and can't see to get to the text within each of the results. I was able to confirm the text exists in the doc using BeautifulSoup and pulling out the XML node the text is in. I also confirmed the text is not part of the tables or the paragraphs objects in the doc.

Is there a way to pull text from within a shape in a doc
If not, is there a way to manipulate the XML directly?

@scanny huge help thanks. That xpath function worked to find the data and change it. Seems like we can only clear it or replace it though .remove() doesn't seem to remove the node. Any additional guidance or help 😄  🙇

Not sure if we are doing it properly, but it seems like you have to find the text via enumeration, but then return to the root (txbx in the example above) and make the change. Doing it within the enumeration wont work as it does with other enumerators in python-docx.
txbx[0][idx].getparent().remove(txbx[0][idx])

@sshadmand what exactly are you trying to remove? It would be either an element, an attribute, or the text enclosed between an open and close element.

Well, I assume the element, but the goal is to remove the part of the Word document so there is no residuals left over (space, bullet, text, new lines etc). For example:
Original
[Shape Starts] [line 1] Hello [line 2] World [line 3] - Something [line 4] more [Shape ends]
After
[Shape Starts] [line 1] [line 2] [line 3] - [line 4] more [Shape ends]
Wanted
[Shape Starts] [only line] more [Shape ends]

How to remove the None. It is getting repeated.

authors: Laxmi530 scanny sshadmand

repo: python-openxml/python-docx


text:
Granting Aya-S the needed permissions to enter gitter channel and review PRs

Hello Everyone,
would be great if someone with the required privileges would Grant me the needed permissions to enter the gitter channel and review PRs.
TIA!

Added you to the organization and the gitter channel. :)

wohoo! thanks @datapythonista

@Aya-S  I know this issue was closed, but I should have clarified it earlier. It seems a team member (e.g. me) can not invite/authorize others to join the team because

There's no such option on my team page
The link I sent you is hidden to non-members (https://github.com/orgs/python-sprints/teams/pandas-mentoring/members)

So I guess only the team maintainer can invite people. And I see Marc has already added 13 people in the team. Thanks @datapythonista

@dujm do you have an Add member button in https://github.com/orgs/python-sprints/teams/pandas-mentoring/members ?
That's how I'm adding people to the team. Afaik I'm adding people as admins, so I guess you should have that button and be able to add other people too, but if that's not the case let me know and I'll check in more detail how to make the rest of the people admins.
Thanks!

@datapythonista  No, I don't have this button, only the 'Leave team' button ;)

ok, don't touch the Leave team button. ;)
@dujm you were already an admin of the team and the repo, but I made you and few more people an admin of the whole organization. Hopefully now you're able to add people, otherwise not sure what else I can do.

:-D thanks @datapythonista Now I can see the 'Add a member button.  I will add people to the 'Team' and 'Gitter'

Thanks a lot @dujm. Feel free to send a message to the google group, if you think someone may be missing.

authors: Aya-S datapythonista dujm

repo: python-sprints/pandas-mentoring


text:
Implementation upgrade document does not mention connection.close_code

It seems that the Attributes of connection objects chapter (or does it belong elsewhere in the document?) does not explain how to deal with the close_code and close_reason attributes that were previously available on the connection.
I skimmed through the docs, and it seems that .protocol.close_code is what I need, but it would be nice to mention in it the upgrade document too. Or maybe there's a better way to deal with this altogether?

Usually, you get a ConnectionClosed exception at some point and it provides the close code and reason. However, in the tests where you need this, you don't have an exception, so that doesn't meet your needs.

More context that you need — I'm writing this to make the right decision.
The WebSocket spec says that the close code and reason is defined by the first close frame received — meaning it's the code and reason from the perspective of the remote side of the connection. This is what close_code and close_reason mean in the legacy implementation and in the Sans-I/O layer (i.e. connection.protocol).
However, most users are interested in the code and reason from their own perspective. As a consequence, this API — and arguably the WebSocket specification itself — breaks the principle of least astonishement. That's why I decided to provide all the details in the ConnectionClosed exception.
I don't remember if I specifically decided not to include close_code and close_reason in the new asyncio and threading APIs.

There should be a public API for this. I see three options

Document connection.protocol. It's there and it isn't going anywhere. Everyone who looks for it will find it and use it (just like you did).
Add connection.close_code and connection.close_reason. This has the benefit of being more backwards compatible but the downside of cluttering the Connection API with information that extremely few people need (e.g. framework / library maintainers like you).
Do 1. + 2. + deprecate immediately the attributes. This provides a slightly smoother migration path. I don't think it's worth the effort.

If I go for option 1. then it must be documented in the upgrade guide.

Thank you for taking the time to report this issue!

Thank you for taking the time to report this issue!

Thanks for really swift action on this :100:
But as I understand/hope, connection.protocol.close_code isn't going anywhere either?
Because at least for Falcon I cannot fully upgrade to the newest version since we kept Python 3.8 support for now, so we are effectively getting 13.1 on Python 3.8, and 14.0 or newer on other versions. In a pinch, we can stop testing on 3.8 though, and rely just on the unit tests.
(Falcon does not depend on Websockets per se, we just use the client for testing the framework on different ASGI servers.)

It's a private API but extremely unlikely to change.

FYI: I'm planning to wait a few days for other reports of incompatibilities between the legacy and new implementation, then I'll make a release with that fix.

repo: python-websockets/websockets


text:
Expose ServerConnection in websockets.init.all

In the new websockets v14, most things are re-exported on the base websockets module, which makes imports easy, I can do from websockets import serve, ConnectionClosed, broadcast for instance.
One that I cannot do is from websockets import ServerConnection. ServerProtocol is exported, can ServerConnection be exported as well? If I have a list of connections, I would want to store them in a list that is type-annotated as list[ServerConnection], and so I would need access to the type. It would be convenient to not have to reach into the separate websockets.asyncio.server module.

Importing from the root package dates back to when the library was much smaller. It has become problematic as the library has grown.
(The complexity is created by not wanting to import three implementations, including client and server, when users need just one of the six combinations. It is compounded by the need to support static typing.)
I kept backwards compatibility because forcing everyone to migrate to the new import paths would be too much busywork. However, I don't think convenience imports are a good idea, so I'm avoiding to add more.
That's how you end up with from websockets.asyncio.server import ServerConnection.
Also, I expected that few users would need to import it, as the new API that became the default in 14.0 supports more use cases without subclassing. I didn't anticipate that everyone who uses typing needs it.
Let me think about that.


text:
Página de error en caso de que no queden entradas a mitad de transacción

En caso de que, después de que Stripe valide los datos bancarios, no queden entradas, justo antes del POST, habrá que redirigir a una página con error.

Parece una experiencia de usuario un poco pobre, no podemos hacer una reserva temporal de la entrada? Claro que finalmente, si pasa cierto tiempo y la reserva caduca, aún necesitamos esta página.

authors: jdevera sdelquin

repo: pythoncanarias/pycan-web


text:
🛑 Friends Api is down

In c0492b3, Friends Api (https://friends.roblox.com/) was down:

HTTP code: 404
Response time: 76 ms

Resolved: Friends Api is back up in 8b3cc0b after 37 minutes.

repo: pythoniaweb/statusboom


text:
Implement support for self-contained .NET project

Example project available in previous pythonnet issue.

I found a neat workaround to run .NET projects in a self-contained manner which could suffice for most people encountering this issue. What we need to do is to bring dotnet by ourselves.
Example:

my-dotnet-directory

libhostfxr.so
shared

Microsoft.NETCore.App

6.0.11

...files

Because we are bringing all dotnet libraries already, we don't need to build our project as self-contained. Normally building the project is enough. This is how I call it in pythonnet
import os
from pythonnet import load

dotnet_root = os.path.abspath('./my-dotnet-directory')
runtime_config = os.path.abspath('runtimeconfig.json')
load("coreclr", dotnet_root=dotnet_root, runtime_config=runtime_config)

import clr
import os

clr.AddReference(os.path.abspath('./my-project/my-project.dll'))

from MyProject import Program
program = Program()
program.ExecuteSync()

This way I'm able to run my application as 'self-contained'
runtimeconfig.json:
{"runtimeOptions": {"tfm": "net6.0", "framework": {"name": "Microsoft.NETCore.App", "version": "6.0.11"}}}

I found a neat workaround to run .NET projects in a self-contained manner which could suffice for most people encountering this issue. What we need to do is to bring dotnet by ourselves.
Example:

my-dotnet-directory

libhostfxr.so

shared

Microsoft.NETCore.App

6.0.11

...files

Because we are bringing all dotnet libraries already, we don't need to build our project as self-contained. Normally building the project is enough. This is how I call it in pythonnet
import os
from pythonnet import load

dotnet_root = os.path.abspath('./my-dotnet-directory')
runtime_config = os.path.abspath('runtimeconfig.json')
load("coreclr", dotnet_root=dotnet_root, runtime_config=runtime_config)

import clr
import os

clr.AddReference(os.path.abspath('./my-project/my-project.dll'))

from MyProject import Program
program = Program()
program.ExecuteSync()

This way I'm able to run my application as 'self-contained'
runtimeconfig.json:
{"runtimeOptions": {"tfm": "net6.0", "framework": {"name": "Microsoft.NETCore.App", "version": "6.0.11"}}}

@vgibilmanno Are you able to run this in linux? I am trying in Ubuntu

@rishky-msft I'm able to run this on Debian 11 and Windows 11. It should work on Ubuntu too since it is based on Debian

Why is this only a "workaround"? What are the limitations/risks of doing this?

I just wanted to add a note about this workaround that was working fine in the past but now it seemed to stop working on my macbook M1.  I did the test with NET8. I was getting an error. It was complaining that it couldn't find the dotnet-root folder. So after some trail and error i noticed an issue with the folder structure. see the example below.
Example:
my-dotnet-directory
•libhostfxr.so

	•shared

		•shared

			•Microsoft.NETCore.App

				•8.0.1

					...files

As you can see there is a 'shared' directory within the 'shared' directory in my case this works now.
The runtimeconfig.json is still the same:
{"runtimeOptions": {"tfm": "net8.0", "framework": {"name": "Microsoft.NETCore.App", "version": "8.0.1"}}}
If anyone has a better solution to this. Please let us know.

authors: adwenn filmor rishky-msft sherryyshi vgibilmanno

repo: pythonnet/clr-loader