text:
Group together decoding options into a single argument

Is your feature request related to a problem? Please describe.
open_dataset() currently has a very long function signature. This makes it hard to keep track of everything it can do, and is particularly problematic for the authors of new backends (e.g., see https://github.com/pydata/xarray/pull/4477), which might need to know how to handle all these arguments.
Describe the solution you'd like
To simple the interface, I propose to group together all the decoding options into a new DecodingOptions class. I'm thinking something like:
from dataclasses import dataclass, field, asdict
from typing import Optional, List

@dataclass(frozen=True)
class DecodingOptions:
    mask: Optional[bool] = None
    scale: Optional[bool] = None
    datetime: Optional[bool] = None
    timedelta: Optional[bool] = None
    use_cftime: Optional[bool] = None
    concat_characters: Optional[bool] = None
    coords: Optional[bool] = None
    drop_variables: Optional[List[str]] = None

    @classmethods
    def disabled(cls):
        return cls(mask=False, scale=False, datetime=False, timedelta=False,
                  concat_characters=False, coords=False)

    def non_defaults(self):
        return {k: v for k, v in asdict(self).items() if v is not None}

    # add another method for creating default Variable Coder() objects,
    # e.g., those listed in encode_cf_variable()

The signature of open_dataset would then become:
def open_dataset(
    filename_or_obj,
    group=None,
    *
    engine=None,
    chunks=None,
    lock=None,
    cache=None,
    backend_kwargs=None,
    decode: Union[DecodingOptions, bool] = None,
    **deprecated_kwargs
):
    if decode is None:
        decode = DecodingOptions()
    if decode is False:
        decode = DecodingOptions.disabled()
    # handle deprecated_kwargs...
    ...

Question: are decode and DecodingOptions the right names? Maybe these should still include the name "CF", e.g., decode_cf and CFDecodingOptions, given that these are specific to CF conventions?
Note: the current signature is open_dataset(filename_or_obj, group=None, decode_cf=True, mask_and_scale=None, decode_times=True, autoclose=None, concat_characters=True, decode_coords=True, engine=None, chunks=None, lock=None, cache=None, drop_variables=None, backend_kwargs=None, use_cftime=None, decode_timedelta=None)
Usage with the new interface would look like xr.open_dataset(filename, decode=False) or xr.open_dataset(filename, decode=xr.DecodingOptions(mask=False, scale=False)).
This requires a little bit more typing than what we currently have, but it has a few advantages:

It's easier to understand the role of different arguments. Now there is a function with ~8 arguments and a class with ~8 arguments rather than a function with ~15 arguments.
It's easier to add new decoding arguments (e.g., for more advanced CF conventions), because they don't clutter the open_dataset interface. For example, I separated out mask and scale arguments, versus the current mask_and_scale argument.
If a new backend plugin for open_dataset() needs to handle every option supported by open_dataset(), this makes that task significantly easier. The only decoding options they need to worry about are non-default options that were explicitly set, i.e., those exposed by the non_defaults() method. If another decoding option wasn't explicitly set and isn't recognized by the backend, they can just ignore it.

Describe alternatives you've considered
For the overall approach:

We could keep the current design, with separate keyword arguments for decoding options, and just be very careful about passing around these arguments. This seems pretty painful for the backend refactor, though.
We could keep the current design only for the user facing open_dataset() interface, and then internally convert into the DecodingOptions() struct for passing to backend constructors. This would provide much needed flexibility for backend authors, but most users wouldn't benefit from the new interface. Perhaps this would make sense as an intermediate step?

Totally in favour of this. Option 2 does seem like a good intermediate step.
This proposal would make error handling easier (https://github.com/pydata/xarray/issues/3020)
I agree that we should add CF to the names. Can we use Decoders instead of DecodingOptions? So decode_cf and CFDecoders?

I agree that we should add CF to the names. Can we use Decoders instead of DecodingOptions? So decode_cf and CFDecoders?

works for me!

I agree, open_dataset() currently has a very long signature that should be changed.
The interface you proposed is obviously clearer, but a class could give a false idea that all backends support all the decoding options listed in the class. I see two other alternatives:

Instead of a class we could use a dictionary. Pros 1, 2 and 3 would still hold.
With the interface proposed by @alexamici in #4309 the pros 2 and 3 would still hold and partially 1 (since the open_dataset interface would be greatly simplified).

For both these proposals, we would lose the autocompletion with the tab but, on the other hand, the user would be relieved of managing a class.
Finally, I'm not sure that for the user it would be clear the separation between backend_kwargs and decode, since they both contain arguments that will be passed to the backend. Especially if the backend needs more specific decoding options that must be set in backend_kwargs. In this sense, #4309 seems less error-prone.

I see this as complementary to @alexamici's proposal in https://github.com/pydata/xarray/issues/4309#issuecomment-697952300, which I also like.  I guess the main difference is moving decode_cf into the explicit signature of open_dataset, rather than leaving it in **kwargs.
Auto-completion is one reason to prefer a class, but enforced error checking and consistency between backends in the data model are also good reasons. In particular, it is important that users get an error if they mis-spell an argument name, e.g., open_dataset(path, decode_times=False) vs open_dataset(path, decode_time=False). We can definitely achieve this with putting decoding options into a dict, too, but we would need to be carefully to always validate the set of dict keys.
I guess cfgrib is an example of a backend with its own CF decoding options? This is indeed a tricky design question. I don't know if it's possible to make xarray.open_dataset() directly extensible in this way -- this could still be a reason for a user to use a backend-specific cfgrib.open_dataset() function.

@shoyer I favour option 2. as a stable solution, possibly with all arguments moved to keyword-only ones:

users don't need to import and additional class
users get the argument completion onopen_dataset
xarray does validation and mangling in the class and passes to the backends only the non default values

I'm for using the words decode/decoding but I'm against the use of CF.
What backend will do is map the on-disk representation of the data (typically optimised for space) to the in-memory representation preferred by xarray (typically optimised of easy of use). This mapping is especially tricky for time-like variables.
CF is one possible way to specify the map, but it is not the only one. Both the GRIB format and all the spatial formats supported by GDAL/rasterio can encode rich data and decoding has (typically) nothing to do with the CF conventions.
My preferred meaning for the decode_-options is:

True: the backend attempts to map the data to the xarray natural data types (np.datetime64, np.float with mask and scale)
False: the backend attempts to return a representation of the data as close as possible to the on-disk one

Typically when a user asks the backend not to decode they intend to insepct the content of the data file to understand why something is not mapping in the expected way.
As an example: in the case of GRIB time-like values are represented as integers like 20190101, but cfgrib at the moment is forced to convert them into a fake CF representation before passing them to xarray, and when using decode_times=False a GRIB user is presented with something that has nothing to do with the on-disk representation.

Taking into account the comments in this issue and the calls, I would propose this solution: https://github.com/pydata/xarray/pull/4547

I had a similar idea in #9379 but forgot to check this issue didn't already exist 🤦♂ . A few points made there that weren't already made here:

I agree that we should add CF to the names.
I'm for using the words decode/decoding but I'm against the use of CF.

It is unclear to me now whether or not the role of the decoding step needs to be CF-specific.
Decoding according to some conventions is something that is enshrined for the Climate Forecast community, but definitely exists for other fields of science too. The ubiquity of netCDF means that it is not actually unusual to see e.g. plasma physics data with their own plasma-specific conventions stored in a netCDF file. Even the full definition of what variables are coordinate variables is mediated by the CF conventions, but other domains mights have different ideas about that. If you're opening Zarr then clearly you don't want to be constrained to follow the CF conventions.
As all these kwargs are passed on to the public xarray.decode_cf function, I suggested in #9379 that we instead make decode a Callable[AbstractDatastore | Dataset, Dataset], with the idea being that it would point to xr.decode_cf by default but be general beyond CF if a different callable is passed.
Though perhaps I'm missing the point here and we are conflating "decode data from format returned by backend to useful in-memory types" with "apply domain-specific metadata-dependent transformations". One could for example use a decode callable to construct flexible indexes based on a variable's metadata...

group together all the decoding options into a new DecodingOptions class.

I had suggested a dict, but a dedicated DecodingOptions class could also work, so long as it doesn't prevent the generality above. Note that since this issue was originally raised we now also have from_array_kwargs (a dict) in addition to backend_kwargs (a dict).

throwing in another idea: what if instead of boolean flags we change decode_cf to:
xr.decode(ds, decoders: Sequence[str | callable] = ["cf"], disable: list[str] = [])

where the order in decoders would dictate the order of application of decoders.
You'd use it like this:
xr.decode(ds, decoders=["cf", custom_decoder_func], disable=["mask", "scale", "datetime"])
xr.decode(ds, decoders=["mask", "scale", custom_decoder_func])

where "cf" is an alias for a list containing all the decoders we currently apply by default (another option would be to make xarray.coding.cf that list, and which allows filtering manually)

I'm not sure what my previous comment was referring to but the current coders are CF-specific and so should be exposed with CF in the name IMO. I agree with using xr.decode instead of xr.decode_cf.
Also some of these "coders" can have complicated options: Example: https://cf-xarray.readthedocs.io/en/latest/coding.html#compression-by-gathering could decode to multiindex or sparse. We could encode/decode multiindex/sparse to a number of Discrete Sampling Geometry specifications etc. So using dataclasses seems like a good idea here.
How about:
xr.decode(ds: Dataset, coders: Sequence[Coder] | None = CFCoders())

with a helper xr.coders.CFCoders(sane_defaults) -> Sequence[Coder] to construct a list of Coders and allow turning specific ones on/off.

the current coders are CF-specific and so should be exposed with CF in the name IMO

I think the point is just that if we are going to refactor this we should take the opportunity to generalize to allow supporting non-CF decoders too.

xr.decode(ds: Dataset, coders: Sequence[Coder] | None = CFCoders())

I like this, this is very neat. Then is the idea is that xr.open_dataset has a kwarg decode: Sequence[Coder] = xr.coders.CFCoders(sane_defaults)?

and would sane_defaults refer to options where you can enable / disable certain coders?

yes to be clear: sane_defaults expands to today's defaults
CFCoders(
     mask: Optional[bool] = None
    scale: Optional[bool] = None
    datetime: Optional[bool] = None
    timedelta: Optional[bool] = None
    use_cftime: Optional[bool] = None
    concat_characters: Optional[bool] = None
    coords: Optional[bool] = None
    drop_variables: Optional[List[str]] = None
    )

for example.
The wrinkle is that we might want to set different options for different backends: example, zarr can roundtrip datetime64 just fine without any encoding. Perhaps this is another reason to keep the separate open_zarr

:+1: for the suggestion
It seems a good idea to check the metadata if a certain coder is applicable or not.
Another thing, currently we do not differentiate mask and scale, it's mask_and_scale.
One possible reason is that we can't mask integer values without casting to float (at least atm, until there will be nullable integers).

zarr can roundtrip datetime64 just fine without any encoding.

Backends could, in theory, filter the coders they forward to store_entrypoint.open_dataset for standard coders that don't apply to them. Another option would be to just go ahead and decode, the decoders should skip variables that are already decoded. This wouldn't work for encoding, but that's not what we're talking about here, right?

Breathing new life into this.
Over in #9618 we are working on time decoding. @dcherian had the idea to split out CFDatetimeCoder to be able for users to parameterize it (deprecating use_cftime kwarg, adding time_unit kwarg later on).
I've split out this implementation from #9618 into #9901. As this adds new API, @pydata/xarray please have a review over there. Thanks!

authors: TomNicholas alexamici aurghs dcherian keewis kmuehlbauer shoyer


text:
auto does not work on Mac OS X

Adding the auto command line option does not work on Mac OS X (10.10.3). The task is run once, but then nothing occurs after that. I have played around with the source code and from what I can tell, the process that doit auto creates is not exited when the file system change event is received.
Here is a simple dodo.py file that shows this problem:
def task_test():
    return {
        'actions': ['echo "test" > test.txt'],
        'file_dep': ['watch.txt']
        }

If you run doit auto and modify watch.txt on Mac OS X, nothing happens.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/11805129-auto-does-not-work-on-mac-os-x?utm_campaign=plugin&utm_content=tracker%2F1885485&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F1885485&utm_medium=issues&utm_source=github).

Thanks for the report. The problem is not so simple.
Check filewatch.py, it contains the notifier loop for both linux and mac.  From what I remember it works in a different way than the linux one.
The linux version would start a new process after every notifcation but the Mac version doesnt.
I guess the difference came from the different API provided by the 2 systems.
One advantage of creating a new process is that it doesnt cache the python imports.
I did not implement the Mac version myself... and I dont use Mac at all. So it would be great if you can contribute a patch.

Sounds good. I will start working on it.

Or maybe we should just use https://github.com/gorakhargosh/watchdog
I didnt look at it but I hope that using watchdog we could could support linux/mac/bsd/windows easily.

I have a working version of doit that uses watchdog. I need to clean up the code before I submit a PR. Also, I only have access to Mac OS and Linux for testing. It would be great if we could test the new watchdog functionality on Windows and other OSes.

fixed on #135

source: gharchive/issue

repo: pydoit/doit

license: mit

license_type: permissive

license_source: bigquery


text:
feat: use sessionId cookie to have access to some endpoints

Follow up to #6
Validated locally with endpoints in https://enphase.com/download/iq-gateway-access-using-local-apis-or-local-ui-token-based-authentication-tech-brief

Not perfect, but a step forward
Pending:

sometimes the envoy return a 500 and retries are necessary (suggestions on aux lib for that?)

Not perfect, but a step forward
Pending:

sometimes the envoy return a 500 and retries are necessary (suggestions on aux lib for that?)

If its a simple 500 you could make a decorator to retry on 500s

source: gharchive/pull-request

authors: bdraco dgomes

repo: pyenphase/pyenphase

license: MIT

license_source: github-api


text:
Upload logs after the assignment has finished

This avoids a situation where the log has not been closed yet and might
still be written to while being uploaded.
As a side effect, _upload_logfile() now no longer takes a log
identifier, since an assignment is only supposed to have one log anyway.

Coverage increased (+0.06%) to 73.67% when pulling de473038fae1b0668e43155230b4cb8c0c3630fd on guidow:upload_log_after_close into 06fbf564eb85941c7c76327659643fd12b0ea959 on pyfarm:master.

LGTM, merging.

authors: coveralls guidow opalmer

repo: pyfarm/pyfarm-agent


text:
Documentation improvements

Done

[x] Fix that "Examples" heading was shown also when there was no mini-gallery to show.
[x] Show mini-gallery at the bottom of the page (instead of between class docstring and its methods/props).
[x] Fix that the mini-gallery showed no examples for certain groups of classes.
[x] Remove the need for setting target_name by making the canvas-selection a wee bit smarter.
[x] Animations behave much better w.r.t. fps.
[x] Use webp for screenshots too, because it's much smaller than png, even in lossless mode.

Proposal 1
While working on this I felt confused about when and how examples are shown in the gallery. There's the "/others" directory, there's the sphinx_gallery_pygx_xx comments ... and then some examples are run by the docs, without their screenshots being taken :/
So I'd like to propose we do this instead:

There are a few modes by which an example can appear in the gallery.
For this purpose the # example_gallery: comment must be present in each
example.

example_gallery: hidden - not present in the gallery.
example_gallery: code - present only as code (don't run the code when building docs).
example_gallery: screenshot - present in the gallery with a screenshot.
example_gallery: animate 3s - present in the gallery with an animation (length specified in seconds).

Simply a one-line comment, present in each example, that says how it behaves w.r.t. the gallery. Sphinx does not provide a builtin way to support this, but it's easy to leverage its config to make it work. I implemented it and applied it to a few examples to show it.
Proposal 2
Let's do the same for the behavior of examples w.r.t. testing. Because that's really an enum too. So instead of # run_example and # test_example, have one # example_testing that can be set to off, run or compare.
Each example must have this comment, so no wondering what the default was again. Not implemented yet (but easy to do).

I changed the comment to # sphinx_gallery_pygfx = 'xxx', because that's the standard that sphinx-gallery. Sphinx also "sees" the comment and tries to eval() the value ... but it does not seem to pass that value to the scraper?
I am a bit frustrated because my general feeling during this whole PR is that on the one hand sphinx-gallery is rather opinionated about how it should be used, but on the other hand that API feels incomplete / half-baked. 😕  I just wanted to simplify the way that examples are configured for the gallery, but it got me jumping hoops.

Aaaanyway ... I'm now on the fence about two options:
# Kinda what we have now. The "problem" is that the second comment is shown in the code in the gallery.
# sphinx_gallery_pygfx = 'screenshot'
# example_testing = compare'

import pygfx as gfx
...

vs
# If we re-use/mis-use the gallery option mechanism, sphinx will remove both comments from the gallery code.
# sphinx_gallery_pygfx_docs = 'screenshot'
# sphinx_gallery_pygfx_test = compare'

import pygfx as gfx
...

I honestly don't care. Either option is fine if we document it. People will just copy paste whatever we have anyway.

People will just copy paste whatever we have anyway.

yup! does anyone actually know how sphinx works fully :laughing:

We don't have a gallery there, so I guess not? Perhaps if/when wgpu-py gets some more examples ...

Well, the changes you made to the example testing stuff for example.

Yeah, I can apply that to wgpu too, for consistency.

authors: Korijn almarklein kushalkolar

repo: pygfx/pygfx


text:
YAML highlight

(Original issue 890 created by piuccio on 2013-07-25T15:44:33.467617+00:00)
Hello,
I've tried to style YAML files but I think the parser should be improved
The following code
resources: some text

generates
resources
resources
some text

It should be something like
resources
resources
some text

resources is a tag and some text is unquoted string

(Original issue 890 last updated on 2018-12-13T18:59:48.300516+00:00)
(Issue automaticaly closed due to status in Bitbucket: resolved)

(Original comment by piuccio on 2013-07-25T15:44:59.194081+00:00)
This project might help https://github.com/textmate/yaml.tmbundle

(Original comment by pathawks on 2014-11-17T17:46:00.184669+00:00)
Because YAMLLexer uses it's own tokens, existing styles do not work when using Pygments with YAML.

(Original comment by thockin on 2015-07-19T04:27:37.972172+00:00)
Is there no plan to make this work?
Is there at least a list of all the CSS ids that yaml code will use, so we can manually style it?

(Original comment by whitelynx on 2016-08-23T15:12:37.420128+00:00)
It seems to me this should actually be marked as a bug instead of an enhancement; as it is, using pygmentize -l yaml on the command line seems to highlight nothing but comments, which makes it next to useless. It seems that we should be able to use a dictionary lookup to map from YAMLLexer's tokens to the standard Pygments tokens.

(Original comment by jessetan on 2017-11-08T13:09:33.476490+00:00)
Mapping YAMLLexer tokens to Pygments tokens would also require changing the lexer; current YAML lexer marks almost everything that is not a comment or punctuation in the example.yaml as Literal.Scalar.Plain, which means that everything will be syntax highlighted in the same color.
Other syntax highlighters have lexers that seem to better at differentiating keys and values.

(Original comment by felixfontein on 2018-05-19T14:22:39.422349+00:00)
I tried to improve this in https://bitbucket.org/birkenfeld/pygments-main/pull-requests/762/

(Original comment by Anteru on 2018-12-13T18:59:42.305171+00:00)
Merged, thanks!

(Original comment by Anteru on 2018-12-13T18:59:48.331689+00:00)
Resolved in 98bdd9a1ebd2

repo: pygments/pygments-migration-test


text:
change style depending on the state

In the draw method I find it extremely difficult to figure out which (especially input) parameters still have to be defined, and I'm sure I'm not the only one feeling this way, but I'm not really sure what would be the best way to distinguish different states. In this example, only Add__y has no color, because it's not connected and no value is assigned:
from pyiron_workflow import Workflow, function_node

def add(x, y):
    output = x + y
    return output

def mul(x, y=2):
    output = x * y
    return output

wf = Workflow("draw")
wf.Add = function_node(add, x=1)
wf.Mul = function_node(mul, x=wf.Add)

wf.draw()

I find it pretty nice because you can immediately see that Add__y has to get a value or be connected. But this example is just a suggestion, which I don't consider to be the best possible product. So let me know what you think. @liamhuber @JNmpi @Tara-Lakshmipathy.

Pull Request Test Coverage Report for Build 10939959740
Details

0 of 0   changed or added relevant lines in 0 files are covered.
14 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.006%) to 91.27%

Files with Coverage Reduction
New Missed Lines
%

draw.py
14
92.45%

Totals

Change from base Build 10927133806:
-0.006%

Covered Lines:
2969

Relevant Lines:
3253

💛  - Coveralls

authors: coveralls samwaseda


text:
IndexError running example code

Heya
Was able to install everything grand. Only problem was running the example code
and also nose_tests
from pylayers.simul.link import *
DL = DLink()
DL.eval()
DL._show3()
File "test.py", line 2, in
DL = DLink()
File "build/bdist.linux-x86_64/egg/pylayers/simul/link.py", line 350, in init
File "build/bdist.linux-x86_64/egg/pylayers/gis/layout.py", line 420, in init
File "build/bdist.linux-x86_64/egg/pylayers/gis/layout.py", line 1498, in load
File "build/bdist.linux-x86_64/egg/pylayers/gis/layout.py", line 1408, in loadini
File "build/bdist.linux-x86_64/egg/pylayers/gis/layout.py", line 866, in g2npy
IndexError: too many indices
Any idea what could be the problem?

Hello Evan,
Thank you for rising the issue. It always a real pleasure to see some interest out there.
I am not able to reproduce the problem on my paltform.
What is exactly your platform. Try to pull request the latest version on the master branch.
Concerning the nose test I confess that I have some cleaning work on my plate. Some test corresponds to older version and are no longer elevant. I will try to have a clean testing procedure asap but it is not case at that time. The input data format has been changed recently and this could cause some problem. Always happy to try finding an issue in order to improve the pylayers platform.
Bernard

Thhis is the output you should get

authors: buguen eokeeffe

repo: pylayers/pylayers


text:
Some questions about adb install/adb shell pm

Not related to xinstaller, but I would like to ask if you know whether there is any method to prevent adb install/adb shell pm.
As adb takes effect at C level, I don't know how to hook it. Very thanks.

Hook InstallStage method and check Binder.getCallerID .. uid of shell is 2000 I think..

authors: aviraxp pylerSM

repo: pylerSM/XInstaller


text:
Add Danh to mentors section

Hi, this PR adds Danh to the mentors section. Thanks!

Thank you @danhphan.

authors: danhphan reshamas

repo: pymc-devs/pymc-data-umbrella


text:
Recalculate frontend from State

I've been using pynecone to develop a interface for some weeks now and must say I'm really enjoying the reduced complexity from dealing directly with react and chakra.
My current issue: I'm trying to present a text stream to pynecone. As the file gets updated/appended to, I need it to refresh automatically.
I've explored a several of possibilities of updating through a callback like: having the callback call a function that would normally cause the pc.var's to update; setting the state of the vars to dirty so that they would be eventually updated.
I've also tried to do it through an async yield loop to refresh the file but that seems to block the interaction with the rest of the page.
Is this currently possible to do with the framework? Is it planned to be supported? Is it possible to force an update through react?

Currently a limitation in Pynecone is that only one event handler can be running at a time, which is why the yield method is blocking the frontend. We need to add a feature to allow async background events to support live streaming like this.
We will make this a high priority feature to add soon as it enables many possibilities for realtime / live-updating apps.

repo: pynecone-io/pynecone


text:
Track data in parameters

Since we will move to SX, it becomes more interesting to store tracking data in fixed parameters instead of directly into the objective function.
One example of this is:
https://github.com/MMehrez/MPC-and-MHE-implementation-in-MATLAB-using-Casadi/blob/master/workshop_github/Codes_casadi_v3_4_5/MHE_code/MHE_Robot_PS_mul_shooting_v2.m

This is now done in Targets

repo: pyomeca/BiorbdOptim


text:
[BUG] setuptools 69.4.0 breaks mpi4py installation

setuptools version
69.4.0
Python version
all
OS
POSIX
Additional environment information
No response
Description
Originally reported at https://github.com/mpi4py/mpi4py/issues/484
The commit https://github.com/pypa/setuptools/commit/a131f83e2967514e2973fb36f2ca64e3ac8efc3c changed the return type of a publich method within a public class within a public module.
Expected behavior
pip install mpi4py should succeed with no traceback.
How to Reproduce
pip install mpi4py should be enough.
However, it cannot be reproduced easily as setuptools 69.4.0 was yanked for other reasons.
Output
File "/tmp/pip-build-env-qxqtzla9/overlay/lib/python3.10/site-packages/setuptools/_distutils/ccompiler.py", line 1232, in gen_lib_options
          lib_opts.extend(always_iterable(compiler.runtime_library_dir_option(dir)))
        File "/tmp/pip-install-u2viu3lx/mpi4py_70e5ecd7d41d4c9fb50ac4e92d5eb59f/conf/mpidistutils.py", line 68, in rpath_option
          if option.startswith('-R'):
      AttributeError: 'list' object has no attribute 'startswith'

changed the return type of a public method within a public class within a public module

Although Python conventions have become more rigorous in the past couple of decades, distutils is ancient and doesn't always conform to modern expectations about public/private visibility (FYI).
That said, you've also observed that this function previously did return strings or lists, and that was why in the review I didn't challenge the change of return type.

Is there any change setuptools would accept [a change back to a single string]?

Yes, maybe. My preference would be to return richer objects that more accurately reflect the semantic meaning of the operation. If --enable-new-dtags and -rpath are independent settings, I'd prefer for them to be specified independently and not bundled into a single string, especially if common usage is to supply them separately.
That said, I'd like not to be breaking our most popular downstream projects.
Here's what I propose:

patch distutils/setuptools to restore the prior expectation with a comment about why the change
with mpi4py and other affected projects, register and issue and add support for lists from this method
after a certain period (6-12 months?), start relying on a new expectation that lists might be returned

From mpi4py side, I'm working on fixing the issue and making a new release. Therefore, a fix in setuptools's side would not be strictly required, as long as users do not try to install old versions.
I'm not sure what's setuptools policy in this case. Does setuptools care about breaking install of old package versions if there is a new (patch, micro) version that fixes the install issue?

If --enable-new-dtags and -rpath are independent settings

They are indeed independent, but related. However, I would still point out that with the current implementation, the --enable-new-dtags will be inserted for every runtime path if more than one rpath is passed to the linker, while only one --enable-new-dtags option is needed, it applies to all rpaths. Of course, this (admittedly minor) nit comes originally from distutils, so maybe you would just prefer to ignore it.

Does setuptools care about breaking install of old package versions if there is a new (patch, micro) version that fixes the install issue?

Short answer is it depends, but generally speaking, Setuptools aims to support 99.9% of all packages currently in use. We want it to be rare for a user to have to pin to an older Setuptools to install an older package. First and foremost, we want users to feel comfortable and safe updating to the latest Setuptools version (so working at head/latest release is a viable option for most). That said, we'd also hope that users are also upgrading their other dependencies at the same cadence so that their use-case can quickly move into the future and not be reliant on older behaviors. How long do you think it would be before 99.9% of mpi4py users (that depend on builds from source) will have moved onto a new release with your fix?

They are indeed independent, but related. However, I would still point out that with the current implementation, the --enable-new-dtags will be inserted for every runtime path if more than one rpath is passed to the linker, while only one --enable-new-dtags option is needed, it applies to all rpaths. Of course, this (admittedly minor) nit comes originally from distutils, so maybe you would just prefer to ignore it.

I'm slightly inclined to just ignore it, but I'm also open to suggestions (or PRs) for improvement.

How long do you think it would be before 99.9% of mpi4py users (that depend on builds from source) will have moved onto a new release with your fix?

Well, it is really hard to tell. From PyPI download stats here https://www.pepy.tech/projects/mpi4py (I believe these are sdist tarballs from PyPI only), looks like there are still around 3K downloads per day for version 3.1.4 that was released in Nov 2022 (16 months ago). It is worth noting that user may not really hit the issue, unless they are building things such that passing a rpath is needed, which I believe is not the usual case (that's the reason the issue went unnoticed until today for a setuptools change from Sep 2023).

I'm slightly inclined to just ignore it, but I'm also open to suggestions (or PRs) for improvement.

I'll take a look at setuptools tomorrow. I think there are better places to inject the --enable-new-dtags flags if any rpath is being using. Then we can get back to the old interface of returning just a string for rpath, and we should be all set. Of course, this could break setuptools users specifically intercepting options to filter out --enable-new-dtags, although I don't think anyone would like the old rpath semantics.
To summarize, there are three options for you to choose.

Leave setuptools as it is now. A few mpi4py users may eventually experience trouble if using an old release, but they will have a new patch release to upgrade.
Make setuptools return the longish and ugly rpath string option "-Wl,--enable-new-dtags,-rpath," + dir, and keep it that way for (a couple?) years to come.
Make setuptools return a short and nice rpath string option "-Wl,-rpath," + dir, and inject the "-Wl,--enable-new-dtags options elsewhere, for example as pre/post link flags in the compiler object, or directly at the link stage within the  compiler.link_shared_object() method if any rpath is being passed in. As with any such changes, there is some slim possibility of breaking some other (likely obscure) use case, like filtering out the --enable-new-dtags flag.

I've presented you all the alternatives with pros and cons. I can contribute a PR for either (2) or (3). The decision on how to proceed is yours, just let me know.

In case it helps with testing, I ran into the problem today when running the CI tests on GitHub for a private repository. The CI tests run the mpi4py/setup-mpi@v1 GitHub action on ubuntu-latest, followed by the equivalent of python -m pip install mpi4py>=3.0.0.

In case it helps with testing

I keep getting reports about this failure. The issue also affected petsc4py and eventually slepc4py. Having a FOSS package in the PyPI top 1000 is not fun :facepalm: I'm starting to think that the best course of action is to start by implementing as setuptools fix as per option (2) in my comment above https://github.com/pypa/distutils/issues/246#issuecomment-2053731926.

(that's the reason the issue went unnoticed until today for a setuptools change from Sep 2023).

To be sure, the change landed in distutils in February and only was merged into Setuptools this week, so the issue is very recent.
#247 implements (2) currently with a retention window of 6 months. I'll expand that to a year for good measure. Good chance it lingers longer than that.
I'll leave (3) for later consideration.

This change is merged into Setuptools and releasing as v69.4.2 and v69.5.1.

authors: dalcinl dalleyg jaraco

repo: pypa/distutils


text:
Obscure error message "Invalid value for provides"

Flit fail when trying to upload a minimal package:
[project]
name = "epot-test"
description = "aaa"
readme = "README.md"
requires-python = ">=3.7"
authors = [{name = "Conchylicultor", email="etils@google.com"}]
dynamic = ["version"]

[build-system]
requires = ["flit_core >=3.5,<4"]
build-backend = "flit_core.buildapi"

Commands:
flit build
flit publish

flit publish fails with:
epot-macbookpro2:epot-test epot$ flit publish
Found 8 files tracked in git                                                                                I-flit.sdist
Built sdist: dist/epot-test-1.1.0.tar.gz                                                                  I-flit_core.sdist
Copying package file(s) from /var/folders/67/mbgvk_q96wd9hmdgm8z9ts4r00fsh4/T/tmpgkxoru5_/epot-test-1.1.0/epot-test  I-flit_core.wheel
Writing metadata files                                                                                I-flit_core.wheel
Writing the record of files                                                                               I-flit_core.wheel
Built wheel: dist/epot_test-1.1.0-py3-none-any.whl                                                        I-flit_core.wheel
Using repository at https://upload.pypi.org/legacy/                                                           I-flit.upload
Uploading dist/epot_test-1.1.0-py3-none-any.whl...                                                            I-flit.upload
Traceback (most recent call last):
  File "/Users/epot/.pyenv/versions/3.10.0/bin/flit", line 8, in <module>
    sys.exit(main())
  File "/Users/epot/.pyenv/versions/3.10.0/lib/python3.10/site-packages/flit/__init__.py", line 185, in main
    main(args.ini_file, repository, args.pypirc, formats=set(args.format or []),
  File "/Users/epot/.pyenv/versions/3.10.0/lib/python3.10/site-packages/flit/upload.py", line 268, in main
    do_upload(built.wheel.file, built.wheel.builder.metadata, pypirc_path, repo_name)
  File "/Users/epot/.pyenv/versions/3.10.0/lib/python3.10/site-packages/flit/upload.py", line 246, in do_upload
    upload_file(file, metadata, repo)
  File "/Users/epot/.pyenv/versions/3.10.0/lib/python3.10/site-packages/flit/upload.py", line 239, in upload_file
    resp.raise_for_status()
  File "/Users/epot/.pyenv/versions/3.10.0/lib/python3.10/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Invalid value for provides. Error: Use a valid Python identifier. for url: https://upload.pypi.org/legacy/

flit --version is Flit 3.7.1
It's very non obvious from the error message what is the issue with my toml file. It would be nice to have more explicit error messages.

Looks like that error message is coming directly from pypi.org as part of the 400 error, not flit

Renaming "epot-test" -> "epot_test" fixed the issue, but it's very non-obvious to guess from the error message.
It would be nice if flit was checking this (maybe during flit build rather than this error in flit publish).

authors: Conchylicultor bpabel

repo: pypa/flit


text:
Proposal: Explicit command to create a Pipfile in the current directory

Pipenv does not currently have an explicit “new project” command, but automatically ensures everything is in order when you run something. While this works well in general, it may leave people in the dark if the environment configuration is slightly off, as is the case of #2185.
#2185 is obviously a user error, and #2186 would help identify the problem, but the user, unless with substantial understanding to Pipenv internals, would still not be able to understand what action should be taken next.
A similar, but non-user error situation is to create nested Pipfile projects. This is suggested in #2148, which introduces the notion of “including” sub-projects. How this inclusion mechanism should be handled, if at all, is up to further discussion, but the idea of a Pipfile sub-project in itself is currently already problematic. You wouldn’t be able to create a sub-project if you create a parent project first, because all Pipenv commands will see the parent project, and refuse to create a new one.
The only reasonable way to work around this is to create the new Pipfile manually (e.g. touch Pipfile).  This is however still hackish, and not at all obvious unless you are knowledgable about Pipenv’s internals. With Pipenv encouraging user to let it handle Pipfile edits entirely (with pipenv install/uninstall [package] commands), it would also confuse users if it breaks consistency here.
The proposal is to provide an obvious way for user to initialise a virtual environment with a Pipfile created in the current directory, similar to npm init. This does not change the current initialisation behaviour, but only provide an additional way to initialise not only the virtual environment, but the Pipfile explicitly, right here. There are several possible ways to implement this:

An environment variable. This is probably not a good idea, since this needs to be obvious to be useful. Environment variabls are not obvious at all, at least to me. It would also conflict with PIPENV_PIPFILE.
An option to pipenv, e.g. pipenv --python=... --project=.. This is not as obvious as a top-level subcommand, but is still reasonably visible, and can be combined with other subcommands to perform tasks. It would still conflict with PIPENV_PIPFILE. What would PIPENV_PIPFILE=/path/a/Pipfile pipenv --python=... --project=/path/b even mean?
An extra sub-command, a la npm init. This is the most obvious solution, but given the history of reluctancy to adding new commands, I can understand this may not be the preferred way to go. I do prefer this the most, and it is the only option that does not conflict with PIPENV_PIPFILE. It can simply throw an error if a Pipfile already exists at the target location, and PIPENV_PIPFILE errors if the specifies location does not exist.

Pipenv init that only does touch Pipfile (and probably set the source section) would be nice !

nope. this was the original design, actually, and it was removed due to user feedback.
not changing.

Workaround of touch Pipfile worked great for me, thanks.

Workaround of touch Pipfile worked great for me, thanks.

authors: clayg gsemet kennethreitz uranusjr

repo: pypa/pipenv


text:
Two patched safety's

See https://github.com/pypa/pipenv/blob/master/pipenv/patched/safety.zip and https://github.com/pypa/pipenv/tree/master/pipenv/patched/safety
Is this intentional? Can the zip be removed?

The zip is used in core.py:do_check (https://github.com/pypa/pipenv/blob/master/pipenv/core.py#L2231), which I just found when looking for an issue which turned out to be in pyup. I've never seen a pattern like this, so I'd be really interested to see how this thread unfolds.

@techalchemy added the non-zip version in this commit and he updated the zip version the next day.
So presumably he holds the answers? Nobody else has committed changes to the non-zip version (in fact, almost nobody else has touched either).
This seems weird though. If both are needed, a simple readme in the pipenv/patched/ dir which explains this would be worthwhile.

@sirosen I doubt both are needed, but I was only recreating what kenneth had done (he created the zipped version but I believe we were bundling the entire safety database then);  I included the other one by accident most likely as a part of a commit at the time, I basically never touch it because the entire process of updating it was incredibly painful
I am mostly confident we can get rid of this but as with all things, time is a large driver of its continued presence

authors: benhowes hroncok sirosen techalchemy


text:
Updating MANIFEST.in does not correctly update the package sdist creates

Originally reported by: spookylukey (Bitbucket: spookylukey, GitHub: spookylukey)

The behaviour of sdist depends on previous contents of MANIFEST.in, not just the current. This is not fixed even by running setup.py clean or setup.py clean --all (although this should not be necessary).
This is very surprising behaviour, and potentially dangerous too - if someone accidentally adds a MANIFEST.in rule that includes a file that must not be distributed and notice the problem, they would expect that removing the rule will remove the file, but it does not.
I've attached a bash script that demonstrates the problem.

Bitbucket: https://bitbucket.org/pypa/setuptools/issue/436

I have been hit by this same bug. My current remedy is to delete the *.egg-info directory before running setup.py sdist.

The problema appears to be in the logic with which the SOURCES.txt file is updated by python setup.py egg_info:
$ python setup.py egg_info
running egg_info
writing sample.egg-info/PKG-INFO
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
writing requirements to sample.egg-info/requires.txt
writing top-level names to sample.egg-info/top_level.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'

Files already in SOURCES.txt are preserved across subsequent runs of python setup.py egg_info. Could please one of the developers clarify if this is intended behaviour and why?

For setuptools v34.3.0 the behaviour is different in Python 3.6.0 (this bug not present) and Python 2.7.13 (this bug present.)
In fact:
$ python2.7 setup.py egg_info
running egg_info
writing requirements to sample.egg-info/requires.txt
writing sample.egg-info/PKG-INFO
writing top-level names to sample.egg-info/top_level.txt
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'

but
$ python3.6 setup.py egg_info
running egg_info
writing sample.egg-info/PKG-INFO
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
writing requirements to sample.egg-info/requires.txt
writing top-level names to sample.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'

Note that under python2.7 there is a reading manifest file 'sample.egg-info/SOURCES.txt line (causing SOURCES.txt to save state from previous setup.py runs) which is missing for python3.6.

See also https://github.com/pypa/setuptools/blob/d4c215a7c61fb1f94b88bd2aa0b332ebaff18193/setuptools/command/egg_info.py#L560-L570
where the actual reading of SOURCES.txt takes place:
        rcfiles = list(walk_revctrl())
        if rcfiles:
            self.filelist.extend(rcfiles)
        elif os.path.exists(self.manifest):
            self.read_manifest()

I cannot understand the reason for this logic.

@jaraco I think I nailed down this old (annoying bug), but nobody cares of these old bugs. Should I reopen or provide a pull request?

@miccoli: If this ticket describes the issue, it's still open. A PR would be most appreciated.

@jaraco yes: this ticket is accurate. Since there is no discussion active, I assume that I should provide my own PR.

I think this change may have broken sdist installs with package data: #1016

The reason for
        rcfiles = list(walk_revctrl())
        if rcfiles:
            self.filelist.extend(rcfiles)
        elif os.path.exists(self.manifest):
            self.read_manifest()

is now clear: in the develop tree (under SCM control) the files to be installed are determined from the SCM system (walk_revctrl()); in an sdist install the files are to be determined from the existing manifest file, which was previously generated in the develop tree + setuptools_scm
This logic is broken for packages that do not use setuptools_scm, for which there is no guarantee that the current manifest file (in the development tree) is correct.
Unfortunately my PR #1014 badly brokes this intended behaviour, so that pip install of sdist packages under SCM control and include_package_data=True is not more possible, see #1016 .

Is there a fix to this now? Even if I delete <package>.egg_info, w hen I update MANIFEST.in and run python setup.py sdist, a new <package>.egg_info was created and the content of <package>.egg_info/SOURCES.txt was still not updated.

I really think SOURCES.txt should be cleared automatically at some point.  This is very surprising behavior.  I've been trying to figure out why the changes I was making to MANIFEST.in didn't seem to be working.

Hi I am running into the same issue, but I haven't quite figured out
how to get only the code in the package "src" folder be added to the final package.
@rsyring  could you help me with a bit more explanations? I don't have a MANIFEST.in file and I am not quite sure what steps I should follow to have python setup.py sdist create the package only with the desired files.
Thanks a lot for the help!

@lucacerone This issue is specific to a setup with MANIFEST.in: please first check the docs at https://packaging.python.org/guides/using-manifest-in/ and see if following those instruction you are able to obtain the desired source distribution.
The only thing that you should be aware, regarding this specific bug, is that after each change to MANIFEST.in you should delete the <package>.egg-info directory to be sure that the SOURCES.txt is regenerated appropriately.

Thanks a lot @miccoli, I managed to get the "data" in my package by adding a Manifest.in.
What I am a bit puzzled about is that from the documentation I understand that if I specify include_package_data=True then a Manifest.in isn't really needed, but that's not true in my experience.
Also, I don't quite understand what happens to "data" that resides outside of the "package" folder (I know it's not the preferred way, and in the end I moved the folder within the package), because if I add it to the Manifest.in, in the resulting archive I see those folders, but then they don't get copied to the site-packages folder (or any other folder I looked).
I think the documentation could be a bit more clear on what's going on under the hood, but in the end I found a way that works for me :)
Many thanks for your answer, I really appreciated it!

@miccoli I've also encountered this without a MANIFEST.in, see the repro case in https://github.com/pypa/setuptools/issues/2347

So annoying this bug... I tried to delete the .egg-info directory and I can not update the files included in my python package by updating the MANIFEST.in. Even if I delete the whole MANIFEST.in file, when I rerun "$ python setup.py sdist" the SOURCE.txt include all the old files I included in the MANIFEST.in.
Is not there any fix for this bug yet?

@MangelFdz do you have a reproduction? This does not seem to be the same problem described previously in this issue (for that problem things should work when you remove the .egg-info directory).

I have this issue. I have fallen back to just adding back in empty directories corresponding to those that were once in the MANIFEST.in
Not a great solution, but moving on I guess.

I can confirm that deleting the .egg_info directory helps. I thought I was going insane. It kept adding files even though I removed those from the manifest...

I have been hit by this same bug. My current remedy is to delete the *.egg-info directory before running setup.py sdist.

Thx! That really solved the problem.

I can confirm the issue still exists, even when building using the PEP 517 builder:

 draft @ cat > MANIFEST.in
include foo*.txt
include bar*.txt
 draft @ cat > pyproject.toml
[build-system]
requires=['setuptools']
build-backend='setuptools.build_meta'
 draft @ touch foo1.txt
 draft @ touch bar1.txt
 draft @ pyproject-build -s .
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: name

creating unknown-0.0.0
creating unknown-0.0.0/UNKNOWN.egg-info
copying files to unknown-0.0.0...
copying MANIFEST.in -> unknown-0.0.0
copying bar1.txt -> unknown-0.0.0
copying foo1.txt -> unknown-0.0.0
copying pyproject.toml -> unknown-0.0.0
copying UNKNOWN.egg-info/PKG-INFO -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/SOURCES.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/dependency_links.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/top_level.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/SOURCES.txt -> unknown-0.0.0/UNKNOWN.egg-info
Writing unknown-0.0.0/setup.cfg
Creating tar archive
removing 'unknown-0.0.0' (and everything under it)
Successfully built unknown-0.0.0.tar.gz
 draft @ rm -r dist
 draft @ cat > MANIFEST.in
include foo*.txt
 draft @ pyproject-build -s .
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: name

creating unknown-0.0.0
creating unknown-0.0.0/UNKNOWN.egg-info
copying files to unknown-0.0.0...
copying MANIFEST.in -> unknown-0.0.0
copying bar1.txt -> unknown-0.0.0
copying foo1.txt -> unknown-0.0.0
copying pyproject.toml -> unknown-0.0.0
copying UNKNOWN.egg-info/PKG-INFO -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/SOURCES.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/dependency_links.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/top_level.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/SOURCES.txt -> unknown-0.0.0/UNKNOWN.egg-info
Writing unknown-0.0.0/setup.cfg
Creating tar archive
removing 'unknown-0.0.0' (and everything under it)
Successfully built unknown-0.0.0.tar.gz
 draft @ ls -la
total 16
drwxr-xr-x   8 jaraco  staff   256 Jul 11 01:03 .
drwxr-x---+ 60 jaraco  staff  1920 Jul 10 22:21 ..
-rw-r--r--   1 jaraco  staff    17 Jul 11 01:03 MANIFEST.in
drwxr-xr-x   6 jaraco  staff   192 Jul 11 01:03 UNKNOWN.egg-info
-rw-r--r--   1 jaraco  staff     0 Jul 11 01:02 bar1.txt
drwxr-xr-x   3 jaraco  staff    96 Jul 11 01:03 dist
-rw-r--r--   1 jaraco  staff     0 Jul 11 01:02 foo1.txt
-rw-r--r--   1 jaraco  staff    77 Jul 11 01:02 pyproject.toml
 draft @ rm -r UNKNOWN.egg-info/
 draft @ pyproject-build -s .
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools
* Getting build dependencies for sdist...
running egg_info
creating UNKNOWN.egg-info
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md

running check
warning: check: missing required meta-data: name

creating unknown-0.0.0
creating unknown-0.0.0/UNKNOWN.egg-info
copying files to unknown-0.0.0...
copying MANIFEST.in -> unknown-0.0.0
copying foo1.txt -> unknown-0.0.0
copying pyproject.toml -> unknown-0.0.0
copying UNKNOWN.egg-info/PKG-INFO -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/SOURCES.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/dependency_links.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/top_level.txt -> unknown-0.0.0/UNKNOWN.egg-info
copying UNKNOWN.egg-info/SOURCES.txt -> unknown-0.0.0/UNKNOWN.egg-info
Writing unknown-0.0.0/setup.cfg
Creating tar archive
removing 'unknown-0.0.0' (and everything under it)
Successfully built unknown-0.0.0.tar.gz

I don't know why I thought that build would always build from a copy of the sources and not the actual directory.
I think what I'd like to see is for setuptools not to write an .egg-info at all when doing a build. I know it's unnecessary in general (not required by the specs or created by other backends).  I wonder if there are any setuptools-specific behaviors that depend on that egg-info directory (in the working directory or the sdist).

authors: Firenze11 MangelFdz Skylark0924 abravalheri ghost jaraco lucacerone miccoli microprediction mxmlnkn pjdelport rsyring tekumara

repo: pypa/setuptools


text:
When an entry point now fails to be loaded, it doesn't disrupt building process.

Summary of changes
Splitted from #2037
Pull Request Checklist

[ ] Changes have tests
[x] News fragment added in changelog.d. See documentation for details

It would be nice if the description above led me to the problem description that this issue tries to solve, rather than referencing another pull request.
The problem description is in #2030, and there are still outstanding questions about the problem that this solves. Therefore, I'm going to close this until the problem description can be refined a bit more.
Thanks for the continued effort.

authors: KOLANICH jaraco


text:
Error on pip search

I found a module on PyPI called profanity that can filter out swearwords. I tried to find it:
$ pip search profanity
Exception:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 595, in urlopen
    chunked=chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 352, in _make_request
    self._validate_conn(conn)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 831, in _validate_conn
    conn.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/connection.py", line 289, in connect
    ssl_version=resolved_ssl_version)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/ssl.py", line 362, in wrap_socket
    _context=self)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/ssl.py", line 580, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/ssl.py", line 807, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/connectionpool.py", line 621, in urlopen
    raise SSLError(e)
pip._vendor.requests.packages.urllib3.exceptions.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/commands/search.py", line 45, in run
    pypi_hits = self.search(query, options)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/commands/search.py", line 62, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/xmlrpc/client.py", line 1105, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/xmlrpc/client.py", line 1445, in __request
    verbose=self.__verbose
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/download.py", line 775, in request
    headers=headers, stream=True)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/sessions.py", line 522, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/pip/_vendor/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
pip._vendor.requests.exceptions.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:600)

Does anyone know why this error occurred, or how to get around it?

Apparently this happens with any given package.

What version of pip are you using? I suspect you need to update pip.

9.0.1

You'll need to upgrade to at least pip 9.0.2. See the pip issue and the original announcement

Oh, thanks. Sorry!

@jamadden, I tried to upgrade but got this:
$ pip install --upgrade pip
Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:600) - skipping
Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages

Yes, you can't use pip to upgrade pip because your pip is broken 😄 You'll need to install manually or download the pip wheel directly yourself using your web browser and then run something like python -m pip install <path to wheel>.

Ok, thanks!

repo: pypa/warehouse


text:
pyBullet installation - URL to MS link tools is not working

Describe the bug

I tried to install pybullet with command "pip install pybullet"
Among other things the command tries to build something, but it fails.
There is an error message
" error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools"
the bug: that http-URL does not work. Currently (September 2018) a working URL is
https://visualstudio.microsoft.com/visual-cpp-build-tools/
Expected behavior

typing the URL in web browser opens a web page where I can download MS build tools.
But it doesn't, probably because the URL is obsolete.
To Reproduce

My Platform

Windows7 64bit
Chrome browser
Additional context

Thank you for the report.
This repository is only for issues with the software that runs pypi.org itself. It cannot help with installing packages distributed there.
However, I did some checking, and the source of that error message appears to be setuptools. The link was changed in setuptools 40.1.0, so updating the version of setuptools installed should resolve this issue.


text:
redirect loop when uploading package named bleach_allowed_list followed by bleach-allowed-list

Describe the bug
redirect loop on the URL: https://pypi.org/project/bleach-allowed-list/
after uploading package named bleach_allowed_list followed by bleach-allowed-list
it's fixed now, because I deleted the bleach_allowed_list release.
Expected behavior
no redirect loop
To Reproduce
^
My Platform
N/A
Additional context

I tried it again here, but I can't repeat it :( https://pypi.org/project/bleach-allowlist/0.0.0a2/ https://pypi.org/project/bleach-allowlist/0.0.0a0/

Closing this as not reproducible (I couldn't duplicate it), feel free to reopen if you can find a way to reproduce.

authors: di graingert


text:
Ftrack sync can't handle recreated entity in different hierarchy

Description
First of all I didn't test it all this is just hypothesis. Happened after conversion from 2.11.8 - 2.17.3 maybe happened because of the huge step inbetween or bug was already fixed in past.
Seems like sync to avalon can't handle if asset is archived_asset, recreated in different hierarchy with same MongoID in Ftrack Custom attribute and synced back.
How to replicate (maybe)

create entity in ftrack
sync to avalon
delete entity using Delete Asset/Subsets action
sync to avalon
recreate entity under different parent
sync to avalon

Missing pieces

how MongoID from deleted(archived) asset got to new entity in ftrack?

Result

sync to avalon is trying to create new document with same MongoID like already existing document which cause conflict of _id key

[cuID:PYPE-1320]

Seems like the issue happened because Delete Asset/Subsets action didn't delete Ftrack entities so their custom attribute values were not removed.

We've since re-written the delete action multiple times and I don't think we've hit this in v3. I'm closing this as hypothetical.

authors: iLLiCiTiT mkolar

repo: pypeclub/OpenPype


text:
Tray action interface

Changes

extracted Launcher from AvalonModule
implemented ITrayAction which simplifies few modules that they don't have to be in folder

converted to actions: Standalone publisher, Settings and Launcher

Looking ok. but a bit fo API docs would be nice, so it doesn't get lost.

Looking ok. but a bit fo API docs would be nice, so it doesn't get lost.

Added README.md to modules directory with basic concepts.

Looking ok. but a bit fo API docs would be nice, so it doesn't get lost.

Added README.md to modules directory with basic concepts.

than you. now conflicts and we're done

repo: pypeclub/pype


text:
IPv6 Inspector

Can we add AAAA records please :). Happy to help if I can.
We only have legacy IP offered today:
[cooper@work ~]$ host inspector.pypi.io
inspector.pypi.io is an alias for inspector.cmh1.psfhosted.org.
inspector.cmh1.psfhosted.org has address 13.58.193.163
inspector.cmh1.psfhosted.org has address 18.217.27.127
inspector.cmh1.psfhosted.org has address 3.13.211.4

Long-term we should put this behind a CDN which provides IPV6. Adding @ewdurbin to see if there's something we can do in the short term.

If those IPs are direct to gunicorn for now all we should need is adding IPv6 addressing to the VPS instances + starting gunicorn with IPv6:
gunicorn -b '[::]:XXXX' - Where XXXX is the port

Linux will need /proc/sys/net/ipv6/bindv6only set to 0 to ensure dual stack binding

But I expect you probably have some kind of load balancer / reverse proxy in front of it ...

authors: cooperlees di

repo: pypi/inspector


text:
"Uncaught exception" warning on bad carbon data

I'm not sure whether this causes any problems other than the warning in the log, so I'm writing it up. The log spam is a bit excessive regardless :)
WARN [2016-04-05 16:21:01,363] nioEventLoopGroup-2-2 - io.netty.channel.DefaultChannelPipeline An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually m
eans the last handler in the pipeline did not handle the exception.
clojure.lang.ExceptionInfo: invalid carbon line: NaN metric for path:collectd.hostname-redacted.memcached.ps_count.processes
        at clojure.core$ex_info.invokeStatic(core.clj:4617) ~[cyanite.jar:na]
        at io.cyanite.input.carbon$parse_line.invokeStatic(carbon.clj:14) ~[cyanite.jar:na]
        at io.cyanite.input.carbon$pipeline$fn__4261.invoke(carbon.clj:40) ~[cyanite.jar:na]
        at io.cyanite.input.carbon.proxy$io.netty.channel.ChannelInboundHandlerAdapter$ff19274a.channelRead(Unknown Source) ~[cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307) [cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293) [cyanite.jar:na]
        at io.netty.handler.timeout.ReadTimeoutHandler.channelRead(ReadTimeoutHandler.java:152) ~[cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307) [cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293) [cyanite.jar:na]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307) [cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293) [cyanite.jar:na]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276) [cyanite.jar:na]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:354) [cyanite.jar:na]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244) [cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307) [cyanite.jar:na]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:293) [cyanite.jar:na]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:840) [cyanite.jar:na]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [cyanite.jar:na]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [cyanite.jar:na]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [cyanite.jar:na]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [cyanite.jar:na]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [cyanite.jar:na]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) [cyanite.jar:na]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [cyanite.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]

running 0b855b24acc7e9de85d0c6d3f3692183c03ea926 (master as of the time this issue was created).

Looks like this is a dup of #144 / #151, reopen if I'm wrong.

repo: pyr/cyanite

license: isc


text:
add evaluate code

In addressing #1, the R code is so slow and hard to understand (for me), so I need a review to double-check the math of my implementation. I'm pretty sure we can get expected weights for each time series from M5Data.get_aggregated_ma_dollar_sales() method. So I only need to review that given weight, wrmsse are wspl are computed as expected.
There are some issues which I will comment below.
TODO

[x] add a smoke test

@fritzo Things look good to me. I'm going to construct a model now.


text:
clean/create more coherent and uniform variable names

Naming conventions are as follows:

arcs and vertices for the full network object

edges and nodes for the simplified graph-theoretic object

[x] vertex_coords, vertex_list, vertices

[x] arcs, arc_lengths, w_network

[x] edges, edge_lengths, w_graph

[x] network_n_components, network_component2arc , network_component_labels, network_component_is_ring

[x] graph_n_components , graph_component2edge, graph_component_labels, graph_component_is_ring

Resolved in #181

repo: pysal/spaghetti


text:
Deprecate zope-interface support

It's a C-based dependency for a dozen of lines of code.
If there's interest, we can publish it as a separate package.

Why, that's quite a lot being deprecated in or after april 2024! How do you keep track?

I don’t. :D
Things get usually removed when you stumble over deprecated code while trying to do something else. I guess there’s not that much of a downside to remove deprecated code later.

Tahoe-LAFS is using this, but I'm not sure it's worth your time to package this separately—we could just copy/paste the relevant code into the codebase with appropriate license/attribution? Or I guess since we're using mypy-zope just rely on that.

authors: chrysle hynek itamarst

repo: python-attrs/attrs


text:
Add paste service utility

Docs:

changelog:

Enable the "Use a longer URI" option, although I don't see it in the API docs.
Return the deletion URL. This would allow us to cache it and potentially use it via a command if we find that a paste contains sensitive information.

I like both of these, added in a force push.

Good ol' reliable Chris.

repo: python-discord/bot-core


text:
"!d set" command gets no response from the bot

When I type a bot command !d set I expect to get documentation similar to the response for !d list or !d dict, but instead the bot doesn't respond at all.

This is because set is a subcommand for the !docs command. Since you do not have permissions to use it, the bot will not respond. If you want the docs for a set, use !d g set instead.

Is it feasible to make !d set without other arguments an alias to !d get set?

authors: Den4200 Mushinako anythingapplied

repo: python-discord/bot


text:
Add a doesntwork or similar tag

People tend to just send in help channels or similar such messages

Hey, can you help me? My code doesn't work.

Although we may have a code snippet from them, which may not be the source of the issue, there is a lot of valuable information that is lost. Useful information would contain things like

What is the user's end goal?
What is the current behaviour of the code? Is it working until some point or not starting at all?
What are some solutions the user has tried?
What is the traceback the user it getting?
...

Such information is lots of times critical to guide someone into fixing their issue or explain why their current code doesn't work.
I am very aware of the !traceback command but do not believe it does reflect exactly everything that is needed to fully help someone at the best and to prevent saying things they may have already tried

I think this would be quite similar to a recent tag we had about how to ask a question.
The problem was it became somewhat toxic and people started spamming it.
We have alot of new joiners in our community, so this is a scenario that we can expect quite often.
So it would be history repeating itself IMO.
I think it's better if we leave potential helpers ask the question themselves, it would also help us to explicitly identify people that show '' hostility '' towards beginners.

Ah, haven't seen that previous tag before, hence haven't seen the effect it had.
Don't want it to get repeated and get more toxicity than the toxicity there already is that can't be avoided.
Will close the issue for that reason and the one you've mentioned - thanks for your input :)


text:
Fix emoji replacement in UwU cog

Follow up to https://github.com/python-discord/sir-lancebot/pull/1596 , discord does not handle the raw emoji text so I replaced it with the actual emojis.
Fixes SIR-LANCEBOT-CF
Here is proof it works:

Did you:

[ ] Join the Python Discord Community?
[ ] Read all the comments in this template?
[ ] Ensure there is an issue open, or link relevant discord discussions?
[ ] Read and agree to the contributing guidelines?

@ChrisLovering

repo: python-discord/sir-lancebot


text:
Remove deprecated stuff: round one.

This is an initial removal of the deprecated code within hyper-h2. There is one more change we'd like to make: we'd like to make the default representation of headers bytes, not unicode. That will come along in a separate PR.

And rightly so. ;)

So, the twisted master failures are unfortunate. Naturally, twisted master fails because this is a breaking change. However, they pin out our new breaking version. Not sure what to do about that: should we temporarily move it to "allowed failures"?

So, the twisted master failures are unfortunate. Naturally, twisted master fails because this is a breaking change. However, they pin out our new breaking version. Not sure what to do about that: should we temporarily move it to "allowed failures"?

That feels a bit icky – especially as the whole point of this change is to fix things for twisted, it would be good to have some guarantee of that before releasing 3.0 and claiming it’s fixed.
I notice that our tox.ini installs twisted from a GitHub repo, not PyPI. We could fork twisted, bump the h2 version requirement in the fork and point tox at the fork – that allows us to check everything works correctly with our changes. Then when the version gets bumped in twisted master, we revert the tox change.
I haven’t had enough sleep to know if that’s a brilliant or a terrible idea.

Alternatively, it looks like the vast majority of twisted errors come from one line, where it uses deprecated parameters that’s being removed by this patch: https://github.com/twisted/twisted/blob/trunk/src/twisted/web/_http2.py#L125
We could just fix this on the twisted side to use H2Configuration?

I’ve opened https://github.com/twisted/twisted/pull/750 to fix this behaviour on twisted master. Once that’s merged, we should be able to re-run the twisted tests and (hopefully!) get a pass.

Cool, good idea. Handily, I'm a Twisted committer, so we can get this through pretty nicely. :grin:

Ok let's rebuild these.

Looks like there are some other problems in that Twisted code too. ;) @alexwlchan, are you interested in making a follow-up patch to Twisted? If not, I'll do it.

Looks like there are some other problems in that Twisted code too. ;) @alexwlchan, are you interested in making a follow-up patch to Twisted?

Yep, I’ll do it later.

See https://twistedmatrix.com/trac/ticket/9092#ticket

See https://github.com/twisted/twisted/pull/751 for what I think is the remaining twisted deprecations.

Okay, I’ve kicked off another round of twisted tests.

🍿

Looks like some stuff got missed. :smile:

Hurrah, looks good! Thanks for the work today @alexwlchan. :sparkles:

authors: Lukasa alexwlchan


text:
feature: InlineShape.image ?

Love your module.
I'm trying to add *.docx import to my python qualitative analysis tool, and python-docx has allowed me to bring content to a wxPython RichTextCtrl really easily.  I'm getting all the character and paragraph level formatting and all of the text, which has come together really quickly.
But I seem to be missing something in reading in images.  I can get the size and type of the images, but how to I get the IMAGE DATA to convert into an image object?

The API doesn't support this, you'll need to dig into the XML with lxml and some maybe some python-docx internals support if you want it bad enough :)
The general gist of the XML is here:
http://python-docx.readthedocs.org/en/latest/dev/analysis/features/shapes-inline.html#specimen-xml
and here:
http://python-docx.readthedocs.org/en/latest/dev/analysis/features/picture.html#specimen-xml
You can get a handle to the wp:inline element using InlineShape._inline:
https://github.com/python-openxml/python-docx/blob/master/docx/shape.py#L57
From there you can navigate to the pic:pic element using something like this:
_inline.graphic.graphicData.pic.blipFill.blip

From there you'll need to parse the embed link to get the relationship to the picture, which will be stored as a separate part.
These are just general guidelines, I don't have time unfortunately to get it down to working details, but should give you an idea of what's involved. I'm sure there must be ways the existing internals can help but I can't remember just now how it all works. You'll need to trace through the code a bit if it's worth the effort to you.
I would start that by tracing through how the .add_picture() bit works, you're basically looking to roughly reverse that.

Hi Steve,
I'm starting to explore the issue of loading images from docx files.
It's very interesting.
I've figured out inline shapes and getting the (internal) image name and
ResourceID values.
I've explored Relationships and have figured out how to link the
ResourceID with a physical file location in the docx archive file.
I've been exploring the underbelly of runs, and have now determined a
way to tell if a run is associated with an image using the run's
element.tags.  But I can't see how you tell WHAT image a run is
associated with.  I can't find any reference to ResourceIDs, internal
image name, relationships, nothing.  How are inline_shapes and runs
linked together so I know which inline shape is associated with whatever
run I find that has an image element?  It seems like the data has to be
there somewhere, or you would lose your images if you loaded, then
re-saved your document (and you don't).  I just can't seem to find it.
Any suggestions would be welcome.
David
On 01/26/2016 12:28 AM, Steve Canny wrote:

The API doesn't support this, you'll need to dig into the XML with
lxml and some maybe some python-docx internals support if you want it
bad enough :)
The general gist of the XML is here:
http://python-docx.readthedocs.org/en/latest/dev/analysis/features/shapes-inline.html#specimen-xml
and here:
http://python-docx.readthedocs.org/en/latest/dev/analysis/features/picture.html#specimen-xml
You can get a handle to the wp:inline element using InlineShape._inline:
https://github.com/python-openxml/python-docx/blob/master/docx/shape.py#L57
From there you can navigate to the pic:pic element using something
like this:
|_inline.graphic.graphicData.pic.blipFill.blip |
From there you'll need to parse the embed link to get the relationship
to the picture, which will be stored as a separate part.
These are just general guidelines, I don't have time unfortunately to
get it down to working details, but should give you an idea of what's
involved. I'm sure there must be ways the existing internals can help
but I can't remember just now how it all works. You'll need to trace
through the code a bit if it's worth the effort to you.
I would start that by tracing through how the .add_picture() bit
works, you're basically looking to roughly reverse that.
—
Reply to this email directly or view it on GitHub
https://github.com/python-openxml/python-docx/issues/249#issuecomment-174852781.

--
David K. Woods, Ph.D.
Researcher, Lead Transana Developer
Wisconsin Center for Education Research
University of Wisconsin, Madison
http://www.transana.org

Most of the grunt work with images is taken care of in the python-docx internals. You should be able to mostly leverage that for what you need, definitely all the bits about looking up image blobs from relationship ids and so on.
The first key thing you need is the so-called rId (relationship id). This comes from the a:blip element in the pic:pic element in the InlineShape object somewhere. I expect you've located that already:
http://python-docx.readthedocs.org/en/latest/dev/analysis/features/shapes/picture.html
With the rId in hand (something like 'rId5'), you can get the "related part", which will be an image part in this case:
document_part = document.part

# OR (if inline_shapes is already handy)
document_part = document.inline_shapes.part

# Then lookup the image part by rId
rId = however_you_get_the_rId_from_the_inline_shape()
image_part = document_part.related_parts[rId]

# docx.parts.image.ImagePart has some useful bits
filename = image_part.filename

# and also provides access to a docx.image.image.Image object which has even more goodies
image = image_part.image
bytes_of_image = image.blob
... and a bunch of other bits like dimensions, filename, content type, extension, etc.

Let me know if you need more once you've had a look at these :)

Hi David, what did you end up doing with this? Just wondering if a reasonably clear new API feature occurred to you after working this challenge that might be handy to have. Maybe InlineShape.image or something that retrieved an Image object for you?
Just sorting through the issue list here and wanted to encapsulate this one in the title if there was something you thought made sense.

Hi Steve,
I haven't done anything about it yet.  I want to get feature/tabstops
finished before I dive into this.  One thing at a time is probably best
right now.
David
On 04/21/2016 11:07 PM, Steve Canny wrote:

Hi David, what did you end up doing with this? Just wondering if a
reasonably clear new API feature occurred to you after working this
challenge that might be handy to have. Maybe |InlineShape.image| or
something that retrieved an Image object for you?
Just sorting through the issue list here and wanted to encapsulate
this one in the title if there was something you thought made sense.
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/python-openxml/python-docx/issues/249#issuecomment-213241474

Hi Steve,
Hi David
I am struggling at the moment with the same issue as David. In particular, I need to copy tables, paragraphs, and images from one docx to another docx. To do so, the item_block_items method outlined in https://github.com/python-openxml/python-docx/issues/40 was of great help for me. However, this method only considers paragraphs and tables. Is there a way to extend this method to also consider inline shapes?
Best regards
Andy

Anything is possible for the diligent developer :)
But not something that anyone is working on at the moment as far as I know, if that's what you're asking :)

Hi Steve,
I am turning my attention to this problem again after a number of distractions such as making sure my kids can eat most days.  You know how that goes.
I've never been able to get a handle on a specific Inline object from a Run when reading a file or when using run.add_picture() rather than document.add_picture().
Document.add_picture() produces a CT_Inline object, and this allows access to everything that's needed.  However, run.add_picture() produces a CT_Run object with something in drawing_lst[0], but that something is not a CT_Inline object and I haven't been able to crack what that something is.  (It's a lxml.etree._Element object, but I can't figure out where to go from there.)  When reading a file, images are held in Runs, not CT_Inline objects no matter how they were created.
My impression is that the pieces are mostly there for reading files and seeing graphics within Runs, but the connections from one level of the XML to the next are getting lost on the oxml level at the w:drawing level.  I believe this because the CT_Drawing object referenced in docs/dev/analysis/features/text/run-content.rst and docs/dev/analysis/features/shapes/shapes-inline.rst does not appear to actually exist in the python_docx code.  CT_Inline exists, and all the objects needed down the line from there to get all the data needed from the XML exist, but without the CT_Drawing object, nothing appears to be accessible.
So I'm thinking that I need to add CT_Drawing to oxml/shape.py, and that if I do this right, linking it to CT_Inline correctly, then I'm just about where I need to be.  I'll be able to read the CT_Drawing object in the Run's drawing_lst to get the information I need to proceed.
Does that seem possible to you?  Does this approach make sense?  Let me know if you need more information to make sense of what I'm saying, beyond what's already in the existing features documentation.
David

Yes, that makes sense to me David. Document.inline_shapes uses XPath (a few calls down the call tree) to locate the wp:inline elements, so explicit navigation to the intervening w:drawing elements was never required before now. My policy has always been not to add oxml features until they were needed to pass a unit test; so that explains its absence.
I notice the MS API has Range.InlineShapes, with Range being a superset of our Run (which doesn't appear by itself in the MS API). So initially I'd be inclined to add an .inline_shapes property to Run. That would allow you to test for the presence of an image and also to access it. It would also give a nice round-trip acceptance testing capability for Run.add_picture.
In practice, I think one finds at most a single picture in a run. But the schema places no limitation on how many can appear in the same run. That's good for us I think; we won't need Run.add_picture() to keep track of whether it already has one.

@aschilling Have you fixed the problem? ' copy tables, paragraphs, and images from one docx to another docx'

authors: DKWoods aschilling saruagithub scanny

repo: python-openxml/python-docx


text:
Bump to 1.2.1

[x] Cythonize on python3 only (cython tried to process external python3 code)
[x] Fix execute_through_host
[x] implement keepalive work

Pull Request Test Coverage Report for Build 106

1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 99.687%

Totals

Change from base Build 105:
0.0%

Covered Lines:
955

Relevant Lines:
958

💛  - Coveralls

authors: coveralls penguinolog

repo: python-useful-helpers/exec-helpers


text:
🛑 User's Api is down

In f84e4ea, User's Api (https://users.roblox.com/) was down:

HTTP code: 404
Response time: 79 ms

Resolved: User's Api is back up in f601505 after 23 minutes.

repo: pythoniaweb/roblox-status


text:
🛑 Roblox site is down

In 6f514c6, Roblox site (https://www.roblox.com/) was down:

HTTP code: 503
Response time: 692 ms

Resolved: Roblox site is back up in eebc7e2 after 35 minutes.

repo: pythoniaweb/statusboom


text:
🛑 Asset's Delivery is down

In 6afc787, Asset's Delivery (https://assetdelivery.roblox.com/) was down:

HTTP code: 404
Response time: 102 ms

Resolved: Asset's Delivery is back up in cf9659e after 24 minutes.


text:
CFP Talks & Workshops

[x] Setup Junction

Timeline: By Feb end

[x] Open CFP
[x] Announce
[x] Close CFP
[x] Announce

All done. Schedule pending. Tracked on #16


text:
🐛 [Bug] interpolation doesn't support

Bug Description
Depending on a number of arguments, interpolation doesn't work.
    # return torch.nn.functional.interpolate(img, scale_factor=2.0)  #ok
    # return torch.nn.functional.interpolate(img, scale_factor=(2.0, 2.0))  #compilation error
    # return torch.nn.functional.interpolate(img, size=200)  #ok
    # return torch.nn.functional.interpolate(img, size=(200, 200))  #compilation error

To Reproduce
$ python3 test_torchrt.py
Traceback (most recent call last):
File "test_torchrt.py", line 30, in
jit_module = torch_tensorrt.ts.compile(jit_module, **compile_settings)
File "/hogehoge/lib/python3.8/site-packages/torch_tensorrt/ts/_compiler.py", line 113, in compile
compiled_cpp_mod = _C.compile_graph(module._c, _parse_compile_spec(spec))
RuntimeError: Expected Tensor but got Uninitialized
import torch
import torch_tensorrt

class TestModule(torch.nn.Module):
    def __init__(self):
        super(TestModule, self).__init__()

    def forward(self, img: torch.Tensor) -> torch.Tensor:
        # return torch.nn.functional.interpolate(img, scale_factor=2.0)  #ok
        # return torch.nn.functional.interpolate(img, scale_factor=(2.0, 2.0))  #compilation error
        # return torch.nn.functional.interpolate(img, size=200)  #ok
        return torch.nn.functional.interpolate(img, size=(200, 200))  #compilation error

height = 100
width = 100
shape = [1, 3, height, width]
compile_settings = {
    "inputs": [torch_tensorrt.Input(
        shape=shape,
        dtype=torch.float,
    )],
    "enabled_precisions": {torch.float},
}

img = torch.rand(shape).to("cuda")

jit_module = torch.jit.script(TestModule())
jit_module = torch_tensorrt.ts.compile(jit_module, **compile_settings)
img = jit_module(img)

Environment

Torch-TensorRT Version (1.1.0):
PyTorch Version (1.11.0):
OS (Linux):
How you installed PyTorch (pip):
Python version: 3.8
CUDA version: 11.3
GPU models and configuration: RTX2070

mistakenly I created two tickets

repo: pytorch/TensorRT


text:
Update theme for 0.7 docs

Update the stable (0.7) docs for the newer theme, which reformats the top navbar

At a glance, the font sizes are different. Is something changed functionally?

The theme is pretty dynamic, so we make little updates to it all the time. Should be fine.

The theme is pretty dynamic, so we make little updates to it all the time. Should be fine.

Thanks for the clarification.

authors: brianjo mattip mthrok

repo: pytorch/audio


text:
Making sure we are picking correct release branch (#2489)

Summary:
Making sure we are picking correct release branch
Ref: https://github.com/pytorch/vision/pull/6168
Pull Request resolved: https://github.com/pytorch/audio/pull/2489
Reviewed By: mthrok
Differential Revision: D37160145
Pulled By: atalman
fbshipit-source-id: 3e4a2208cbe47f85147573159f9adb8d6a824956

There is a typo. https://github.com/pytorch/audio/blob/35ff7055df055c5f0bb56ba3301619768a910138/.github/workflows/build-m1-binaries.yml#L145

authors: atalman mthrok


text:
Signature of apply_sparse_mask method in MultiheadAttention

🐛 Bug
Additional context

In MultiheadAttention module, apply_sparse_mask method signature
https://github.com/pytorch/fairseq/blob/master/fairseq/modules/multihead_attention.py#L445
is invalid (but it still works). It is written like a function, not a class method.
I suggest fixing the signature from
def apply_sparse_mask(attn_weights, ...

to
def apply_sparse_mask(self, attn_weights, ...

and the line https://github.com/pytorch/fairseq/blob/master/fairseq/modules/multihead_attention.py#L320 from
attn_weights = MultiheadAttention.apply_sparse_mask(...)

to
attn_weights = self.apply_sparse_mask(...)

Why am I suggesting using an instance method instead of a class- or staticmethod?
I noticed that SparseMultiHeadAttention redefines this apply_sparse_mask method https://github.com/pytorch/fairseq/blob/master/fairseq/modules/sparse_multihead_attention.py#L101.
So it seems that def apply_sparse_mask(self, ...) signature would be more appropriate to use in MultiheadAttention.
P.S.
It is not quite a bug, but it is something not quite right.
P.P.S.
If the idea is OK, I could make a pullrequest.

Nice catch!  Definitely agree with your proposed fix, feel free to submit a PR, thanks!

Ok! Here https://github.com/pytorch/fairseq/pull/2587
but for some reason the tests did not run after submitting the pull request...

I think it might be because there was a lint error that got merged somehow.  It should be fixed now if you want to rebase.

:balloon: :smiley:

repo: pytorch/fairseq


text:
Improve the bounding boxes implementation

PR #2785 introduced a new utility to draw bounding boxes on images. The code changes were merged but there are a few nice-to-have improvements missing.
More specifically:

[ ] The unit-test test_draw_boxes() uses PIL instead of write_image() to save the image. We should use TorchVision's method instead.
[ ] The implementation of the draw_bounding_boxes() initializes the text font within the loop which is suboptimal. We should move it outside of the loop and initialize it once.

Looks really minor fixes @datumbox can I take this up?

@oke-aditya that would be great, looking forward to your PR. :)

Hi @oke-aditya and @datumbox
Sorry I did not read that you guys have concluded assignee on this one already.
I am starting new on open source contributions. This one looked really good for the "good first issue". So, I have created PR for this one.
You guys can decide which one to MR. @datumbox really thanks for accurate labels. This really helped me to understand the flow. Also, please have a look at the PR. Your review and feedback will help me a lot.
Thank you.

No worries @AdityaKhursale this is open source and all are welcome to do pr. You can go ahead.

No worries @AdityaKhursale this is open source and all are welcome to do pr. You can go ahead.

Thank you @oke-aditya. Really appreciate.

authors: AdityaKhursale datumbox oke-aditya

repo: pytorch/vision