text:
memory leak

The examples/harmonic_oscillator.py leaks memory. Is this a pervasive problem?

I ran memray on
memray run --trace-python-allocators examples/test_children_from_state.py
python3.10 -m memray flamegraph --leaks examples/memray-test_children_from_state.py.3910945.bin
And it says basically the whole render loop leaks tons of memory.

PySide6 has known leaks
https://bugreports.qt.io/browse/PYSIDE-2299

Nice writeup about Python leaks from the very capable maartenbreddels at widgetti https://github.com/widgetti/solara/discussions/172

Looking at the memray flamegraph more closely, I'm seeing a lot of memory retention in

d = dict(newprops._items)
self._props.update(props),
super().__setattr(k,v) in Element.__setattr__()
_RenderContext.set()

So basically all this leaking is happening in the mechanisms which David Deng put in to allow “rollback” of state changes if an exception is thrown. But we don't use any of that stuff for the use_state Hooks, and I think the whole idea of “rollback” of state after exceptions is a mistake anyway. If the user code throws exceptions into Edifice, there is no correct thing to do other than crash. So it looks like it's time to finally tear all that stuff out.

https://github.com/bloomberg/memray/discussions/570

Techniques for debugging Python memory from within a VS Code debugging session:
Get a reference to an object when you have the object id:
import ctypes
x = ctypes.cast(0x7fffc3533490, ctypes.py_object).value

Render a graph of backreferences to an object:
import objgraph
objgraph.show_backrefs([x])

Done in #129

repo: pyedifice/pyedifice


text:
Broadcasting in arithmetic operations

General

pyfar version: 0.4.2

Description
During the implementation of PR #277 I stumbled across the following problem:
The docstrings of the arithmetic operations state that combinations of array likes and audio objects are allowed. For N-D signals, this is not quite intuitive. For example, a channelwise amplification raises a broadcasting error
a = np.arange(2 * 3 * 4).reshape((2, 3, 4))
b = np.ones((2, 3, 4))
d = a * pf.signals.impulse(10, amplitude=b)

So, the internal broadcasting along the time/frequency dimension of the array is missing, while this works:
a = np.arange(2 * 3 * 4).reshape((2, 3, 4, 1))
b = np.ones((2, 3, 4))
d = a * pf.signals.impulse(10, amplitude=b)

And even this
a = np.arange(2 * 3 * 4 * 6).reshape((2, 3, 4, 6))
b = np.ones((2, 3, 4))
d = a * pf.signals.impulse(10, amplitude=b)

ah, ok. Then adding an axis b = b[..., np.newaxis] before the internal multiplication might already solve the issue (maybe with an if case, not sure)

Note: the examples above only work on the feature/matrix_multiplication branch, where an
# indicate use of _Audio arithmetic operations (e.g. __rmatmul__)
__array_priority__ = 1.0

was added to the audio class.

Note: the examples above only work on the feature/matrix_multiplication branch, where an
# indicate use of _Audio arithmetic operations (e.g. __rmatmul__)
__array_priority__ = 1.0

was added to the audio class.

If that's a separate commit, you could try cherry-picking it.

source: gharchive/issue

authors: f-brinkmann mberz sikersten

repo: pyfar/pyfar

license: MIT

license_type: permissive

license_source: github-api


text:
Move filter classes into separate file

Separate the filter objects form the coefficient calculating generating functions.

Only one naming suggestion - ready to merge from my point of view...

👍

source: gharchive/pull-request


text:
Font loading can be incredibly slow and cause errors in console

Booting my app begins with a 10 second lag spike. and a few errors in console of the form:
Exception ignored on calling ctypes callback function: <function _enum_font_names at 0x0000023342CB1750>
Traceback (most recent call last):
  File "C:\Users\digid\.conda\envs\Charm-Tests\lib\site-packages\pyglet\font\win32query.py", line 343, in _enum_font_names
    name = lf.lfFaceName.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x99 in position 8: invalid start byte)

It only seems to happen once a call to something that uses fonts is made (e.g.: draw_text() from Arcade.)
It seems to happen because pyglet's _enum_font_names() is called on every font on the system.
After modifying pyglet's _enum_font_names() to count how many times it's called, it's returning... 5369. I may have a lot of fonts. 😅
The errors I were getting were from pyglet.font.win32query line 343, and were because of these fonts:
=== FONTS CONTAINING \x99 (™) ===
b'Abaddon\x99'
b'Mephisto\x99'
=== FONTS CONTAINING \xa9 (©) ===
b'Key Extended \xa9 SD'
b'Key Normal \xa9 SD'

Do you have about 5000 fonts installed? We're using EnumFontFamiliesExA, which as far as I can tell is going to call _enum_font_names (or whatever callback is assigned) for each font it finds on the system. I'm not sure if there is a way that the performance could be improved or not. It would be interesting to make a minimal test with just this code segment, and get a baseline of the performance.
You might try the new DirectWrite backed font loader, by setting pyglet.options["advanced_font_features"] = True after pyglet is imported. This is not as widely tested yet, but should work fine on modern Windows.

For context, I have 813 fonts installed on my system. I've been collecting fonts for over a decade 😅
But I wouldn't want a fellow font-collector like me to have this issue, either! I'm glad I ran into it.
Just an off-the-top idea I had to maybe prevent this issue is to have an option to have it only cache fonts the user specifies (so you'd tell it what fonts you plan to use in the app), which could cut down on this a lot.
If that's not possible, of course, my only concern would be that turning on pyglet.options["advanced_font_features"] = True may break compatibility with non-Windows platforms, based on the way you worded it?
This also doesn't deal with the error of the characters "™" or "©" causing errors in console when they exist on the system.

Switching to the advanced font features did seem to significantly increase load times!|
However, the fonts with the "™" or "©" characters in the name still cause errors in the console, even with that feature enabled.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 11: invalid start byte
Exception ignored on calling ctypes callback function: <function _enum_font_names at 0x000002746BB35750>
Traceback (most recent call last):
  File "C:\Users\digid\.conda\envs\Charm-Tests\lib\site-packages\pyglet\font\win32query.py", line 343, in _enum_font_names
    name = lf.lfFaceName.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x99 in position 8: invalid start byte

That font is b'Key Normal \xa9 SD', or rather, "Key Normal © SD". It seems it's not a fan of either the copyright or trademark symbol being in the font name.
You can freely download that font here: https://fonts2u.com/key-normal-sd.font

Aren't those characters invalid for the font name? I recall reading that the name should be the ascii subset of Unicode8, but I could be wrong. In any case, this doesn't even appear to be valid unicode :thinking:
On my Linux machine's system font viewer it gets replaced with a "?" mark, so it becomes "Key Normal ? SD".
This also would indicate that the characters are invalid.
I suppose we could do something similar in pyglet.

Pretty sure win32query is just used for GDI+ So the advanced font features shouldn't even need to load that file.

#584 Should resolve this issue and prevent from crashing.
Tested using the font mentioned:
font = pyglet.font.load("Key Normal © SD")

print(font, font.name, font.have_font("Key Normal © SD"))

results: <pyglet.font.win32.GDIPlusFont object at 0x02094B10> Key Normal © SD True
Also in posters last update they mention this line:
`` File "C:\Users\digid.conda\envs\Charm-Tests\lib\site-packages\pyglet\font\win32query.py", line 343, in _enum_font_names
name = lf.lfFaceName.decode('utf-8')```
Which indicates they never did switch to using DirectWrite. Either way this should be resolved in both font renderers.
(Also note, it seems GDI cannot render this Key Normal font, only DirectWrite does)

Which indicates they never did switch to using DirectWrite.

I did turn on pyglet.options["advanced_font_features"] = True, which did make a noticable change in load times (indicating it was in fact doing something,) but it still seemed to run that line.

Arcade do curretly force GDI in 2.6. The idea was to swap to directwrite in 2.7

Closing as now the original issue of unicode font names has been resolved.

authors: DigiDuncan benmoran56 caffeinepills einarf

repo: pyglet/pyglet


text:
Some fixes

Now is_item_deactivated returns the actual value instead of the function callable
Implemented io.fonts.texture_desired_width

The texture_desired_width can be essential when importing many and large fonts, because the default max of 4096 can run out quite quick and will easily cause GL_INVALID_VALUE errors when refreshing the font texture due to the ridiculous resulting texture height. I don't know why it isn't done by default (might be something to look into implementing in the integrations?), but I've found that now setting:
io.fonts.texture_desired_width = gl.glGetIntegerv(gl.GL_MAX_TEXTURE_SIZE)

allows me to get the most out of what I can work with. In my case it was 8192, double the default 4096, and allowed me to scale my interface to 4x+ when without it only went up to 1.7x before crashing.

This looks good to me, thanks!

authors: KinoxKlark Willy-JL

repo: pyimgui/pyimgui


text:
Add PyMC calendar

As we're opening our meetings to the public, we should add our calendar on the website: https://calendar.google.com/calendar/u/2?cid=cHltYy5kZXZzQGdtYWlsLmNvbQ

It looks like right now, the permissions for the calendar need to be updated so it is publicly viewable.
@OriolAbril suggested putting a pointer to the calendar here: http://docs.pymc.io/
I wonder if we had a menu item called "Community" and then sub-items in this way:
Community

Calendar
Blog

cc: @cluhmann

Can this issue be closed?
Calendar appears here:
https://www.pymc.io/community/calendar.html

authors: reshamas twiecki


text:
Hot loading asset folder on development

All Submissions:

[x] Have you followed the guidelines stated in CONTRIBUTING.md file?
[x] Have you checked to ensure there aren't any other open Pull Requests for the desired changed?

Type of change
Please delete options that are not relevant.

[x] New feature (non-breaking change which adds functionality)

New Feature Submission:

[x] Does your submission pass the tests?
[x] Have you linted your code locally prior to submission?

Description
Add feature for watching assets folder changes, and do same changes on .web/public folder.
To work this just execute pc run with development, but not production.
Close #540.

This also needs to be rebased since we just merged a PR that updated the dependencies.

Ok, I will.

Fix reviews.

repo: pynecone-io/pynecone


text:
[FR] pkg_resources.parse_requirements: supporting nested requirements.txt

What's the problem this feature will solve?
I parse requirements.txt with pkg_resources.parse_requirements, and this has worked great in the normal case.
It's also possible to have nested requirements.txt:
-r another-requirements.txt
package==1.0.0

Currently, pkg_resources.parse_requirements doesn't support this, it throws an InvalidRequirement exception:
  File "/path/to/venv/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3084, in parse_requirements
    yield Requirement(line)
  File "/path/to/venv/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3094, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/path/to/venv/lib/python3.10/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 100, in __init__
    raise InvalidRequirement(
pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'-r requi'": Expected W:(abcd...)

Describe the solution you'd like
It would be nice to have pkg_resources.parse_requirements support reading in nested requirements files.
This would probably entail pkg_resources.parse_requirements recursively opening a nested requirements file, and parsing it, while checking if a line starts with "-r" to recurse.
Alternative Solutions
Here is an example showing how to accommodate one-layer of nesting.  A better solution would be recursively implemented, and have a max depth argument
with open("requirements.txt", encoding="utf-8") as outer_f:
    requirements: list[str] = []
    for req in outer_f.readlines():
        if req.startswith("-r"):
            with open(req.split("-r")[1].strip(), encoding="utf-8") as inner_f:
                requirements += inner_f.readlines()
        else:
            requirements += [req]
    install_requires = [
        str(requirement)
        for requirement in pkg_resources.parse_requirements(requirements)
    ]

Additional context
No response
Code of Conduct

[X] I agree to follow the PSF Code of Conduct

Hi @jamesbraza, sorry for getting back to you this late.
The main motivation behind pkg_resources.parse_requirements is to parse a string where each line corresponds to a dependency specification. This is not compatible with requirements.txt files, since this kind of files may contain pip specific logic (for example the -e flag), and therefore do not fit in the same mindset.
I personally don't think this kind of feature would fit nicely within the pkg_resources or setuptools packages (it seems to be a bit outside of the objectives of the project, considering the previous issues/discussions about requirements.txt).
Have you considered some other alternatives, like using pip-tools to expand the file into a proper flat list of requirements?

Hello @abravalheri thank you for the response!
Yes I am aware of pip-tools, unfortunately pip-tools currently doesn't support what I am looking to do.  Hence why I am DIY'ing stuff.  I was hoping to not re-invent a requirements parser, and came across pkg_resources.parse_requirements as a viable option.
Currently I do employ a DIY solution, so I am not blocked by this, it would just be nice to use a 3rd party library.

This is not compatible with requirements.txt files, since this kind of files may contain pip specific syntax (for example the -e flag), and therefore do not fit in the same mindset.

I was not aware of this, thank you for pointing out!
It sounds then like setuptools doesn't support this, and doesn't plan on it, so feel free to close this ticket out.

Let's close this issue for now. The main idea is that in the future pkg_resources will be replaced by more optimised tools like importlib.metadata and importlib.resources. Because of that, it is very unlikely that we want to augment the scope of pkg_resources.

authors: abravalheri jamesbraza

repo: pypa/setuptools

license: mit

license_source: bigquery


text:
[BUG] recent versions of setuptools broke editable mode

setuptools version
65.2
Python version
3.9.13
OS
Ubuntu
Additional environment information
No response
Description
Since some recent releases (likely 64, when PEP-660 was implemented), installing psycopg with -e results in a broken package, where psycopg is an empty namespace.
This is an example of broken run. This is a similar run whose difference is installing psycopg without the -e flag, in order to test a depending package.
Expected behavior
See above
How to Reproduce
A more complete reproduction with stable git reference can be found in https://github.com/tox-dev/tox/issues/2479.
Output
See above

Thank you very much @dvarrazzo for reporting this issue.
Could you please try to create a minimal/standalone reproducer (that does not involve using tox or pytest)?
I tried to do the following:
sudo apt update -y
sudo apt install -y libpq5 libpq-dev
cd /tmp && rm -rf /tmp/psycopg /tmp/empty_dir
git clone https://github.com/psycopg/psycopg.git /tmp/psycopg
cd /tmp/psycopg
python3.8 -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e ./psycopg
.venv/bin/python -m pip install -e ./psycopg_c
mkdir /tmp/empty_dir
cd /tmp/empty_dir  # <-- To avoid any problems with automatic injection of CWD into `sys.path`
cat <<EOS | /tmp/psycopg/.venv/bin/python -
import psycopg, psycopg_c
print(psycopg.Cursor)
print(psycopg.AsyncCursor)
print(dir(psycopg_c))
EOS
# Output ==>
# <class 'psycopg.Cursor'>
# <class 'psycopg.AsyncCursor'>
# ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_psycopg', 'pq', 'sys', 'version']

And I can see that the module members mentioned in the tox issue seem to exist.
Is there any chance this happens because Python automatically adds the current working dir to sys.path? (This way, the import system could mistake "/tmp/psycopg/psycopg" as the root of the pyscopg package). I tried to run the following example to investigate this theory:
# Continuation of the previous script
# still inside /tmp/psycopg
mkdir psycopg
echo "raise SystemError('kaboom!')" > psycopg/__init__.py
/tmp/psycopg/.venv/bin/python -c 'import psycopg'
# Output ==>
# Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
#  File "/tmp/empty_dir/psycopg/__init__.py", line 1, in <module>
#    raise SystemError('kaboom!')
# SystemError: kaboom!

I thought that, and in my investigation with tox I tried to chdir away, but it didn't fix the problem.
Anyway, does it mean that it is now mandatory to adopt a "src" style project organization?

Anyway, does it mean that it is now mandatory to adopt a "src" style project organization?

I don't think so...
In terms of "plain Python" usage", I would say things should not change a lot. The flat-layout should have the same flaws as before, right?
Specifically, if you have a folder under CWD with a name coinciding with the installed package, this folder might be imported (but that was already a well-known risk ...).
In terms of pytest things may change. I know that they try some magic to compensate the flat-layout flaws... I have not received any report about that yet (maybe this error is the first?).

The limitations of the implementation are documented here. So far the biggest impact is for old-style namespace packages using pkgutil and pkg_resources, but some workarounds are suggested in the docs (e.g., the strict mode should work).

I thought that, and in my investigation with tox I tried to chdir away, but it didn't fix the problem.

So do you confirm that the package itself works on an editable install, but things start to get weird when a 3rd tool is added to the mix? (Maybe tox, maybe pytest?)
Could it be the case one of these tools relies on a specific assumption about the editable installation layout? (I know that some static analysis tools will try to read a .egg-link file, which is gone in setuptools>=64...)

I can reproduce the problem with:
git clone git@github.com:psycopg/psycopg.git psycopg-test
cd psycopg-test
git checkout a7d39bf740d0d97dbe230d07177463668327431a

python3 -m venv .venv
source .venv/bin/activate
pip install "pip==22.2.2" "setuptools==65.2.0" "wheel==0.37.1"
pip install -e psycopg
python -c "import psycopg; psycopg.ProgrammingError" # fails
python -c "import psycopg; print(psycopg)"
<module 'psycopg' (namespace)>

changing cwd works. Using pip install psycopg works too:
python -c "import psycopg; print(psycopg)"
<module 'psycopg' from '/home/piro/dev/psycopg-test/.venv/lib/python3.8/site-packages/psycopg/__init__.py'>

Using a rather older set of libraries, editable mode works as expected:
pip install -U "setuptools==44.0.0" "pip==20.0.2" "wheel==0.37.1"
pip install -e psycopg
python -c "import psycopg; psycopg.ProgrammingError" # works
python -c "import psycopg; print(psycopg)"
# <module 'psycopg' from '/home/piro/dev/psycopg-test/psycopg/psycopg/__init__.py'>

Tested on my laptop using Python 3.8.10 (Ubuntu 20.04 package)

changing cwd works. Using pip install psycopg works too:

Thank you very much @dvarrazzo. So it is confirmed that it is happening because of the existence of the package CWD...
I can see that a regular installation does not have the same problem. If I had to guess I would say that Python starts by creating a namespace package, but then when it finds the other folder (the actual package) on sys.path and realises it contains a __init__.py file, it "regrets" the decision of having a namespace package and "collapses" it into a regular module.
This behaviour does not seem to be very well defined/coherent though...
In theory a folder in CWD should get priority during the import. We can do a series of experiments showing that if we don't mix namespace packages and traditional packages (the ones with __init__.py). I need to seek for some advice regarding this behaviour.
For the time being let's consider these "accidental namespace packages" as a limitation of the editable mode.

Ok, but what does it mean? Is this a regression that you will address (why would a random directory in cwd be more meaningful that an explicit entry in the pythonpath?) Or is this going to stay broken?
Should we pin a dependency on setuptools < 64 for development? Should we rename our code directory and make all git history operations more painful in order to restore editable mode?
Please let us know.

Ok, but what does it mean? Is this a regression that you will address (why would a random directory in cwd be more meaningful that an explicit entry in the pythonpath?) Or is this going to stay broken?

I am seeking for advice about this matter, before taking any final decision.

Why would a random directory in cwd be more meaningful that an explicit entry in the pythonpath?

This is how Python operates, right? Directories in CWD are meant to take precedence over other things in the site-packages. We can do the following test:
rm -rf /tmp/development
mkdir -p /tmp/development/pkgA/pkgA
cd /tmp/development

touch pkgA/pyproject.toml
echo "x = 42" > pkgA/pkgA/a.py

# --- Simulate regular installation ---
python3.8 -m venv .venv
cp -r pkgA/pkgA .venv/lib/python3.8/site-packages/
tree .venv/lib/python3.8/site-packages/pkgA
# .venv/lib/python3.8/site-packages/pkgA
# └── a.py

# --- Experiment ---

mkdir -p /tmp/workdir/pkgA
echo "raise ValueError('kaboom')" > /tmp/workdir/pkgA/a.py
cd /tmp/workdir
/tmp/development/.venv/bin/python -c 'from pkgA import a; print(a.x)'   # <------ error
# ValueError: kaboom

If we add a __init__.py file to both /tmp/development/pkgA/pkgA and /tmp/workdir/pkgA, the results would be the same: CWD would take precedence over site-packages...

Should we pin a dependency on setuptools < 64 for development? Should we rename our code directory and make all git history operations more painful in order to restore editable mode?

You don't need to rename your directory. Right now, you can try pip install -e . --editable-mode strict or pip install -e . --editable-mode compat, maybe that will work for you?

This is how Python operates, right? Directories in CWD are meant to take precedence over other things in the site-packages.

I don't think so, no. This is not how Python used to operate until a couple of weeks ago.
Giving precedence to directories in cwd seems a design error and a security problem. A program would change its behaviour according to the cwd and an attacker could convince someone to run a program from their directory to hijack it. It is the same reason why cwd is not in PATH: it's considered insecure since MS-DOS time.

You don't need to rename your directory. Right now, you can try pip install -e . --editable-mode strict or pip install -e . --editable-mode compat, maybe that will work for you?

I don't see such a thing:
(venv-new) piro@baloo:~/dev/psycopg3$ pip --version
pip 22.2.2 from /home/piro/dev/psycopg3/tmp/venv-new/lib/python3.8/site-packages/pip (python 3.8)
(venv-new) piro@baloo:~/dev/psycopg3$ pip install -e ./psycopg --editable-mode strict

Usage:

  ...

no such option: --editable-mode

Sorry for that. I never remember it by heart :/
These are the options, copied straight from the docs:
pip install -e . --config-settings editable_mode=strict

pip install -e . --config-settings editable_mode=compat

Hi @dvarrazzo , thank you very much for testing these solutions and confirming that they work.
The implementation details for both strict  mode and the default editable installation may vary. They should be equally reliable.
.pth files, import hooks, file links... Those are all different techniques that can be used under the hood. But having a public commitment on using specifically one of them brings no value to setuptools.
Indeed, the opposite is quite true: if we don't commit to a particular mechanism,  as Python and the ecosystem evolves, setuptools can select a different approach for the implementation, and potentially surpass existing limitations.
This should be no different from any functionality implemented in a Python project: the public API is the contract and the exact implementation details may vary. It does not make these functionality more or less reliable.

If you consider that "the default solution is broken", than you also have to consider that the legacy behaviour (python setup.py develop) is broken and that the behaviour in other backends are broken... They are all broken, in different ways.
Editable installs are complicated and all the existing implementation mechanisms in the ecosystem have severe limitations.
If you go around asking people what are their requirements for editable installs, you are going to realise that is technically impossible to meet all the requirements at the same time. You have to select a subset.
Right now setuptools offers two options for the user to select: the default + strict. They are equally reliable and they both satisfy just a subset of the requirements. But togheter they should cover the entire list.

@abravalheri I have reported a regression. Things were working previously and now they don't. I appreciate the hard work behind the hard problem of packaging, don't get me wrong. Regressions happen.
It is not true that a local directory used to have, or even has, precedence over installed packages:

this wasn't the case in the past
this is not the case with packages installed without -e.

Since a certain version, the default behaviour of editable packages changed and now presents a noticeable difference w.r.t. normally-installed packages, and present difficulties developing certain repositories layout (a monorepo where the directory name matches the package name, for instance - like psycopg 3 repository).
So, this is a regression. My questions are:

is this regression wanted? As in, is the current behaviour the wanted behaviour? If so please document it (but with a WARNING, not with obscure options which need a pass-through pip command to activate, hence don't even appear in pip --help) It would seem exceedingly bizarre to me. I invite you to look at the psycopg readme: we used to suggest to hack on the project using -e, whereas now we should explain setuptools internals.

is this regression unexpected? It sounds so to me, because other modes work differently (non-editable install, strict-editable, compat-editable). If so, is it expected to be fixed and can it become part of the spec/tests?

Thank you

I am not able to install https://github.com/tensorflow/tfx-bsl in editable mode.
Setting SETUPTOOLS_ENABLE_FEATURES="legacy-editable" fixed the issue.

Unfortunatelly no one in the community has demonstrated a bullet-proof mechanism to implement editable installs in such a way that it mimics 100% the regular installation. Instead we are left in a "pick you poison" situation, and the mechanism we have for doing that is via the --config-settings editable_mode=....

Just to cross link it, there are some limitations in how config-settings operate in pip: https://github.com/pypa/pip/issues/12310
What's at stake for me is the ability to use vscode/pylance with my editable repos:
https://github.com/microsoft/pylance-release/issues/3473
Per recommendation above, I tried:
export SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
pip install -e .

but I get the error:
ERROR: Project file:///home/a/b/c has a 'pyproject.toml' and its build backend is missing the 'build_editable' hook. Since it does not have a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode. Consider using a build backend that supports PEP 660.

based on some quick searching, I'm not clear if there's a way to get SETUPTOOLS_ENABLE_FEATURES to work for me...

Hi @qci-amos, please note the following caveat for SETUPTOOLS_ENABLE_FEATURES.

Finally, as long as pip allows it, you can use the environment variable SETUPTOOLS_ENABLE_FEATURES="legacy-editable" to fallback to access the previous implementation and be bug-by-bug compatible.

Pip no longer allows backends to not implement PEP 660. So that is no longer an option.

What's at stake for me is the ability to use vscode/pylance with my editable repos

I think the right question here is to ask how vscode and pylance allow you to pass customised --config-settings to pip.

I think the right question here is to ask how vscode and pylance allow you to pass customised --config-settings to pip.

Well to be clear, what I don't currently have a solution for is how to use --config-settings in the case of pip install -r. I've been having that discussion here: https://github.com/pypa/pip/issues/12310

shouldn't this just be an option:
mypackage = { path = "../mypackage/", develop = true, editable_mode=strict}
then poetry can pass that flag along when installing?
or even
mypackage = { path = "../mypackage/", develop = true, flags={editble_mode: strict} }
so we have full control over how packages are installed

shouldn't this just be an option:\n\nmypackage = { path = "../mypackage/", develop = true, editable_mode=strict}

Not really. This does not look like something setuptools would work with.
Setuptools is a build backed and does not directly work with dependencies.

Yes, it has a config settings flag.  Look at docs for the build command
On Sun, Mar 31, 2024, 10:37 PM Anderson Bravalheri @.***>
wrote:

shouldn't this just be an option:\n\nmypackage = { path = "../mypackage/",
develop = true, editable_mode=strict}
Not really. This does not look like something setuptools would work with.
Setuptools is a build backed and does not directly work with dependencies.
—
Reply to this email directly, view it on GitHub
https://github.com/pypa/setuptools/issues/3557#issuecomment-2029057644,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAMMUOEFZAPT6C6UEM4KR3Y3DB5RAVCNFSM57HCJ6T2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBSHEYDKNZWGQ2A
.
You are receiving this because you commented.Message ID:
@.***>

Yes, it has a config settings flag. Look at docs for the build command.

Even though you can pass a dict to setuptools via config_settings, the dict you are proposing doesn't make much sense to setuptools.
Setuptools handle the build of a single distribution as a whole. It does not make sense to express editable installations piecewise, package by package.
There is already package-dir for expressing explicit paths to packages. Reworking it to path in the way you are suggesting would require major incompatible changes. Finally, what is the difference between develop and editable, these 2 settings seem redundant.
If you want to pass editable-mode=strict via config settings, you already can do that in the current version of setuptools.

It seems like the regression comes from the "new-style" .pth files adding the _EditableFinders to the end of sys.meta_path, after the default FileFinder. When FileFinder finds the subdirectory, it creates the empty namespace even though it's not a real Python package. The old/compat mode would expose the paths directly to FileFinder, which correctly prioritizes those paths over a non-package in the working directory.
I think the current behavior is unintuitive and a bug. When the user requests an editable installation, it should take precedence before FileFinder as the single intended method of importing the package, not act as a fallback option that only runs after FileFinder fails. Adding _EditableFinder earlier in meta_path with e.g. sys.meta_path.insert(0,  would make it so editable installs aren't masked by local files, which is the behavior that probably all users expect and is also the safer option as mentioned above.

t should take precedence before PathFinder as the single intended method of importing the package,

Please note that PathEntryFinder is not the single intended method of importing a package in Python. The import machinery is specifically designed for extension and multiple finders. That said, we can try to put the editable finder before the PathEntryFinder, however we need to wait until pip adopts pyproject-hooks>=1.2 AND that version of pip to be prevalent in the ecosystem to avoid breaking things.

Please note that PathEntryFinder is not the single intended method of importing a package in Python. The import machinery is specifically designed for extension and multiple finders.

Yes, although it seems to me that the system was designed for adding finders/loaders for filetypes that PathFinder won't reach, it's not very flexible for adding a different way to add Python packages unfortunately.

That said, we can try to put the editable finder before the PathEntryFinder, however we need to wait until pip adopts pyproject-hooks>=1.2 AND that version of pip to be prevalent in the ecosystem to avoid breaking things.

Could you elaborate on this? I assume you're referencing https://github.com/pypa/pip/issues/11812 here. Does that bug occur with both sys.meta_path and sys.path_hooks modifications or could a version of editable packages that only goes through sys.path_hooks work?

Yes, although it seems to me that the system was designed for adding finders/loaders for filetypes that PathFinder won't reach, it's not very flexible for adding a different way to add Python packages unfortunately.

I have a different impression. It is well documented when it comes to loading Python packages, and for editable installs, it is mentioned in the PEP that standardised editable installs as one of the possible forms of implementations. However, I agree that it is not straightforward to finders or loaders for namespace packages... that is in my opinion an area that the import machinery is lacking.

I assume you're referencing pypa/pip#11812 here.

Yes, please see the details on that issue. I believe that both approaches would be subject to that problem.

authors: abravalheri alchzh dvarrazzo earonesty qci-amos tvalentyn


text:
Add timestamp-based dependency system to build_clib

build_clib is sorely missing one thing in distutils - checking outdated dependencies when compiling. This makes it so that, when building a static c library, all objects are always all compiled.
The function to create the static library itself does implement something of this sort, but since all objects have been recompiled it has no practical effect (everything still runs every time).
This is a small patch designed to fix that, with a couple of qol improvements along the way. Two new keys were added to the build_info dictionary:

obj_deps - Allows specifying extra dependencies for each source so they can be recognized as being newer than the object being compiled. Example:

setup(
...
libraries=[(
    'examplelib',
    {'sources': 'example.c',
     'obj_deps': {'': 'global.h',  # use an empty string to define global (for all sources) dependencies
                  'example.c': 'example.h'}
    }
)]
...
)

cflags - This just allows the user to provide an extra set of flags to the compiler.

That is pretty much it, a simple (but important) quality of life improvement for those of us who need to compile static libs with python.
If the implementation is fine I can get started on documentation, just not sure where to put it.

This seems reasonable, and I appreciate the effort put into making this first request a complete and thorough one.
I welcome subsequent improvements to the documentation. I also encourage you to see this implementation made the default in distutils (stdlib) with setuptools providing forward compatibility but eventually delegating that role back to distutils. I won't stipulate that as a requirement, however.

I'll definitely look into that, time to start learning mercurial :) Thanks!

authors: GandaG jaraco


text:
Link to public package security info on package detail pages

What's the problem this feature will solve?
Provide information about security vulnerabilities of PyPI packages.
Describe the solution you'd like
I suggest adding links on project detail pages to the packages' pages on the Snyk Advisor website. The most basic implementation would be to add a link in the sidebar, alongside the existing link to the libraries.io page.
Snyk Advisor pages include security information about packages, both summarized and in detail, as well as other useful info and stats.
For example: The package detail page for networkx would include an additional link to its Snyk Advisor page. This page currently prominently displays a red "SECURITY ISSUES FOUND" label for the latest version of networkx, with details (high severity!) and links to more information in the "Security" section of the page. (For future reference, here's an archived version from today on the Internet Archive Wayback Machine.)
Additional context
Packaging tools in some other programming ecosystems have begun providing security information about packages. One prominent example of this is npm, whose CLI tool began showing warnings for installed packages with known security vulnerabilities. The availability of tooling and information for Python packages is only recently becoming available, but awareness in the community is still low. Adding this to PyPI could help a great many projects and people discover and address security issues much earlier.
Full disclosure: I worked for a while developing Snyk Advisor as a contractor. I'm no longer affiliated with Snyk in any way, and have no direct or potential financial interest in their services being used or succeeding. Snyk did reach out to ask what I thought about this and whether I'd be willing to suggest this publicly, which I'm happy to do with no compensation of any sort, simply because I think it will benefit the community.

I'd definitely give it the thumbs up!
As a data engineering team lead I find myself many times emphasize to my team members, the importance of some background check of a python package, before they just throw it into our requirements.txt files.
Specifically, I instruct them to check in Snyk Advisor as part of some other examinations.
It is a great idea to put all in one place, an PyPi is exactly the right place for it!
👍👍👍

This looks like a great idea. We can all improve our security awareness, and this looks like a low effort-high reward way to do this.
👍

authors: ohz taleinat zach-beniash

repo: pypa/warehouse


text:
Settings UI use OpenPype styles

Goal
Setting UI should use OpenPype styles instead of having it's own.
Changes

local settings and studio settings use openpype's stylesheet
most of settings stylesheet copied to openpype stylesheet

not all are same

most of buttons in settings have real image instead of using characters

their size is based on font size

use new checkbox which has size defined by font size

few fixes in new checkbox

Note
If anything is so different that is' bad tell me in comments.

Task linked: OP-1775 Settings UI use openpype styles

I made a small comparison screen and have some comments

Modified icon images

Modified colors of checkbox

Current state. Left: Previous - Right: this PR

authors: iLLiCiTiT mkolar

repo: pypeclub/OpenPype


text:
503 (Service Unavailable) during test.pypi upload

Hi,
Describe the bug

I am trying to upload updates to my existing test.pypi project https://test.pypi.org/project/cytotrace2-py/1.0.2/, but keep getting "503 (Service Unavailable)" error.
Expected behavior

Previously I was able to push updates and got 200, OK response.
To Reproduce

I have run "twine check dist/*" and it passed, and also tried connecting to the server with "curl -I https://upload.pypi.org/legacy/" and received 200, OK.
To upload, I am running twine upload --repository-url https://test.pypi.org/legacy/ dist/*
My Platform

My OS is macOS Sequoia 15.1.1
Additional context

Thanks for your help!

Can you provide the output of the following?
twine upload --repository testpypi --verbose dist/*

Thanks for your response.
This is the output of the command:
`twine upload --repository testpypi --verbose dist/*
INFO     Using configuration from /Users/savagyan/.pypirc
Uploading distributions to https://test.pypi.org/legacy/
INFO     dist/cytotrace2_py-1.0.3-py3-none-any.whl (82.9 MB)
INFO     dist/cytotrace2_py-1.0.3.tar.gz (82.8 MB)
INFO     username set by command options
INFO     password set from config file
INFO     username: token
INFO     password:
Uploading cytotrace2_py-1.0.3-py3-none-any.whl
56% ━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ 48.9/86.9 MB • 01:15 • 513.3 kB/s
Traceback (most recent call last):
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/urllib3/response.py", line 748, in _error_catcher
yield
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/urllib3/response.py", line 873, in _raw_read
data = self._fp_read(amt, read1=read1) if not fp_closed else b""
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/urllib3/response.py", line 856, in _fp_read
return self._fp.read(amt) if amt is not None else self._fp.read()
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/http/client.py", line 458, in read
n = self.readinto(b)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/http/client.py", line 502, in readinto
n = self.fp.readinto(b)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
TimeoutError: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/requests/models.py", line 820, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/urllib3/response.py", line 1060, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/urllib3/response.py", line 949, in read
data = self._raw_read(amt)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/urllib3/response.py", line 902, in _raw_read
self._fp.close()
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/contextlib.py", line 135, in exit
self.gen.throw(type, value, traceback)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/urllib3/response.py", line 775, in _error_catcher
raise ProtocolError(f"Connection broken: {e!r}", e) from e
urllib3.exceptions.ProtocolError: ("Connection broken: TimeoutError(60, 'Operation timed out')", TimeoutError(60, 'Operation timed out'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/savagyan/anaconda3/envs/test_pypi/bin/twine", line 8, in
sys.exit(main())
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/twine/main.py", line 33, in main
error = cli.dispatch(sys.argv[1:])
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/twine/cli.py", line 129, in dispatch
return main(args.args)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/twine/commands/upload.py", line 258, in main
return upload(upload_settings, parsed_args.dists)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/twine/commands/upload.py", line 202, in upload
resp = repository.upload(package)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/twine/repository.py", line 188, in upload
resp = self._upload(package)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/twine/repository.py", line 174, in _upload
resp = self.session.post(
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/requests/sessions.py", line 746, in send
r.content
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/requests/models.py", line 902, in content
self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
File "/Users/savagyan/anaconda3/envs/test_pypi/lib/python3.9/site-packages/requests/models.py", line 822, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: TimeoutError(60, 'Operation timed out')", TimeoutError(60, 'Operation timed out'))`

It seems like you might have an unstable network connection. Can you file a network access issue here instead?

It seems like you might have an unstable network connection. Can you file a network access issue here instead?

I was connected to VPN, tried without and get this log:
`twine upload --repository testpypi --verbose dist/*
INFO     Using configuration from /Users/savagyan/.pypirc
Uploading distributions to https://test.pypi.org/legacy/
INFO     dist/cytotrace2_py-1.0.3-py3-none-any.whl (82.9 MB)
INFO     dist/cytotrace2_py-1.0.3.tar.gz (82.8 MB)
INFO     username set by command options
INFO     password set from config file
INFO     username: token
INFO     password:
Uploading cytotrace2_py-1.0.3-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.9/86.9 MB • 01:10 • 1.3 MB/s
WARNING  Received "503: Service Unavailable"
Package upload appears to have failed. Retry 1 of 5.
Uploading cytotrace2_py-1.0.3-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.9/86.9 MB • 01:10 • 1.3 MB/s
WARNING  Received "503: Service Unavailable"
Package upload appears to have failed. Retry 2 of 5.
Uploading cytotrace2_py-1.0.3-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.9/86.9 MB • 01:09 • 1.2 MB/s
WARNING  Received "503: Service Unavailable"
Package upload appears to have failed. Retry 3 of 5.
Uploading cytotrace2_py-1.0.3-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.9/86.9 MB • 01:11 • 1.3 MB/s
WARNING  Received "503: Service Unavailable"
Package upload appears to have failed. Retry 4 of 5.
Uploading cytotrace2_py-1.0.3-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.9/86.9 MB • 01:09 • 1.2 MB/s
WARNING  Received "503: Service Unavailable"
Package upload appears to have failed. Retry 5 of 5.
INFO     Response from https://test.pypi.org/legacy/:
503 Service Unavailable
INFO

         <meta name="defaultLanguage" content="en">
         <meta name="availableLanguages" content="en">

         <title>Service Unavailable (503)</title>

         <meta name="description" content="The Python Package Index (PyPI) is a repository of software for the Python programming language.">

         <!-- Fonts -->
         <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400italic,600,600italic,700,700italic|Source+Code+Pro:500">

         <style>
           html,
           body {
             box-sizing: border-box;
             font-family: 'Source Sans Pro', 'Helvetica', Arial, sans-serif;
             font-size: 17px;
             line-height: 1.4;
             font-weight: 300;
             color: #fff;
             -webkit-text-size-adjust: 100%;
             font-size: 100%;
             padding: 0;
             margin: 0;
           }

           h1, strong {
             font-weight: 600;
           }

           ul {
             list-style-type: none;
             padding-left: 0;
           }

           hr {
             margin: 50px auto;
             width: 300px;
             max-width: 100%;
             border: 0;
             height: 2px;
             background: rgba(255, 255, 255, 0.5);
           }

           .viewport-section {
             height: 100vh;
             display: flex;
             align-items: center;
             justify-content: center;
             background-color: #006dad;
             color: #fff;
           }

           .viewport-section a {
             color: #fff;
           }

           .viewport-section__content {
             text-align: center;
             padding: 30px;
             min-width: 0;
           }

           .viewport-section__heading {
             font-size: 2.4em;
             padding: 20px 15px 0;
             margin: 0;
           }

           @media screen and (max-width: 600px) {
             .viewport-section__content {
               padding: 15px;
             }

             .viewport-section__heading {
               font-size: 1.5em;
             }
           }
         </style>

       </head>

       <body>
         <div class="viewport-section">
           <div class="viewport-section__content">
             <img alt="PyPI logo"
     src="data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI2NS44MTIiIGhlaWdodD0iNTgiIHZpZXdCb3g9IjAgMCA2NS44MTIwMzUgNTguMDAwMDAxIj48ZyBmaWxsPSIjYTI5ZDg2IiBzdHJva2U9IiNjY2MiIHN0cm9rZS13aWR0aD0iLjIxNCIgc3Ryb2tlLWxpbmVqb2luPSJiZXZl
[encoded data, 18936 characters]
     ">
             <h1 class="viewport-section__heading">Sorry, something went wrong</h1>

             <div class="viewport-section__text">
               <p>PyPI is down for maintenance or is having an outage.</p>
               <p>
                 This is affecting several of our services, including our web interface.</br>
                 If you are trying to install a package, you should be able to <strong>pip install</strong> packages without problem.
               </p>

               <ul class="unstyled">
                 <li><a href="https://status.python.org/">Check our status page</a>, or</li>
                 <li><a href="https://twitter.com/PythonStatus">View Python Status on Twitter</a></li>
               </ul>

               <hr>

               <p>
                 <strong>
                   Rely on PyPI to get your job done?</br>
                   Consider <a href="https://github.com/pypa/warehouse">contributing</a> or <a href="https://psfmember.org/civicrm/contribute/transact?reset=1&id=13">donating</a> to help us build a more stable and secure platform.
                 </strong>
               </p>
             </div>
           </div>
         </div>
       </body>
     </html>

ERROR    HTTPError: 503 Service Unavailable from https://test.pypi.org/legacy/
Service Unavailable   `
Should I still open a network access issue?

repo: pypi/warehouse


text:
Disallow creation of non-CRS (eg pipeline) in CRS class

From discussion with @snowman2
Currently, you can create a CRS object from eg a proj pipeline:
In [4]: pyproj.CRS("+proj=pipeline +ellps=GRS80 +step +proj=merc +step +proj=axisswap +order=2,1")
Out[4]:
<CRS: +proj=pipeline +ellps=GRS80 +step +proj=merc +step ...>
Name: PROJ-based coordinate operation
Ellipsoid:
- UNDEFINED
Area of Use:
- UNDEFINED
Prime Meridian:
- UNDEFINED
Axis Info:
- UNDEFINED

which does not give you an actual CRS, but generic coordinate transformation. So it is probably a good idea to disallow this in the construction of a CRS object.
For this specific case we could check for proj=pipeline in the string. Or more in general we could check the proj_type and raise an error if it is not one of the CRS types (we could still do the first pipeline check as well to have a better error message for that specific case).

I think just need to check that proj_type == PJ_TYPE_CRS

I think just need to check that proj_type == PJ_TYPE_CRS

Not sure if that is enough, as you have also PJ_TYPE_PROJECTED_CRS etc (don't know how such an equality check works, but I assume it is like an enum, and not aware that PJ_TYPE_PROJECTED_CRS is a subset of PJ_TYPE_CRS) ?
There is also proj_is_crs that can be probably used for this

The proj_is_crs looks perfect. I am glad you found that.

I have a less than optimal implementation in #263.

I am going to close this as the solution is merged into master. Thanks for bringing up the idea!


text:
Add setuptools dependency when running with Python >=3.12

setuptools is no longer included with the Python distro as of 3.12. Explicitly add as a dependency if building with >= 3.12.

Without explicitly depending on setuptools, tests fail with the following:
ImportError while loading conftest '/Users/tomward/dev/pyranges/tests/unit/conftest.py'.
tests/unit/conftest.py:5: in <module>
    from pyranges import PyRanges
pyranges/__init__.py:8: in <module>
    import pkg_resources
E   ModuleNotFoundError: No module named 'pkg_resources'

repo: pyranges/pyranges


text:
Remove Kaleido

To prevent errors by kaleido in default setup (see #195), plotly plots are not converted to static images anymore by default (so kaleido is not needed and therefore dependency can be removed). This also enables interactive elements in .plot() and ._repr_html_().
The plotting API is also refactored to force some figure defaults for both backends.
Users of ReprMixin now must only override _plot_plotly_ and/or _plot_matplotlib_ methods to enable plotting. They must return a figure object of the respective backend.

Please resolve so we can merge and finalize the release.

Still yields errors in tests. Please fix.

repo: pyroll-project/pyroll-core


text:
Using Kubernetes labels/annotation as Prometheus alert labels and annotations is limiting

The strategy of using kubernetes labels and annotations, prefixed with pyrra.dev/, as a strategy to generate Prometheus Alert labels and annotations works great for most cases! However, Kubernetes labels and annotations need to pass through different regular expressions than Prometheus requires, limiting the possible label values that one could use.
One common case is passing URLs, e.g. Runbook URL and Dashboard URL, as alert labels, but Kubernetes rejects such cases, another example is the 63 characters limitation:
ServiceLevelObjective.pyrra.dev "pyrra-pyrra-reconciliations" is invalid: [metadata.labels: Invalid value: "https://grafana.xxx.xxx/d/ccssRIenz/pyrra-slos?orgId=1&var-CX_ENV_ID=xxxx&from=now-1d&to=now": must be no more than 63 characters, metadata.labels: Invalid value: "https://grafana.xxx.xxx/d/ccssRIenz/pyrra-slos?orgId=1&var-CX_ENV_ID=xxx&to=now": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')]

We could easily fix that by adding an option for adding extra labels/annotations in the Alerting spec

Oh, that's an interesting point with the URLs. I haven't run into this myself yet.
I guess in that case your proposal makes sense.
The original reasoning for having them as CRD labels is to make filtering via kubectl get slo -l pyrra.dev/team=backend possible. I wonder how much usage that feature has in the first place.

The original reasoning for having them as CRD labels is to make filtering via kubectl get slo -l pyrra.dev/team=backend possible. I wonder how much usage that feature has in the first place.

I usually search for SLOs using ArgoCD UI or Pyrra UI. I haven't used kubectl in a while, honestly 😬. If people rely on that feature, I guess they could keep their labels there, but I understand the duplicated effort to keep labels in sync in two different places.
Do you see users requesting the original use case?

In our case, we use annotations for runbooks and dashboard links, they are also propagated when prefixed with pyrra.dev/ and allow for much larger texts. I think labels are meant to be used more to filter, which is not something we usually require for those links.

authors: ArthurSens PabloPie metalmatze

repo: pyrra-dev/pyrra


text:
Feature: Allow customisation of warning/critical alert levels

For some SLOs, we don't want high burn rate alerts to have severity critical, but instead have a different value (for example severe or p1 etc)
Allow high/low burn rate severities to be configured within an SLO
This could potentially look like the following
spec:
  alerting:
    severity:
      high: critical
      low: warning

There's a PR open for quite some time that we just need to finish and finally merge: https://github.com/pyrra-dev/pyrra/pull/617

I'll close this as it's a duplicate of that PR. Feel free to comment on the PR or here again, if anything else comes up.

authors: metalmatze vCra


text:
Add explicit classifications for supported python versions

Many tools such as caniusepython3 rely on classification to determine library support.

None taken - I did not verify each version. I referenced their documentation of supported versions:
github.com/python-intelhex/intelhex/blob/master/Readme.rst#supported-python-versions

That specific line was last changed before Python 3.7 was released :)
There's no reason to assume the package won't work with 3.7, just thought I'd ask...

repo: python-intelhex/intelhex


text:
Tags not recognised

Hi,
I'm trying to use versioneer to use a GitHub Action to publish to PyPI automatically when commits are tagged. Unfortunately the tags I have on my branches do not appear to be recognised either when building locally or via a GitHub action.
Locally
❱ git tag --list
v0.0.1.post1
v0.01
v0.02
v0.1.0
v0.1.1
❱ python -m build
 * Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.9.0, versioneer, wheel)
* Getting dependencies for sdist...
running egg_info
writing pgfinder.egg-info/PKG-INFO
writing dependency_links to pgfinder.egg-info/dependency_links.txt
writing entry points to pgfinder.egg-info/entry_points.txt
writing requirements to pgfinder.egg-info/requires.txt
writing top-level names to pgfinder.egg-info/top_level.txt
reading manifest file 'pgfinder.egg-info/SOURCES.txt'
adding license file 'COPYING'
adding license file 'COPYING.LESSER'
writing manifest file 'pgfinder.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing pgfinder.egg-info/PKG-INFO
writing dependency_links to pgfinder.egg-info/dependency_links.txt
writing entry points to pgfinder.egg-info/entry_points.txt
writing requirements to pgfinder.egg-info/requires.txt
writing top-level names to pgfinder.egg-info/top_level.txt
reading manifest file 'pgfinder.egg-info/SOURCES.txt'
adding license file 'COPYING'
adding license file 'COPYING.LESSER'
writing manifest file 'pgfinder.egg-info/SOURCES.txt'
running check
creating pgfinder-0+untagged.582.gdd3feca.dirty
creating pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
creating pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
creating pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder/logs
creating pgfinder-0+untagged.582.gdd3feca.dirty/tests
copying files to pgfinder-0+untagged.582.gdd3feca.dirty...
copying COPYING -> pgfinder-0+untagged.582.gdd3feca.dirty
copying COPYING.LESSER -> pgfinder-0+untagged.582.gdd3feca.dirty
copying README.md -> pgfinder-0+untagged.582.gdd3feca.dirty
copying pyproject.toml -> pgfinder-0+untagged.582.gdd3feca.dirty
copying setup.cfg -> pgfinder-0+untagged.582.gdd3feca.dirty
copying setup.py -> pgfinder-0+untagged.582.gdd3feca.dirty
copying pgfinder/__init__.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder/_version.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder/find_pg.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder/io.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder/matching.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder/pgio.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder/utils.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder/validation.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder
copying pgfinder.egg-info/PKG-INFO -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
copying pgfinder.egg-info/SOURCES.txt -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
copying pgfinder.egg-info/dependency_links.txt -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
copying pgfinder.egg-info/entry_points.txt -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
copying pgfinder.egg-info/not-zip-safe -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
copying pgfinder.egg-info/requires.txt -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
copying pgfinder.egg-info/top_level.txt -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder.egg-info
copying pgfinder/logs/__init__.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder/logs
copying pgfinder/logs/logs.py -> pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder/logs
copying tests/__init__.py -> pgfinder-0+untagged.582.gdd3feca.dirty/tests
copying tests/test_matching.py -> pgfinder-0+untagged.582.gdd3feca.dirty/tests
copying tests/test_regression.py -> pgfinder-0+untagged.582.gdd3feca.dirty/tests
copying tests/test_validation.py -> pgfinder-0+untagged.582.gdd3feca.dirty/tests
Writing pgfinder-0+untagged.582.gdd3feca.dirty/setup.cfg
UPDATING pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder/_version.py
set pgfinder-0+untagged.582.gdd3feca.dirty/pgfinder/_version.py to '0+untagged.582.gdd3feca.dirty'
Creating tar archive
removing 'pgfinder-0+untagged.582.gdd3feca.dirty' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.9.0, versioneer, wheel)
* Getting dependencies for wheel...
running egg_info
writing pgfinder.egg-info/PKG-INFO
writing dependency_links to pgfinder.egg-info/dependency_links.txt
writing entry points to pgfinder.egg-info/entry_points.txt
writing requirements to pgfinder.egg-info/requires.txt
writing top-level names to pgfinder.egg-info/top_level.txt
reading manifest file 'pgfinder.egg-info/SOURCES.txt'
adding license file 'COPYING'
adding license file 'COPYING.LESSER'
writing manifest file 'pgfinder.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/pgfinder
copying pgfinder/validation.py -> build/lib/pgfinder
copying pgfinder/utils.py -> build/lib/pgfinder
copying pgfinder/pgio.py -> build/lib/pgfinder
copying pgfinder/matching.py -> build/lib/pgfinder
copying pgfinder/io.py -> build/lib/pgfinder
copying pgfinder/find_pg.py -> build/lib/pgfinder
copying pgfinder/_version.py -> build/lib/pgfinder
copying pgfinder/__init__.py -> build/lib/pgfinder
creating build/lib/pgfinder/logs
copying pgfinder/logs/logs.py -> build/lib/pgfinder/logs
copying pgfinder/logs/__init__.py -> build/lib/pgfinder/logs
running egg_info
writing pgfinder.egg-info/PKG-INFO
writing dependency_links to pgfinder.egg-info/dependency_links.txt
writing entry points to pgfinder.egg-info/entry_points.txt
writing requirements to pgfinder.egg-info/requires.txt
writing top-level names to pgfinder.egg-info/top_level.txt
reading manifest file 'pgfinder.egg-info/SOURCES.txt'
adding license file 'COPYING'
adding license file 'COPYING.LESSER'
writing manifest file 'pgfinder.egg-info/SOURCES.txt'
UPDATING build/lib/pgfinder/_version.py
set build/lib/pgfinder/_version.py to '0+untagged.582.gdd3feca.dirty'
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/pgfinder
copying build/lib/pgfinder/_version.py -> build/bdist.linux-x86_64/wheel/pgfinder
creating build/bdist.linux-x86_64/wheel/pgfinder/logs
copying build/lib/pgfinder/logs/__init__.py -> build/bdist.linux-x86_64/wheel/pgfinder/logs
copying build/lib/pgfinder/logs/logs.py -> build/bdist.linux-x86_64/wheel/pgfinder/logs
copying build/lib/pgfinder/__init__.py -> build/bdist.linux-x86_64/wheel/pgfinder
copying build/lib/pgfinder/find_pg.py -> build/bdist.linux-x86_64/wheel/pgfinder
copying build/lib/pgfinder/io.py -> build/bdist.linux-x86_64/wheel/pgfinder
copying build/lib/pgfinder/matching.py -> build/bdist.linux-x86_64/wheel/pgfinder
copying build/lib/pgfinder/pgio.py -> build/bdist.linux-x86_64/wheel/pgfinder
copying build/lib/pgfinder/utils.py -> build/bdist.linux-x86_64/wheel/pgfinder
copying build/lib/pgfinder/validation.py -> build/bdist.linux-x86_64/wheel/pgfinder
running install_egg_info
Copying pgfinder.egg-info to build/bdist.linux-x86_64/wheel/pgfinder-0+untagged.582.gdd3feca.dirty-py3.10.egg-info
running install_scripts
adding license file "COPYING.LESSER" (matched pattern "COPYING*")
adding license file "COPYING" (matched pattern "COPYING*")
creating build/bdist.linux-x86_64/wheel/pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/WHEEL
creating '/home/neil/work/projects/pgfinder/dist/tmpglxmc3mb/pgfinder-0+untagged.582.gdd3feca.dirty-py2.py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'pgfinder/__init__.py'
adding 'pgfinder/_version.py'
adding 'pgfinder/find_pg.py'
adding 'pgfinder/io.py'
adding 'pgfinder/matching.py'
adding 'pgfinder/pgio.py'
adding 'pgfinder/utils.py'
adding 'pgfinder/validation.py'
adding 'pgfinder/logs/__init__.py'
adding 'pgfinder/logs/logs.py'
adding 'pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/COPYING'
adding 'pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/COPYING.LESSER'
adding 'pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/METADATA'
adding 'pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/WHEEL'
adding 'pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/entry_points.txt'
adding 'pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/top_level.txt'
adding 'pgfinder-0+untagged.582.gdd3feca.dirty.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built pgfinder-0+untagged.582.gdd3feca.dirty.tar.gz and pgfinder-0+untagged.582.gdd3feca.dirty-py2.py3-none-any.whl

The version returned by versioneer is 0+untagged.582.gdd3feca.dirty.
In turn the GitHub action fails because this doesn't conform to PEP440.
The relevant parts of my configuration are...
setup.cfg
[metadata]
name = pgfinder
description =
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/Mesnage-Org/pgfinder
author = PGFinder Team
author_email = pgfinder@sheffield.ac.uk
license = GNU Lesser GPLv3
project_urls =
  Bug Tracker = https://github.com/Mesnage-Org/pgfinder/issues
  Documentation = https://mesnage-org.github.io/pgfinder/
classifiers =
    Programming Language :: Python :: 3
...
[versioneer]
VCS = git
style = pep440
versionfile_source = pgfinder/_version.py
versionfile_build = pgfinder/_version.py
tag_prefix = "v-"
parentdir_prefix = "pgfinder-"

NB - I've tag_prefix = v- (i.e. without quotes) but with the same result.
pyproject.toml
[build-system]
requires = [
  "setuptools >= 40.9.0",
  "wheel",
  "versioneer"
  ]
build-backend = "setuptools.build_meta"

setup.py
"""Setup package for building"""
import sys
import setuptools
import versioneer

sys.path.insert(0, ".")
setuptools.setup(
    version=versioneer.get_version(),
    cmdclass=versioneer.get_cmdclass(),
)

I've probably overlooked something but can't work out why and would be very grateful of advice.
Thanks,
@ns-rse

And right after posting I notice I've set the tag_prefix = v- when it should be tag_prefix = v :facepalm:

Note also that you don't need sys.path.insert(0, ".") if you're using non vendored versioneer.

authors: effigies ns-rse

repo: python-versioneer/python-versioneer

license: Unlicense


text:
python setup.py build fails

Using the latest head version of versioneer, I am running into the following error when doing "python setup.py build"
running build
running build_py
UPDATING build/lib/_version.py
error: [Errno 2] No such file or directory: 'build/lib/_version.py'

The build/lib actually have the _version.py present - is there anything I missed?
Thanks

never mind, I misconfigured versionfile_build, my bad.

The response pointed me to the right direction but I think it the answer can be better since there are multiple versionfile_build in the files added by versioneer. You have to go to setup.cfg and fix versionfile_build.
In my case, I am using a project structure pkt/src/pkt and so my setup.cfg should be:
[versioneer]
VCS = git
style = pep440
versionfile_source = src/pkt/_version.py
versionfile_build = pkt/_version.py
tag_prefix = v


text:
🛑 Private Message Api is down

In bd18fbd, Private Message Api (https://privatemessages.roblox.com/) was down:

HTTP code: 503
Response time: 703 ms

Resolved: Private Message Api is back up in 89a3bd4 after 35 minutes.

repo: pythoniaweb/statusboom


text:
🐛 [Bug] InvalidVersion error when trying to use torch_tensorrt that was compiled in a Jetson container image

Bug Description
import torch_tensorrt

raises an InvalidVersion error when installed using:
python3 setup.py install --use-cxx11-abi

on a Jetson Xavier AGX using the dustynv/ros:humble-pytorch-l4t-r35.3.1 container base image found here. The issue appears to be caused by the fact that torch.__version__ == 2.0.0.nv23.05. Changing the line here:
if version.parse(torch.__version__) >= version.parse("2.dev"):

to:
if version.parse(".".join(torch.__version__.split(".")[:3])) >= version.parse("2.dev"):

makes the error go away.
To Reproduce
Start a container using the dustynv/ros:humble-pytorch-l4t-r35.3.1 base image found here.
apt update
apt install default-jdk unzip zip

PYTHON_VERSION=$(python --version | sed -n 's/^Python \([0-9]\+\.[0-9]\+\).*$/\1/p')

CUDA_VERSION=$(nvcc --version | sed -n 's/^.*release \([0-9]\+\.[0-9]\+\).*$/\1/p')

CUDNN_MAJOR=$(cat /usr/include/cudnn_version.h | sed -n 's/^.*CUDNN_MAJOR \([0-9]\+\).*$/\1/p')
CUDNN_MINOR=$(cat /usr/include/cudnn_version.h | sed -n 's/^.*CUDNN_MINOR \([0-9]\+\).*$/\1/p')
CUDNN_VERSION=${CUDNN_MAJOR}.${CUDNN_MINOR}

TENSORRT_MAJOR=$(cat /usr/include/aarch64-linux-gnu/NvInferVersion.h | sed -n 's/^.*NV_TENSORRT_MAJOR \([0-9]\+\).*$/\1/p')
TENSORRT_MINOR=$(cat /usr/include/aarch64-linux-gnu/NvInferVersion.h | sed -n 's/^.*NV_TENSORRT_MINOR \([0-9]\+\).*$/\1/p')
TENSORRT_VERSION=${TENSORRT_MAJOR}.${TENSORRT_MINOR}

git clone -b v1.4.0 https://github.com/pytorch/TensorRT.git

cd TensorRT
cp toolchains/jp_workspaces/WORKSPACE.jp50 WORKSPACE
sed -i "s/python[0-9]\+.[0-9]\+/python${PYTHON_VERSION}/g" WORKSPACE
sed -i "s/cuda-[0-9]\+.[0-9]\+/cuda-${CUDA_VERSION}/g" WORKSPACE

BAZEL_VERSION=$(cat .bazelversion)
mkdir bazel
cd bazel
curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-dist.zip
unzip bazel-${BAZEL_VERSION}-dist.zip
bash ./compile.sh
cp output/bazel /usr/local/bin/

cd ../py
TORCH_TENSORRT_VERSION=$(cat versions.py | sed -n 's/^__version__ = "\(.*\)"/\1/p')
echo "__version__ = \"${TORCH_TENSORRT_VERSION}\"" > versions.py
echo "__cuda_version__ = \"${CUDA_VERSION}\"" >> versions.py
echo "__cudnn_version__ = \"${CUDNN_VERSION}\"" >> versions.py
echo "__tensorrt_version__ = \"${TENSORRT_VERSION}\"" >> versions.py

python3 setup.py install --use-cxx11-abi

cd
python3 -c "import torch_tensorrt; print(torch_tensorrt.__version__)"

Expected behavior
No error.
Environment
A Jetson Xavier AGX and the container base image found here.
Additional context

I'm realizing now this is probably not the correct workflow, and that I should instead be using Torch-TensorRT outside of the container to compile the model and then providing the compiled model to the container.

Re-opening because the fact that the repository points to containers suggests this should be an OK strategy.

Hi - thanks for the report. I am able to reproduce the issue, and I've linked a PR that will address this versioning check.

repo: pytorch/TensorRT


text:
[Quantization + FSDP] Support quantize_() for DTensor

While trying out INT8 mixed precision pretraining (#748) with torchtitan, I came across an issue that if the model is FSDP-sharded, quantize_() won't work. The fix would be adding an extra logic to handle DTensor, similar to what FP8 is doing
https://github.com/pytorch/ao/blob/f5703b07acc683653556d04ef970709ba47dba10/torchao/float8/float8_tensor.py#L161-L183

Yeah this came up in some discussions with inference providers like SGLang as well

authors: gau-nernst msaroufim

repo: pytorch/ao


text:
Fix mixed signed/un-signed types in comparison

Fix the warning of mixed integer type. See also https://github.com/pytorch/audio/issues/1229#issuecomment-772652248
/scratch/moto/torchaudio/torchaudio/csrc/sox/effects_chain.cpp: In function ‘int torchaudio::sox_effects_chain::{anonymous}::fileobj_input_drain(sox_effect_t*, sox_sample_t*, size_t*)’:
/scratch/moto/torchaudio/torchaudio/csrc/sox/effects_chain.cpp:357:20: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (num_consumed > priv->buffer_size) {
       ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
/scratch/moto/torchaudio/torchaudio/csrc/sox/effects_chain.cpp:369:20: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     if (num_refill < num_consumed) {
         ~~~~~~~~~~~^~~~~~~~~~~~~~

Thanks!

repo: pytorch/audio


text:
Roberta: Outputs after BPE encoding step and Pre-processing (Binarize) steps

In the fine-tuning steps of Roberta and BART, we first encode raw text into tokens using roberta.multiprocessing_bpe_encoder. Then we run fairseq-preprocess to binarize the data. Finally, we feed the tokens to the model using the binarized file in the training process.
But, when I check the tokens I get for a given sentence using BPE encoding and binarized files are not similar to each other. Why is that?

Can you post some examples of the inconsistencies?  Thanks!

Hi @lematt1991 ,
Thank you for your help.  Please go through the following example to understand the issue.
The input Sentence
"What?!?? Why are people saying this is "mind blowing?" Just face it the ending is on of the worst endings in the history of cinematography! 4 left and the whole world has ended! Not to mention the character 9 was a idiot the whole time he got everyone killed. 1 was right the whole time, if he sacrificed 9 then non of this would have happened. People giving there lives for a stupid cause and for what?... to make it rain? I admit the movie had it's parts, and the whole concept was fascinating. But a lot of it was clichés one after another. And did anybody else get this feeling that this is a lot like "Lord of The Rings?" Characters died for stupid reasons, there was almost no character development, and honestly ask yourself is it good to have only four guys left in the world; its pointless and stupid. It was one of the shortest movies I've ever seen, and thank god! How is robots turning against humans creative in any way! It's been done like a hundred times! This movie is really stupid, go see a movie that's worth watching like Star Trek, The Hangover, or Inglorious Basterds, those were good movies!"
Tokens I get after running  examples.roberta.multiprocessing_bpe_encoder
'2061 12248 3548 4162 389 661 2282 428 318 366 10155 19280 1701 2329 1986 340 262 7464 318 319 286 262 5290 38168 287 262 2106 286 13483 45501 0 604 1364 290 262 2187 995 468 4444 0 1892 284 3068 262 2095 860 373 257 22324 262 2187 640 339 1392 2506 2923 13 352 373 826 262 2187 640 11 611 339 27445 860 788 1729 286 428 561 423 3022 13 4380 3501 612 3160 329 257 8531 2728 290 329 644 30 986 284 787 340 6290 30 314 9159 262 3807 550 340 338 3354 11 290 262 2187 3721 373 13899 13 887 257 1256 286 340 373 35478 20954 530 706 1194 13 843 750 9599 2073 651 428 4203 326 428 318 257 1256 588 366 22438 286 383 26028 1701 26813 3724 329 8531 3840 11 612 373 2048 645 2095 2478 11 290 12698 1265 3511 318 340 922 284 423 691 1440 3730 1364 287 262 995 26 663 27158 290 8531 13 632 373 530 286 262 35581 6918 314 1053 1683 1775 11 290 5875 5770 0 1374 318 14193 6225 1028 5384 7325 287 597 835 0 632 338 587 1760 588 257 3470 1661 0 770 3807 318 1107 8531 11 467 766 257 3807 326 338 2861 4964 588 2907 12338 11 383 24300 2502 11 393 17589 4685 699 347 1603 9310 11 883 547 922 6918 0'
The output I get in the training phase (after fairseq-preprocess )
After  print(sample['net_input']['src_tokens'])
tensor([[    0,  2264, 17516, 28749,  2612,    32,    82,   584,    42,    16,
22, 28583, 13897,  1917,  1801,   652,    24,     5,  3558,    16,
15,     9,     5,  2373, 41677,    11,     5,   750,     9, 18535,
34226,   328,   204,   314,     8,     5,  1086,   232,    34,  1249,
328,  1491,     7,  4521,     5,  2048,   361,    21,    10, 30603,
5,  1086,    86,    37,   300,   961,   848,     4,   112,    21,
235,     5,  1086,    86,     6,   114,    37, 26936,   361,   172,
786,     9,    42,    74,    33,  1102,     4,  1806,  1311,    89,
1074,    13,    10, 12103,  1303,     8,    13,    99,   116,   734,
7,   146,    24,  1895,   116,    38,  8109,     5,  1569,    56,
24,    18,  1667,     6,     8,     5,  1086,  4286,    21, 12509,
4,   125,    10,   319,     9,    24,    21, 43848,  5739,    65,
71,   277,     4,   178,   222,  5670,  1493,   120,    42,  2157,
14,    42,    16,    10,   319,   101,    22, 37565,     9,    20,
32106,  1917, 44217,   962,    13, 12103,  2188,     6,    89,    21,
818,   117,  2048,   709,     6,     8, 10728,  1394,  2512,    16,
24,   205,     7,    33,   129,   237,  1669,   314,    11,     5,
232,   131,    63, 31321,     8, 12103,     4,    85,    21,    65,
9,     5, 28617,  4133,    38,   348,   655,   450,     6,     8,
3392,  9069,   328,  1336,    16, 12129,  3408,   136,  5868,  3904,
11,   143,   169,   328,    85,    18,    57,   626,   101,    10,
6317,   498,   328,   152,  1569,    16,   269, 12103,     6,   213,
192,    10,  1569,    14,    18,   966,  2494,   101,  2141, 20351,
6,    20, 12403,  2137,     6,    50, 11996, 10159,  6514,   163,
8831, 11622,     6,   167,    58,   205,  4133,   328,     2]],
device='cuda:0')
So does fairseq-preprocess again assign tokens to its own dictionary?

repo: pytorch/fairseq


text:
Model can't locate bindings, even after installation

I've successfully installed Python Bindings. :-
****@**-gpu:~/wav2letter/bindings/python$ pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///path/wav2letter/bindings/python
Installing collected packages: wav2letter
  Running setup.py develop for wav2letter
Successfully installed wav2letter

Still  while running below command :-
****@**-gpu:~/wav2letter/bindings/python$ PYTHONPATH=/path/fairseq python3 examples/speech_recognition/infer.py /path/audio_file/wav2vec/ --task audio_pretraining --nbest 1 --path /path/audio_file/wav2vec_small.pt --gen-subset valid --results-path /path/audio_file/wav2vec/tmp/am/ --w2l-decoder viterbi --word-score -1 --sil-weight 0 --criterion ctc --labels ltr --max-tokens 4000000 --post-process letter
Getting below errors :-

INFO:fairseq.data.audio.raw_audio_dataset:loaded 29, skipped 0 samples
INFO:main:| /path/audio_file/wav2vec/ valid 29 examples
INFO:main:| decoding with criterion ctc
INFO:main:| loading model(s) from /path/audio_file/wav2vec_small.pt
/path/fairseq/examples/speech_recognition/w2l_decoder.py:39: UserWarning: wav2letter python bindings are required to use this functionality. Please install from https://github.com/facebookresearch/wav2letter/wiki/Python-bindings
"wav2letter python bindings are required to use this functionality. Please install from https://github.com/facebookresearch/wav2letter/wiki/Python-bindings"
Traceback (most recent call last):
File "examples/speech_recognition/infer.py", line 428, in
cli_main()
File "examples/speech_recognition/infer.py", line 424, in cli_main
main(args)
File "examples/speech_recognition/infer.py", line 361, in main
hypos = task.inference_step(generator, models, sample, prefix_tokens)
File "/path/fairseq/fairseq/tasks/fairseq_task.py", line 400, in inference_step
return generator.generate(models, sample, prefix_tokens=prefix_tokens)
File "/path/fairseq/examples/speech_recognition/w2l_decoder.py", line 76, in generate
emissions = self.get_emissions(models, encoder_input)
File "/path/fairseq/examples/speech_recognition/w2l_decoder.py", line 84, in get_emissions
emissions = models[0].get_normalized_probs(encoder_out, log_probs=True)
File "/path/fairseq/fairseq/models/fairseq_model.py", line 53, in get_normalized_probs
return self.get_normalized_probs_scriptable(net_output, log_probs, sample)
File "/path/fairseq/fairseq/models/fairseq_model.py", line 74, in get_normalized_probs_scriptable
raise NotImplementedError
NotImplementedError

Can you please suggest what might be the issue @alexeib  . Thanks in advance.

i think you must have solved this since you had a different error in another issue, right?

Hi @manthanthakker, did you solve this issue? I'm having the same issue after installing wav2letter bindings with CUDA enabled. Please let me know, thanks.

Hi @MrityunjoyS , did you solve this issue? I'm having the same issue after installing wav2letter bindings with CUDA enabled. Please let me know, thanks.

authors: MrityunjoyS alexeib taiman9


text:
[C++ Frontend] Most tests fail on OSX

I've just tried running the tests for the C++ frontend on an OSX, with PyTorch nightly from today and torchvision from master, and most tests fail with an error similar to the following:
Traceback (most recent call last):
  File "test/test_cpp_models.py", line 43, in test_alexnet
    process_model(models.alexnet(self.pretrained), self.image, _C_tests.forward_alexnet, 'Alexnet')
  File "test/test_cpp_models.py", line 16, in process_model
    cpp_output = func("model.pt", tensor)
RuntimeError: new_refcount != 1 INTERNAL ASSERT FAILED at /Users/fmassa/anaconda3/lib/python3.6/site-packages/torch/include/c10/util/intrusive_ptr.h:185, please report a bug to PyTorch. intrusive_ptr: Cannot increase refcount after it reached zero. (retain_ at /Users/fmassa/anaconda3/lib/python3.6/site-packages/torch/include/c10/util/intrusive_ptr.h:185)
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 135 (0x111340917 in libc10.dylib)
frame #1: c10::intrusive_ptr<c10::TensorImpl, c10::UndefinedTensorImpl>::retain_() + 360 (0x11f9b3dd8 in _C_tests.cpython-36m-darwin.so)
frame #2: at::Tensor std::__1::__invoke_void_return_wrapper<at::Tensor>::__call<at::Tensor& (*&)(at::Tensor), at::Tensor>(at::Tensor& (*&&&)(at::Tensor), at::Tensor&&) + 124 (0x11f9dbe6c in _C_tests.cpython-36m-darwin.so)
frame #3: std::__1::__function::__func<at::Tensor& (*)(at::Tensor), std::__1::allocator<at::Tensor& (*)(at::Tensor)>, at::Tensor (at::Tensor)>::operator()(at::Tensor&&) + 18 (0x11f9dbdb2 in _C_tests.cpython-36m-darwin.so)
frame #4: torch::nn::FunctionalImpl::forward(at::Tensor) + 53 (0x115d98a55 in libtorch.dylib)
frame #5: torch::nn::AnyModule::Value torch::nn::AnyModule::Holder<torch::nn::FunctionalImpl, at::Tensor>::InvokeForward::operator()<at::Tensor>(at::Tensor&&) + 54 (0x11f9e03f6 in _C_tests.cpython-36m-darwin.so)
frame #6: torch::nn::AnyModule::Holder<torch::nn::FunctionalImpl, at::Tensor>::forward(std::__1::vector<torch::nn::AnyModule::Value, std::__1::allocator<torch::nn::AnyModule::Value> >&&) + 86 (0x11f9dffe6 in _C_tests.cpython-36m-darwin.so)
frame #7: torch::nn::AnyModule::Value torch::nn::AnyModule::any_forward<torch::nn::AnyModule::Value>(torch::nn::AnyModule::Value&&) + 168 (0x11f9cad48 in _C_tests.cpython-36m-darwin.so)
frame #8: at::Tensor torch::nn::SequentialImpl::forward<at::Tensor, at::Tensor&>(at::Tensor&&&) + 94 (0x11f9c86ce in _C_tests.cpython-36m-darwin.so)
frame #9: vision::models::AlexNetImpl::forward(at::Tensor) + 68 (0x11f9d5e94 in _C_tests.cpython-36m-darwin.so)
frame #10: at::Tensor forward_model<vision::models::AlexNet>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor) + 107 (0x11f9ae76b in _C_tests.cpython-36m-darwin.so)
frame #11: forward_alexnet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor) + 49 (0x11f9ae641 in _C_tests.cpython-36m-darwin.so)
frame #12: at::Tensor pybind11::detail::argument_loader<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor>::call_impl<at::Tensor, at::Tensor (*&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor), 0ul, 1ul, pybind11::detail::void_type>(at::Tensor (*&&&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor), pybind11::detail::index_sequence<0ul, 1ul>, pybind11::detail::void_type&&) + 42 (0x11f9b931a in _C_tests.cpython-36m-darwin.so)
frame #13: void pybind11::cpp_function::initialize<at::Tensor (*&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor), at::Tensor, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor, pybind11::name, pybind11::scope, pybind11::sibling, char [16]>(at::Tensor (*&&&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor), at::Tensor (*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, at::Tensor), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&, char const (&) [16])::'lambda'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const + 89 (0x11f9b8a29 in _C_tests.cpython-36m-darwin.so)
frame #14: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 3392 (0x11f9ba6e0 in _C_tests.cpython-36m-darwin.so)
<omitting python frames>

cc @ShahriarSS

The problem is most likely in tests bindings and not in the models themselves. I'll look into it.

I recently tried running the tests on macOS. All pass except:

MNASNet0_5
MNASNet0_75
MNASNet1_3

Error message:
test/test_cpp_models.py:136 (Tester.test_mnasnet1_3)
self = <test_cpp_models.Tester testMethod=test_mnasnet1_3>

    def test_mnasnet1_3(self):
>       process_model(models.mnasnet1_3(self.pretrained), self.image, _C_tests.forward_mnasnet1_3, 'MNASNet1_3')

test/test_cpp_models.py:138:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

model = MNASNet(
  (layers): Sequential(
    (0): Conv2d(3, 40, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)...Sequential(
    (0): Dropout(p=0.2, inplace=True)
    (1): Linear(in_features=1280, out_features=1000, bias=True)
  )
)
tensor = tensor([[[[0.0902, 0.1059, 0.1059,  ..., 0.2863, 0.2667, 0.2784],
          [0.1059, 0.0980, 0.1059,  ..., 0.3098, 0.3...41, 0.1059, 0.0824,  ..., 0.0667, 0.0784, 0.0706],
          [0.0902, 0.1059, 0.0784,  ..., 0.0667, 0.0706, 0.0667]]]])
func = <built-in method forward_mnasnet1_3 of PyCapsule object at 0x7fcba9eb2d50>
name = 'MNASNet1_3'

    def process_model(model, tensor, func, name):
        model.eval()
        traced_script_module = torch.jit.trace(model, tensor)
        traced_script_module.save("model.pt")

        py_output = model.forward(tensor)
>       cpp_output = func("model.pt", tensor)
E       RuntimeError: Given groups=32, expected weight to be divisible by 32 at dimension 0, but got weight of size [[40, 1, 3, 3]] instead

test/test_cpp_models.py:22: RuntimeError

@fmassa Give the deprecation of the C++ model classes, does it make sense to close the issue?

I was able to reproduce this issue on a linux box as well

I recently tried running the tests on macOS. All pass except:
* MNASNet0_5

* MNASNet0_75

* MNASNet1_3

Error message:
test/test_cpp_models.py:136 (Tester.test_mnasnet1_3)
self = <test_cpp_models.Tester testMethod=test_mnasnet1_3>

    def test_mnasnet1_3(self):
>       process_model(models.mnasnet1_3(self.pretrained), self.image, _C_tests.forward_mnasnet1_3, 'MNASNet1_3')

test/test_cpp_models.py:138:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

model = MNASNet(
  (layers): Sequential(
    (0): Conv2d(3, 40, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)...Sequential(
    (0): Dropout(p=0.2, inplace=True)
    (1): Linear(in_features=1280, out_features=1000, bias=True)
  )
)
tensor = tensor([[[[0.0902, 0.1059, 0.1059,  ..., 0.2863, 0.2667, 0.2784],
          [0.1059, 0.0980, 0.1059,  ..., 0.3098, 0.3...41, 0.1059, 0.0824,  ..., 0.0667, 0.0784, 0.0706],
          [0.0902, 0.1059, 0.0784,  ..., 0.0667, 0.0706, 0.0667]]]])
func = <built-in method forward_mnasnet1_3 of PyCapsule object at 0x7fcba9eb2d50>
name = 'MNASNet1_3'

    def process_model(model, tensor, func, name):
        model.eval()
        traced_script_module = torch.jit.trace(model, tensor)
        traced_script_module.save("model.pt")

        py_output = model.forward(tensor)
>       cpp_output = func("model.pt", tensor)
E       RuntimeError: Given groups=32, expected weight to be divisible by 32 at dimension 0, but got weight of size [[40, 1, 3, 3]] instead

test/test_cpp_models.py:22: RuntimeError

I was able to reproduce the same tests failing on a linux box as well

authors: ShahriarSS ameroueh datumbox fmassa

repo: pytorch/vision