text:
Increasing package size limit for apache-flink

Hi, I would like to request you to increase the size limit of the package "apache-flink" to 400MB, because in the next release, apache-flink will package the Flink Cord JARs files and opt(connector's JRAs), in that case the size will be increased almost 400MB. Please help me increase the upload limit, thank you. There is a developmental release in both PyPI and Test PyPI for the moment, but it is incomplete. It's actual just a test release package.
A link to your project on PyPI (or Test PyPI):
https://test.pypi.org/project/apache-flink/
https://pypi.org/project/apache-flink/
The size of your release, in megabytes:
Almost 400MB (After official release）
Which index/indexes you need the increase for (PyPI, Test PyPI, or both)：
Both
A brief description of your project, including the reason for the additional size.
Apache-Flink allows you to write Flink programs in Python. The additional size comes from the flink JARs files.
BTW：The discussion and vote mail thread can be found here: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-td31201.html

I appreciate if you can have look at this apply @jamadden @nlhkabu
Best, Jincheng

There is a previous distribution of apache flink on PyPI. In https://github.com/pypa/warehouse/issues/6116 it was given a limit of 300MB, though it appears that was never used. If this project is to officially replace that one, perhaps there could be some coordination to remove it from PyPI so as to prevent user confusion and excess resource usage.
300MB is on the very upper end of sizes that moderators are suggested to approve without consulting with other moderators. 400MB is in the area that requires more scrutiny. In particular, if "the package is attempting to ship ... non-Python stuff like OpenJDK" moderators are suggested to seek further clarification from the requestor and discuss with the other moderators.
Besides the PyPi resource usage, large packages don't produce a very good user experience, so we like to help maintainers find alternate solutions.
@sunjincheng121 could you please provide more details about why this project needs to be so large? Were any other distribution options considered and rejected (and if so, why)? For example, one suggestion is that the package could "side-load large models or datasets via downloads from elsewhere at runtime/installtime." Or, since you mention multiple different JARs, another option might be to package them independently and "break the package into smaller packages that depend on each other as necessary."

Hi @jamadden, I am very sorry that I did not explain the relationship between https://pypi.org/project/pyflink/ and https://pypi.org/project/apache-flink/.
Before we did the Flink community discussion, we wanted to use pyflink as the release project, but pyflink was created by other people not related to Apache Flink. After discussions in the Flink community, we finally decided to create a new apach-flink project as the official release.(The project(pyflink) is not part of Flink, we can not delete it for now.)
Regarding the size of the package, I am worried about the future expansion, so I applied for 400MB. It is also possible to apply 300MB at present. Other packages if needed in the future, such as your suggestions, we can also use the script to download dynamically.
So a brief summary:

https://test.pypi.org/project/apache-flink/ is an official project of Apache Flink (on behalf of the Flink PMC)
Apply for package size up to 300MB

If there is anything I do not explanation clearly, please let me know :)
Best, Jincheng

@jamadden @nlhkabu is there something I should explain? :)

@jamadden @sunjincheng121 Thanks a lot for the discussion. Any progress? @jamadden Do you think it makes sense to increase the package size for now and we can continue to investigate other options such as breaking the package into a few smaller packages per your suggestion.

@jamadden @nlhkabu @di could you please have look at this application, Is there any suggestions for me to explain or improve?

My apologies for the delay, my time for this has been extremely limited lately.
Thank you for the detailed information. Based on it, I've taken the following actions:

Increase apache-flink to 300MB on both pypi.org and test.pypi.org (larger sizes might need other moderators to take a look)
Restore pyflink to the default limit, reverting https://github.com/pypa/warehouse/issues/6116

Thanks @jamadden !

authors: dianfu jamadden sunjincheng121


text:
Add Travis CI badge to README

Warehouse uses Travis CI to run integration tests. Our README.md file doesn't have a badge at the top saying that our Travis CI builds are passing. You can look at https://shields.io/category/build to see how to add one.

Good First Issue: This issue is good for first time contributors. If you've already contributed to Warehouse, work on another issue without this label instead. If there is not a corresponding pull request for this issue, it is up for grabs. For directions for getting set up, see our Getting Started Guide. If you are working on this issue and have questions, feel free to ask them here, #pypa-dev on Freenode, or the pypa-dev mailing list.

@brainwane Can I work on this issue?

Sure, go ahead @Patil2099!

A longer response for @Patil2099 -- Yes, please go ahead and grab this issue! At least within Warehouse, our rule is: If there is not a corresponding pull request for this issue (visibly linked by issue number/PR number), and it's not marked as assigned to anyone, it is up for grabs. So in cases like this, you don't even need to ask. :) Just comment and say that you're starting to work on it, and link in a comment to your work-in-progress branch once you start.
As always, if you have questions along the way as you work on this, please feel free to ask them here, in #pypa-dev on Freenode, or the pypa-dev mailing list. Thank you!

source: gharchive/issue

repo: pypa/warehouse

license_type: permissive

license_source: github-api


text:
fix parenthesis error in deprecation docs

I'd like for @Manthan03583 to take this one.
https://warehouse.readthedocs.io/application/#historical-context-deprecations has a grammar error in the line

download counts visible in the API: instead, use the Google BigQuery service)

This is a good first contribution for someone who's new to this repository. You open up docs/application.rst and change
- `download counts visible in the API <https://warehouse.readthedocs.io/api-reference/xml-rpc/#changes-to-legacy-api>`_:
  instead, use `the Google BigQuery service <https://packaging.python.org/guides/analyzing-pypi-package-downloads/>`_)

to
- `download counts visible in the API <https://warehouse.readthedocs.io/api-reference/xml-rpc/#changes-to-legacy-api>`_
  (instead, use `the Google BigQuery service <https://packaging.python.org/guides/analyzing-pypi-package-downloads/>`_)

and then open a pull request.

--
Good First Issue for @Manthan03583: For directions for getting set up, see our Getting Started Guide. If you are working on this issue and have questions, feel free to ask them here, #pypa-dev on Freenode, or the distutils-sig.python.org mailing list.

Thank you

mailing list link in developer docs updated #8030

Hi @Manthan03583 - please come into IRC https://webchat.freenode.net/?channels=%23pypa-dev now, if you can.

sorry mam , my internet connection is running slowly now. And I am sorry i mixed both the commits in one pull request.

@Manthan03583
I think you need some basic help using git before you continue. Instead of committing your proposed changes to your master branch, you need to create a new branch and commit the changes to that new branch. I suggest that you read a training kit from
https://github.github.com/training-kit/ (available in several languages)
from start to finish, and that you read
https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/proposing-changes-to-your-work-with-pull-requests
as well. This will probably take you a few days. If you need some more guidance to understand how to make a new branch and create a pull request from it, please take a look at https://zulip.readthedocs.io/en/latest/git/ which may also help you.
Then, after you've done that, please open a new PR in this repository, one that just contains https://github.com/pypa/warehouse/pull/8030/commits/bbeb7d11e9cf4ddd6770d91f0d48ae144d47fa2c .
Thanks.

parenthesis error in deprecation docs fixed #8046


text:
Add help for pasting password in Windows

Following up on https://github.com/pypa/twine/issues/671#issuecomment-674510081 and the subsequent discussion. In short, I think tokens are causing more folks to copy/paste instead of typing by hand, but that doesn't work great in the Windows Command Prompt. It seemed worth documenting, and this felt like an appropriate location.

@di Can you take a look at this, both for the content, and the pending Travis check?

source: gharchive/pull-request


text:
pyproj ImportError : undefined symbol: proj_context_set_ca_bundle_path

Code Sample, a copy-pastable example if possible
A "Minimal, Complete and Verifiable Example" will make it much easier for maintainers to help you:
http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
 jldz9@r10smithryang:~/InSAR_Sentinel_3/Mississippi$ save_qgis.py
Traceback (most recent call last):
  File "/usr/local/home/jldz9/tools/MintPy/mintpy/save_qgis.py", line 17, in <module>
    from mintpy.utils import ptime, readfile, utils as ut
  File "/usr/local/home/jldz9/tools/MintPy/mintpy/utils/utils.py", line 24, in <module>
    from mintpy.utils.utils0 import *
  File "/usr/local/home/jldz9/tools/MintPy/mintpy/utils/utils0.py", line 23, in <module>
    from pyproj import Proj, Transformer
  File "/usr/local/home/jldz9/anaconda3/envs/mintpy/lib/python3.8/site-packages/pyproj/__init__.py", line 49, in <module>
    import pyproj.network
  File "/usr/local/home/jldz9/anaconda3/envs/mintpy/lib/python3.8/site-packages/pyproj/network.py", line 10, in <module>
    from pyproj._network import (  # noqa: F401
ImportError: /usr/local/home/jldz9/anaconda3/envs/mintpy/lib/python3.8/site-packages/pyproj/_network.cpython-38-x86_64-linux-gnu.so: undefined symbol: proj_context_set_ca_bundle_path

Problem description
Pyproj is imported by mintpy function save_qgis.py
When I tried it out it it simply give me the import error, I opened the issue  there   but the author told me it might be the pyproj install issue.
Expected Output
expect to output help page if I just simply input save_qgis.py without any parameters
Environment Information

Output from: pyproj -v

pyproj info:
    pyproj: 3.1.0
      PROJ: 7.2.0
  data dir: /usr/local/home/jldz9/anaconda3/envs/mintpy/share/proj
user_data_dir: /usr/local/home/jldz9/.local/share/proj

System:
    python: 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05)  [GCC 9.3.0]
executable: /usr/local/home/jldz9/anaconda3/envs/mintpy/bin/python
   machine: Linux-5.4.0-73-generic-x86_64-with-glibc2.10

Python deps:
   certifi: 2021.05.30
       pip: 21.1.1
setuptools: 52.0.0.post20210125
    Cython: None

Output from: python -m pyproj -v

Output from: python -c "import pyproj; pyproj.show_versions()"

pyproj version (python -c "import pyproj; print(pyproj.__version__)")
PROJ version (python -c "import pyproj; print(pyproj.proj_version_str)")
PROJ data directory (python -c "import pyproj; print(pyproj.datadir.get_data_dir())")
Python version (python -c "import sys; print(sys.version.replace('\n', ' '))")
Operation System Information (python -c "import platform; print(platform.platform())")

Installation method

conda, pip wheel, from source, etc...
conda

Conda environment information (if you installed with conda):

Environment (conda list):

$ conda list proj
# packages in environment at /usr/local/home/jldz9/anaconda3/envs/mintpy:
#
# Name                    Version                   Build  Channel
proj                      7.2.0                h277dcde_2    conda-forge
pyproj                    3.1.0            py38h53229fd_3    conda-forge

$conda list
(mintpy) jldz9@r10smithryang:~/InSAR_Sentinel_3/Mississippi$ conda list
# packages in environment at /usr/local/home/jldz9/anaconda3/envs/mintpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
alsa-lib                  1.2.3                h516909a_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
blosc                     1.21.0               h9c3ff4c_0    conda-forge
bokeh                     2.3.2            py38h578d9bd_0    conda-forge
boost-cpp                 1.74.0               hc6e9bd1_3    conda-forge
brotli                    1.0.9                h9c3ff4c_4    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
ca-certificates           2021.5.30            ha878542_0    conda-forge
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
cartopy                   0.19.0.post1     py38hc9c980b_0    conda-forge
cdsapi                    0.5.1              pyhd8ed1ab_0    conda-forge
certifi                   2021.5.30        py38h578d9bd_0    conda-forge
cffi                      1.14.5           py38ha65f79e_0    conda-forge
cfitsio                   3.470                hb418390_7    conda-forge
cftime                    1.5.0            py38hb5d20a5_0    conda-forge
chardet                   4.0.0            py38h578d9bd_1    conda-forge
charls                    2.2.0                h9c3ff4c_0    conda-forge
click                     8.0.1            py38h578d9bd_0    conda-forge
cloudpickle               1.6.0                      py_0    conda-forge
configobj                 5.0.6                      py_0    conda-forge
cryptography              3.4.7            py38ha5dfef3_0    conda-forge
curl                      7.77.0               hea6ffbf_0    conda-forge
cvxopt                    1.2.6            py38h55e5319_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cytoolz                   0.11.0           py38h497a2fe_3    conda-forge
dask                      2021.6.2           pyhd8ed1ab_0    conda-forge
dask-core                 2021.6.2           pyhd8ed1ab_0    conda-forge
dask-jobqueue             0.7.2              pyhd8ed1ab_1    conda-forge
dbus                      1.13.6               h48d8840_2    conda-forge
decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distributed               2021.6.2         py38h578d9bd_0    conda-forge
dsdp                      5.8               hd9d9efa_1203    conda-forge
eccodes                   2.21.0               ha0e6eb6_0    conda-forge
expat                     2.4.1                h9c3ff4c_0    conda-forge
fasteners                 0.16               pyhd8ed1ab_0    conda-forge
fftw                      3.3.9           nompi_h74d3f13_101    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
freexl                    1.0.6                h7f98852_0    conda-forge
fsspec                    2021.6.1           pyhd8ed1ab_0    conda-forge
gdal                      3.2.1            py38hc0b2d6b_7    conda-forge
geos                      3.9.1                h9c3ff4c_2    conda-forge
geotiff                   1.6.0                h2b14fbe_4    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glib                      2.68.3               h9c3ff4c_0    conda-forge
glib-tools                2.68.3               h9c3ff4c_0    conda-forge
glpk                      4.65              h9202a9a_1004    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gsl                       2.6                  he838d99_2    conda-forge
gst-plugins-base          1.18.4               hf529b03_2    conda-forge
gstreamer                 1.18.4               h76c114f_2    conda-forge
h5py                      2.10.0          nompi_py38h9915d05_106    conda-forge
hdf4                      4.2.15               h10796ff_3    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
imagecodecs               2021.3.31        py38h1455ab2_0    conda-forge
imageio                   2.9.0                      py_0    conda-forge
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
json-c                    0.15                 h98cffda_0    conda-forge
jxrlib                    1.1                  h7f98852_2    conda-forge
kealib                    1.4.14               hcc255d8_2    conda-forge
kiwisolver                1.3.1            py38h1fd1430_1    conda-forge
krb5                      1.19.1               hcc1bbae_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
lerc                      2.2.1                h9c3ff4c_0    conda-forge
libaec                    1.0.5                h9c3ff4c_0    conda-forge
libblas                   3.9.0                9_openblas    conda-forge
libcblas                  3.9.0                9_openblas    conda-forge
libclang                  11.1.0          default_ha53f305_1    conda-forge
libcurl                   7.77.0               h2574ce0_0    conda-forge
libdap4                   3.20.6               hd7c4107_2    conda-forge
libdeflate                1.7                  h7f98852_5    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgdal                   3.2.1                h38ff51b_7    conda-forge
libgfortran-ng            9.3.0               hff62375_19    conda-forge
libgfortran5              9.3.0               hff62375_19    conda-forge
libglib                   2.68.3               h3e27bee_0    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libkml                    1.3.0             h238a007_1013    conda-forge
liblapack                 3.9.0                9_openblas    conda-forge
libllvm11                 11.1.0               hf817b99_2    conda-forge
libnetcdf                 4.7.4           nompi_h56d31a8_107    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     13.3                 hd57d9b9_0    conda-forge
librttopo                 1.1.0                h1185371_6    conda-forge
libspatialite             5.0.1                he52d314_3    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libtiff                   4.2.0                hbd63e13_2    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.2.0                h7f98852_2    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.12               h72842e0_0    conda-forge
libxslt                   1.1.33               h15afd5d_2    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
llvm-openmp               8.0.1                hc9558a2_0    conda-forge
locket                    0.2.0                      py_2    conda-forge
lxml                      4.6.3            py38hf1fe3a4_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
markupsafe                2.0.1            py38h497a2fe_0    conda-forge
matplotlib                3.4.2            py38h578d9bd_0    conda-forge
matplotlib-base           3.4.2            py38hcc49a3a_0    conda-forge
metis                     5.1.0             h58526e2_1006    conda-forge
monotonic                 1.5                        py_0    conda-forge
mpfr                      4.0.2                he80fd80_1    conda-forge
msgpack-python            1.0.2            py38h1fd1430_1    conda-forge
mysql-common              8.0.25               ha770c72_0    conda-forge
mysql-libs                8.0.25               h935591d_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
netcdf4                   1.5.6           nompi_py38hf887595_102    conda-forge
networkx                  2.5                        py_0    conda-forge
nspr                      4.30                 h9c3ff4c_0    conda-forge
nss                       3.67                 hb5efdd6_0    conda-forge
numcodecs                 0.7.3            py38h709712a_0    conda-forge
numpy                     1.21.0           py38h9894fe3_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openmp                    8.0.1                         0    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pandas                    1.2.5            py38h1abd341_0    conda-forge
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pillow                    8.2.0            py38ha0e1e83_1    conda-forge
pip                       21.1.2             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pooch                     1.4.0              pyhd8ed1ab_0    conda-forge
poppler                   0.89.0               h2de54a5_5    conda-forge
poppler-data              0.4.10                        0    conda-forge
postgresql                13.3                 h2510834_0    conda-forge
proj                      7.2.0                h277dcde_2    conda-forge
psutil                    5.8.0            py38h497a2fe_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pygrib                    2.1.3            py38h549f6ee_0    conda-forge
pyhdf                     0.10.3           py38he643327_0    conda-forge
pykdtree                  1.3.4            py38h0b5ebd8_0    conda-forge
pykml                     0.2.0                    pypi_0    pypi
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    3.1.0            py38h53229fd_3    conda-forge
pyqt                      5.12.3           py38h578d9bd_7    conda-forge
pyqt-impl                 5.12.3           py38h7400c14_7    conda-forge
pyqt5-sip                 4.19.18          py38h709712a_7    conda-forge
pyqtchart                 5.12             py38h7400c14_7    conda-forge
pyqtwebengine             5.12.1           py38h7400c14_7    conda-forge
pyresample                1.20.0           py38h1abd341_0    conda-forge
pyshp                     2.1.3              pyh44b312d_0    conda-forge
pysocks                   1.7.1            py38h578d9bd_3    conda-forge
pysolid                   0.1.2                    pypi_0    pypi
python                    3.8.10          h49503c6_1_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pywavelets                1.1.1            py38h5c078b8_3    conda-forge
pyyaml                    5.4.1            py38h497a2fe_0    conda-forge
qt                        5.12.9               hda022c4_4    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
scikit-image              0.18.1           py38h51da96c_0    conda-forge
scipy                     1.6.3            py38h7b17777_0    conda-forge
setuptools                49.6.0           py38h578d9bd_3    conda-forge
shapely                   1.7.1            py38haeee4fe_5    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.36.0               h9cd32fc_0    conda-forge
suitesparse               5.10.1               hd8046ac_0    conda-forge
tbb                       2020.2               h4bd325d_4    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
tifffile                  2021.4.8           pyhd8ed1ab_0    conda-forge
tiledb                    2.2.9                h91fcb0e_0    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
toolz                     0.11.1                     py_0    conda-forge
tornado                   6.1              py38h497a2fe_1    conda-forge
tqdm                      4.61.1             pyhd8ed1ab_0    conda-forge
typing_extensions         3.10.0.0           pyha770c72_0    conda-forge
tzcode                    2021a                h7f98852_1    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
urllib3                   1.26.5             pyhd8ed1ab_0    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xarray                    0.18.2             pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.3                h9d8b166_2    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
zarr                      2.8.3              pyhd8ed1ab_0    conda-forge
zfp                       0.5.5                h9c3ff4c_5    conda-forge
zict                      2.0.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge

Details about  conda and system ( conda info ):

$ conda info
(mintpy) jldz9@r10smithryang:~/InSAR_Sentinel_3/Mississippi$ conda info

     active environment : mintpy
    active env location : /usr/local/home/jldz9/anaconda3/envs/mintpy
            shell level : 2
       user config file : /usr/local/home/jldz9/.condarc
 populated config files :
          conda version : 4.10.1
    conda-build version : 3.21.4
         python version : 3.8.8.final.0
       virtual packages : __cuda=9.1=0
                          __linux=5.4.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /usr/local/home/jldz9/anaconda3  (writable)
      conda av data dir : /usr/local/home/jldz9/anaconda3/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /usr/local/home/jldz9/anaconda3/pkgs
                          /usr/local/home/jldz9/.conda/pkgs
       envs directories : /usr/local/home/jldz9/anaconda3/envs
                          /usr/local/home/jldz9/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.1 requests/2.25.1 CPython/3.8.8 Linux/5.4.0-73-generic ubuntu/20.04.2 glibc/2.31
                UID:GID : 73268:5513
             netrc file : /usr/local/home/jldz9/.netrc
           offline mode : False

@jldz9 I would recommend re-creating your conda environment. Whenever I see the undefined symbol error, it usually indicates that pyproj was built against a different version of PROJ than it is running against.

I am going to recommend re-creating your environment and ensuring that you use conda activate <envonment name> before use. If that does not solve your issue, I recommend raising an issue here: https://github.com/conda-forge/pyproj-feedstock/

license: mit

license_source: bigquery


text:
Binaries for v13.3.1 Node 16 (v93)

Hi, looking at https://supabase-public-artifacts-bucket.s3.amazonaws.com/ tha last version uploaded is v13.3.0.
Would you mind adding v13.3.1 for Node 16 v93, both linux x64 and arm64?
Also v14.0.0 and v15.0.0 would be ideal.
This is preventing me to upgrade some dependencies.
Thanks in advance

13.3.1 has been uploaded. The pre-built binaries are an optimization, and should not be a blocker for upgrades--it'll fall back to compiling from source if needed.

repo: pyramation/libpg-query-node

license: MIT


text:
fix: Fixed media content upload to bucket

As witnessed over the past day, we've had some troubles with media upload. Some were fixed in #95 and #96 fortunately, but two issues remained:

adding the subfolder in the bucket name was not supported by the CSP client
empty file could be uploaded, if the byte position of a file were to be offset for instance

This PR fixes both issues by adding a generalized bucket key resolution and resetting the byte position after hashing to avoid uploading empty files. I tried it on my end by deploying the feature branch on Heroku, and it works:

Feel free to try by yourself before the next deployment :ok_hand:
Any feedback is welcome!

Added a unittest for bucket key resolution (after all, I introduced a feature, this should be tested)
And additionally, I think I found the problem with ConnectionError: the tests were run right after the docker started, while the FastAPI server needs some time to run and become reachable :sweat_smile:
I added a workflow fix (moving the docker start as early as possible, and add a sleep before running the tests) and it seems to be doing the trick :raised_hands:

repo: pyronear/pyro-api


text:
Replace log4j with another logger

We don't use logging that extensively in our agent, but due to recent events it seems as though it may be worthwhile to just switch from log4j to use another logger. Would be open to any other suitable logger here.

Considering the amount of attention of security researchers log4j has been enjoying recently, it probably can be considered quite secure now :)

@ivanyu fair enough... We'll consider them on strike two. One more and I think we will have to strongly consider the switch :)

repo: pyroscope-io/pyroscope-java


text:
To evaluate or not to evaluate ... that's the question!

Background
This module is based on coincident orchestration and a great achievement of that module, despite requiring inevitably special headers to allow SharedArrayBuffer, is that no code evaluation ever happens: it's all about proxies, Atomics or standard Web features, without "eval" like security concerns ever.
At the same time, because of those required special headers and to make this module as portable as possible when it comes to workers support, without needing to rely on foreign headers at all when that's desired, this module pre-bundles some file before publishing, also providing a nonce that 3rd party bundlers can use to eventually allow the xworker Blob code whenever CSP rules are both present and rightly strict / paranoid in terms of which code is trusted and which one isn't.
Such nonce is indeed embedded in the package.json as sha-256 value and, for whoever is wondering what is that about indeed, that's the checksum for the ./worker/xworker.js file that the build artifact creates before this module reaches npm (or just to even pass integration tests).
Current State
I had strong discussions against evaluating any code in the worker space because of CSP constraints and possible attached shenanigans, but while I am still fully convinced not evaluating anything in the worker space is actually desirable, automatically defeating tons of different classes of potential attacks to the code, this choice, and my position, is very likely limiting the DX related to hooks and/or the ability to actually pass callbacks to the worker as JS things to do and execute at the right time, like it is already for the main thread story, helped by the fact in there there's no need to survive postMessage related dances, so it's easier to add callbacks to onBeforeRun and friends without ever worry about CSP contraints.
However, while working, thinking, refactoring, and working on other things around this PR, I've wondered myself why shouldn't developers be able to define for either main or workers the very same list of hooks, also providing a way in the worker to deal with the current interpreter before (or after) any code runs at all, enabling tons of features that otherwise need to branch between main and workers all the time ... heck, I even think that PR is wrong in somehow branching main VS thread, as in an ideal world we want to just define these hooks, and stop caring about which one is running where, as long as there is any mean/way to disambiguate the hook is running in a main thread, or a worker one:

onReady
onBeforeRun
onBeforeRunAsync
onAfterRun
onAfterRunAsync
codeBeforeRun
codeBeforeRunAsync
codeAfterRun
codeAfterRunAsync

If we had these normalized no matter where, simply adding some detail to their callbacks when possible (I still think code is about python or other interpreters code, and these already provide a way to disambiguate, as it s for pyscript module, but other hooks can also provide as global IS_WORKER flag if needed), the only hook that'd be left over to date is the onWorkerReady one, because this is a hook that makes sense only on the main thread, as that provides the xworker referece to eventually attach utilities before the worker even runs via sync or anything else we want or need in the future.
In short, I wonder if just having all hooks normalized for all worlds makes sense, but then again, callbacks posted as strings to workers won't be able to carry their own outer scope like it is for main, so that it is still somehow desirable to disambiguate hooks by main one and thread one, where callbacks without outer scope dependencies can work regardless for both cases, but it's still possible to fine-tune those callbacks that wouldn't.
The code story would be the same for both main and thread, so that even in main it would be possible to add code related hooks, instead of just JS one, but basically anyone would be free to actually hook their own code or plugin at any time around the lifecycle of a polyscript script enabled script or custom type ... after all, for workers that would be just enabling extra trust to an already inevitably evaluated part of the project.
What do you you think?
/cc @antocuni @fpliger @ntoll @JeffersGlass @bugzpodder

I've ready 3 times what I wrote and I know some part might feel confusing ... I'll try to hopefully explain some part:
No outer scope in Workers
import { hooks } from '@pyscript/core';

let i = 0;
// this hooks works only in *main*
hooks.onBeforeRun(() => { console.log(i++) });

// in workers, this can be serialized only as fn.toString()
// producing just "() => { console.log(i++) }" as string.
// that `i++` trusts an outer scope entry or a global one
// failing miserably, so that simple hook is not
// cross main/worker usable/reliable

Disambiguation in JS code
To solve previous caveat, we can pass a worker boolean (or any other name) as wrapper utility, asking users to branch out their intents:
import { hooks } from '@pyscript/core';

let i = 0;
hooks.onBeforeRun(({ worker }) => {
  if (worker) {
    // possible workaround
    if ('i' in globalThis) globalThis.i = 0;
    // do we really go down this branching path
    // in user provided hooks, or even our own hooks?
  }
  console.log(i++);
});

The issue here is that the branching burden would be all over the place, for both main or worker only meant code (see py-terminal plugin as example) ... and I personally don't fully like it, but in a world where some project prefer worker or main first, I think this might be a path forward, as it incentives a default choice yet allowing exception to be handled ... I'm a bit torn about this approach though ...
Disambiguation in Python code
The pyscript module, as example, provides a RUNNING_IN_WORKER export boolean value so that it's easy to do different things in there when such value is True or False, and the current py-terminal plugin uses that detail to behave differently ... I felt like that was OK and it solved also the stdlib bundling branching behavior, where we can with ease provide either different exports, with same names, or different behavior, erroring when desired in either main or worker scenarios ... I am also not sure about this, but it worked well to me to date.
That's it, these are the things that make me wonder if my idea makes any sense, if you understand its constraints and/or caveats, and if it would be a better way forward in general, not just for PyScript, but for the sake of anyone using even just polyscript to create amizing things out there!
Thank you for reasoning with me, happy to answer any outstanding question or concern, or happy to listen to even better ideas than this one!

@WebReflection I glanced over the comments and it deserves more time/attention. Just wanted to acknowledge that.
I'll try to do that tomorrow before schedule gets too hectic

Hi, just catching up with all the discussions after last week's travels... :airplane: :earth_africa: :+1:
Here's the key quote from your initial comment @WebReflection:

in an ideal world we want to just define these hooks, and stop caring about which one is running where.

This certainly reduces the cognitive burden on developers, in the same way they always have a windows or document reference, that works in the same way, no matter if they're in the main thread or on a worker.
But I also hear you when say that callbacks run on workers won't have their outer context available to them, because they're serialized to strings via the postMessage to the worker. This is clearly a difference between running on the main thread or on a worker.
So we can only go so far to help folks stop caring about where their code is run.
I believe, for most people, most of the time, and in most situations, this won't be a problem. However, for a significant minority of (likely highly technical) users who are expecting an outer context, we should very clearly flag this difference. That such behaviour is imposed upon us by the browser, for very good security reasons, means we can say something along the lines of, "that's how browsers work, and all we can do is just flag the side effects of this on your code". Given what I hope will be clear documentation and guidance on our part, I hope such technical users will be able to adjust their code to "fit".

so, if we agree on normalizing and branching instead of going full (repeated) separation such as main and worker namespaces, I need to "rewrite" and change a lot of things, but I think whatever decision we make will be the right one, as long as we don't change mind and we are OK in allowing code evaluation in the worker by default ... if anything, this caveat is the only one that keeps me with my feet on the ground believing that maybe @antocuni sugggestion was the best one to reason about, otherwise we also risk to evaluate a lot of code maybe meant to work on main only or on worker only ...
As summary, I do want to allow evaluation on workers too, but maybe we're better off making it explicit as proposed in this MR which can be updated as follow up after this discussion: https://github.com/pyscript/polyscript/pull/58
I do like that MR already, it's surely verbose but it helps separating all concerns and logic around the two very different worlds.

FYI I've updated both description and code, after rebase, for the current proposed MR: https://github.com/pyscript/polyscript/pull/58#issue-1930242841
The more I think about it, the more it feels like the right way to move forward, as it's never ambiguous, it can avoid repeated entries by spreading objects, whenever that's needed, it clearly define a path for callbacks that work on main and those that don't + it allows different Python code too to run in main (in the future, see future proof section of the description).

We've moved the idea forward with latest polyscript so we can close this.

authors: WebReflection fpliger ntoll

repo: pyscript/polyscript


text:
Build better website using github page

Hello i like idea of python IDE pyscripter, when i search the project home page I find blogpost with announcement of release. i like to help build a simple website for pyscripter using static site generators like Hugo and hosted in github page. any interest in this project? I will try to build a sample site and make PR
-sorry if my english is not good, i still try to learn Thanks

Thanks for the offer!   If you have a go and I like the results I will accept you PR.

authors: gunungpw pyscripter


text:
Increase test coverage for requests by checking input params

Given the nature of the package, it is not currently possible to test all the different request methods through automation, since testing responses requires valid Amazon Seller credentials to make a connection. Even if we were able to use Seller credentials to perform an automated test, there is no (last I heard) testing environment on the MWS side, so any request made has potential to wreak havoc in that Seller's production environment.
Rather than try to check MWS's response, we can make testing simpler by stopping short of actually sending a request. All that is really needed is to get either the parameter dict or the request string that a given request method constructs, testing that against pre-set data.

I am thinking the best option is to have an attribute on the base MWS class, something like debug_params, as a boolean. When testing, we just switch it to True, then make_request would be set to return the full param set instead of sending a request. Our test cases for these requests then need to initialize the APIs and switch on this "debug mode" in order to proceed.
This does mean that some code would not be covered, specifically the latter half of make_request that builds a DictWrapper or DataWrapper from the response. We can find some ways to cover that separately later, though, since right now I think covering the request methods is more important.

Framework for this test coverage added by updates ala #54.

repo: python-amazon-mws/python-amazon-mws

license: Unlicense


text:
[WIP] Remove Pipfiles and add pyproject.toml

This PR removes the files required by pipenv and adds poetry's requirements instead.
Using poetry will decrease our build time significantly, and make it much easier to publish projects to PyPi.

What is keeping this WIP?

@sco1 I need to add the appropriate changes to azure-pipelines.yml. I'll see if I can get this done tonight.

Any ideas on this? Why are we changing the pip cache directory anyway?
These warnings also appear on master.
The directory '/home/vsts/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/vsts/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

As it is, the docker image builds and runs just fine. One minor thing is a suggestion from fuzzywuzzy to install python-Levenshtein. Don't know if we care enough to bother. If we do, we should consider making it extra (optional) dependency.
I've experimented with building a wheel using poetry build -f wheel and then installing it in the docker image. However, when installing the wheel, it doesn't respect VCS dependencies and fetches discord.py from PyPI. The same behaviour is exhibited if sdist is used (and it takes way longer to install anyway).
Apart from this dependency issue, it's pretty easy to use a wheel in the image. If it worked, I think it would speed up building the image since poetry wouldn't have to be installed. The time saved doesn't seem to be huge though.

Levenshtein is really slow to install, which is why we decided against it. build times are significantly increased by its inclusion, and its only used for an incredibly rarely used snake feature.

I have some concerns regarding dependencies. We don't actually run the bot in the Lint & Test job; we only run flake8. How many of the base packages are actually needed to install the Python dependencies? Some examples which don't require them to build:

PIL will install fine but the bot will crash if ImageFont is imported and libfreetype isn't installed.
lxml (along with its dependencies libxml2 and xlst) is an extra dependency of bs4. Our code base only seems to make use of html.parser so the dependencies are not needed. They aren't in the Dockerfile anyway.

And there are some other packages which I can't figure out the purpose of:

curl
zlib

Definitely ditch lxml, it's not being used anywhere. You can probably remove git as well, since I think discord.py is installed via http.

Git is needed for git dependencies, Poetry farms out to git with a subprocess call

I'm taking a stab at updating the readme, but Poetry has some differences from pipenv that make this problematic:
Installation of Poetry
The documentation for installation recommends using their custom installer, which is retrieved via curl. Windows users don't have curl by default. There is an alternate installation section which installs it with pip install --user poetry, but it doesn't explain what a user install is. The concern here is that the user site may not be on the PATH.
Run Scripts
Poetry's scripts section isn't the same as pipenv's. Rather, it's more akin to setup.py's scripts. This isn't very useful for us; I don't think it can even run modules directly. A suggestion here https://github.com/sdispater/poetry/issues/241#issuecomment-470200353 is to use a makefile. A makefile is an interesting suggestion, but it'd add make as yet another thing contributors have to install (more of an annoyance on Windows if anything). Another idea pointed out by Joseph was https://github.com/sdispater/poetry/issues/241#issuecomment-445434646.
.env Files
Poetry does not support .env files. There's some some stuff with bash or env that can take care of this, but it's quite cumbersome. We'd need a cross-platform solution anyway. One idea is to have __main__.py try to load the .env file if it exists using python-dotenv.
No PyCharm Support
PyCharm will not automatically create a virtual environment from the Poetry files. However, I believe it will automatically detect an existing virtual environment and automatically add it as an interpreter. It probably only looks in the project's directory so poetry config settings.virtualenvs.in-project true is needed. The guide would need to be shuffled around a bit so that the virtual environment is made before the project is opened in PyCharm.

I understand the logic for not recommending the pip install but ultimately it boils down to the same pitfall as using the wrong Python environment's pip version to install any other module. I don't know if there's a scenario where a successful pip install --user install would fail to provide the correct entry point. Regardless, I've non-user installed it and it works just fine.
Run scripts are definitely an annoying one, for the sake of making folks' lives easier we'll just have to toy around with the proposed alternatives until the point where Poetry provides first class support (if it happens).
Loss of .env support seems well mitigated by python-dotenv.
From seeing some recent issues folks have been having with PyCharm's pipenv support I don't know if the loss of support here is too big of a deal. I don't use PyCharm so I can't attest but it seems like its support can be spotty. Since Poetry does not install to the typical environments folder, users will have to pay attention to the settings.virtualenvs.in-project configuration option, which introduces a pain point, but it shouldn't be too terrible to resolve if someone isn't paying attention and runs into it. Unless poetry respects a project-specific config.toml? The docs are unclear.

I don't know if there's a scenario where a successful pip install --user install would fail to provide the correct entry point.

I had to add the user site to PATH on my machine. The path of least resistance for less experience contributors would be a non-user pip install, but I think that's worth coming to a consensus on before moving forward.

Unless poetry respects a project-specific config.toml

It doesn't seem to, unfortunately.
We need to come to some decisions so this PR can move along. My proposals:

Create a Python script to delegate launch options. It's technically abuse of the scripts feature of Poetry but screw it.
Use python-dotenv to load a .env file if one exists.

Still undecided on how to install Poetry.

Sorry, my first paragraph was bad and wrong

Sorry this half turned into a CI revamp PR. Updates on the situation:
Azure Pipelines
At this point, I think I've done all I can to clean things up. The Lint & Test job has been sped up by almost 40 seconds. Both jobs in total, without actually building the image, have been sped up by over 1.5 minutes. Unfortunately, this has all been due to clean up, not because of Poetry; Poetry is actually about twice as slow to install dependencies compared to pipenv.

All apt-get packages were removed. They were either not used or already came with the Ubuntu agent from MS. Removing the redundant docker especially saved time.
setuptools and salt-pepper were removed, the former of which took especially long to install. They were remnants of a legacy deployment system.

Docker Image
The image's size has been reduced by over 200 MB; it's currently at about 140 MB. This is largely thanks to disabling caches and removing build dependencies once done building. There are some important changes to note, however:

At least Docker Engine 18.06 is required now.
The base image was updated to python:3.7.2-alpine3.9.

There's more info on these changes in the commit messages. These two changes are actually for quite minor reasons, so I am not against rolling them back if someone has a problem with them.

The downside to going with a non-user install is it could potentially require sudo/admin permissions.

@sco1 Good point. Maybe we should go with a user install and leave a "note" about adding the user site to PATH if poetry isn't a recognised command. If so, how extensive would the instructions in this "note" be? We could just model it off pipenv's docs. After all, that's what we were relying on before.

I think it may have been a mistake to cut down on the intermediate layers so much. It looks like docker push/pull only download layers that have changed rather than the whole image. Will have to see what the size of a layer for just poetry installing the dependencies is.

We've decided to stick with pipenv after all.

authors: AnonGuy GhostofGoes MarkKoz heavysaturn sco1

repo: python-discord/seasonalbot


text:
Add a 5 minute cooldown to the topic command

Relevant Issues

Closes #868
Description

Using the command while it's on cooldown will hit the error handler, which sends an error message showing how long is left on the cooldown, which is deleted after 7.5 seconds.

Did you:

[x] Join the Python Discord Community?
[x] Read all the comments in this template?
[x] Ensure there is an issue open, or link relevant discord discussions?
[x] Read and agree to the contributing guidelines?

I'm not a fan of this solution. I think there are relevant reasons to use two topic commands in a five minutes span. I'd propose one of the following:

Reduce to one minute
Add a re-roll reaction that can be only used by the command runner

repo: python-discord/sir-lancebot


text:
automate security headers and add them in google sheets

all sites have security headers and we have to keep track of it so i made a script to automate that work

Hey! I'm just starting to get involved in open source and, based on the tags, thought this might be a good place to start. I'm still a bit confused on what needs to be done though. Do you mind elaborating on the task ?

Hey @avyayjain can you please explain it a Lil bit more

So there is a site called securityheaders.com where you can check how secure the site is and where does the website lacks so I made a script where you just have to enter the website url and you can check the tags

can i make a pr for it

Cool, go ahead

pls check my pr as it is showing flake8 error and greeting but i have pass it in my code but it is showing error for other codes

authors: avyayjain ehildebrandtrojo pawangeek

repo: python-geeks/Automation-scripts


text:
Are relative JSON pointers supported?

Sorry it's me again :smile:
I'm making some progress on my "schema walker" experiments, and wanted to see how relative references using relative JSON pointers were handled.
The docstring of the Resolver class states:
https://github.com/python-jsonschema/referencing/blob/7d6069a7dfa76be679b02587b1ba89a0bfe7348d/referencing/_core.py#L615-L621
However, trying with the following example (and making use of the jsonschema library for demonstration purposes), I get the following:
from jsonschema.validators import Draft7Validator

schema = {
    "$id": "my_schema",
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "properties": {
        "first_email": {
            "type": "string",
            "format": "email"
        },
        "second_email": {
            "$ref": "1/first_email"
        }
    }
}

Draft7Validator(alt).validate({"first_email": "a@example.com", "second_email": "b@example.com"})

I couldn't find anything in the codebase that would handle these relative JSON pointers. I'm also unsure when this was introduced in the json schema spec(s), I couldn't find anything in the changelogs. Thanks in advance!

Hi!
That isn't a valid JSON Schema (and this library of course only deals with references which are valid JSON Schema).
"Relative" in that docstring meant "relative URI" not "relative JSON Pointer" which is what you have there, albeit of course it'd be nice to clarify that there.
More specifically the $ref keyword in all current dialects of JSON Schema does not take a relative JSON pointer. The JSON Schema spec does reference the relative JSON pointer spec, but IIRC just to allow keywords to define things in terms of them, and to have a format: relative-json-pointer format. It does not define any of its own "official dialect keywords" to take relative pointers.
It probably would be nice to have such referencing support here at some point but the initial focus was certainly only on existing JSON Schema dialects.

Interesting! Got misled by this example, reading more about it Opis seems to extand the json schema specification to allow relative pointers. Thanks again

authors: Julian Viicos

repo: python-jsonschema/referencing


text:
handle deprecation of poetry.core.masonry.builder.Builder

fixing deprecation warnings in the unit tests

I think this breaks the Yocto Project usage, where we do not want to build the full poetry but DO want to support the poetry-core PEP-517 backend.
https://git.yoctoproject.org/poky/tree/meta/classes-recipe/python_poetry_core.bbclass
https://git.yoctoproject.org/poky/tree/scripts/lib/recipetool/create_buildsys_python.py#n738

Nevermind. A colleague pointed out that the poetry.core.masonry.builders.wheel is still there and I noticed nothing seems to have changed the "poetry.core.masonry.api" build-backend callout. I tested a build of tomlkit (and ran it's tests) and it was JustFine(TM).

authors: dimbleby moto-timo

repo: python-poetry/poetry-core


text:
Add a CloseCode Enum

I was looking for an Enum containing definitions of the standard close codes, and couldn't find one. Is this something that could be provided to users? I think an IntEnum would be a good way to implement this.
It seemed like this library would be the correct place to implement such an Enum for Python users.
References

https://www.rfc-editor.org/rfc/rfc6455.html#section-7.4.1
https://developer.mozilla.org/en-US/docs/Web/API/WebSocket/close
https://github.com/Luka967/websocket-close-codes
https://learn.microsoft.com/en-us/dotnet/api/system.net.websockets.websocketclosestatus?view=net-8.0

In practice, this would mean turning https://github.com/python-websockets/websockets/blob/main/src/websockets/frames.py#L55-L73 into an Enum and making it a public API, similar to Opcode.

There's a problem, though: extensions can use arbitrary close codes in the 2xxx range and applications can use arbitrary close codes in the 4xxx range, as explained in https://www.rfc-editor.org/rfc/rfc6455.html#section-7.4.2. This cannot be represented with an Enum. How did you plan to handle these cases?

I was thinking about these as similar in use to http.HTTPStatus in the standard library.
They don't cover all possible cases, but they provide self-describing names for the standard codes so that user code is clearer and has few magic numbers that can be prone to typo errors.
So in this case, the actual close() methods would still all take int, but if you as a user explicitly want to use one of these standard codes or compare against them, you have a nice clear, safe way to do that with an IntEnum.
For example:
# before:
close(1002)

# after:
close(WebsocketCloseStatus.ProtocolError)

# before
if status == 1002:

# after
if status == WebsocketCloseStatus.ProtocolError:

For example, in the Django Channels library, I saw a magic 1000 that could be replaced with a more self-describing IntEnum:
https://github.com/django/channels/blob/7e7b71a405db1dbc7509f290b1b158bd07b74c1b/channels/worker.py#L12
User applications can also have these kinds of situations. These applications might also define Enums for custom errors in the 4xxx, but that is out of the scope of something standardized. In these cases they might mix usages of the standard websocket codes with their own Enums.

The difference is that HTTP error codes cover all possible cases and HTTP libraries crash if they get a code that they don't recognize.

In other words, I'm not convinced by:
@property
def close_code(self) -> CloseCode | int:
    """Return a CloseCode or, if not recognized, an int."""

This doesn't look like a convenient API for the caller.

To be clear: I recognize that the problem is valid but I don't think that the proposed solution is an improvement over the current situation.

In other words, I'm not convinced by:

@aaugustin I also do not think any APIs should change and would not advocate for this change. I was thinking that this Enum/IntEnum would be used exclusively by end users when referencing these common codes as shown in https://github.com/python-websockets/websockets/issues/1335#issuecomment-1508392907
To me, this is similar to how most HTTP libraries still use an int for their API for status codes, here's an example from httpx:

https://github.com/encode/httpx/blob/4b5a92e88e03443c2619f0905d756b159f9f0222/httpx/_models.py#L445-L448

But httpx still defines an IntEnum for end users to use when setting/checking the ints that used in the API:

https://github.com/encode/httpx/blob/4b5a92e88e03443c2619f0905d756b159f9f0222/httpx/_status_codes.py#L4

But HTTP status codes are still useful for users of the library to avoid magic numbers when possible.
So, to be clear I would only be advocating for defining an Enum/IntEnum (probably an IntEnum given this is explicitly not exhaustive) in this library. None of the APIs would need to change because, as you show, given that it's not exhaustive. I agree that CloseCode | int is not a useful type.

The difference is that HTTP error codes cover all possible cases and HTTP libraries crash if they get a code that they don't recognize.

Is this true? The HTTP libraries I've seen use int in their APIs and I've always assumed they would pass back the exact int sent over the wire. I suppose we could try sending something like 209 to httpx/requests/http.client.

The difference is that HTTP error codes cover all possible cases and HTTP libraries crash if they get a code that they don't recognize.

I tried out a quick Flask/HTTPX application and non exhaustive HTTP status codes are supported.
# server.py
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return "
Hello, World!
", 209

# client.py
import httpx

response = httpx.get("http://127.0.0.1:5000")
print(response)
print(response.status_code)

flask --app server run

py .\client.py
<Response [209 UNKNOWN]>
209

I'm still skeptical about this. Overall, WebSocket close codes are much less important than HTTP status codes.
In ten years of maintaining this library, I never came across any situation where the WebSocket close code matters 🤷
In practice, all you care about is that the connection dropped. Maybe you'll reopen it. The details are only for debugging.

No problem, definitely understand. I trust your judgement. Thanks for considering. 😄

Maybe you could tell me a bit more about why you wanted this Enum? What's your use case?

I sketched a pull request. Now the question is -- to what extent would it make sense to use this enum in the internal implementation?

PR looks great!

to what extent would it make sense to use this enum in the internal implementation?

My personal opinion is to use them wherever they improve readability. I would personally default to using the Enums unless there was some compelling reason not to since to they express intent, are less prone to small typos, and are self documenting.
But there is no reason you have to use them everywhere.

authors: aaugustin johnthagen

repo: python-websockets/websockets


text:
New worker request: kushaldas-wasi

Username
kushaldas
GitHub username
No response
Name
No response
Email address
mail@kushaldas.in
Password status
I need a new owner password
Processor architecture
amd64
Operating System
Ubuntu 20.04
Anything special about the worker?
For WASI builds, fresh setup.

Hey @kushaldas, thank you for your interest in setting up a new buildbot worker for CPython!
I have updated the buildbot master configuration and I will be emailing you shortly on the email address you specified with your worker password.
Note that you will need to follow up with a PR to add the specific worker!

authors: itamaro kushaldas

repo: python/buildmaster-config


text:
🛑 Chat Api is down

In 8dc9162, Chat Api (https://chat.roblox.com/) was down:

HTTP code: 404
Response time: 121 ms

Resolved: Chat Api is back up in d6070d2 after 12 minutes.

repo: pythoniaweb/roblox-status


text:
🛑 Account Settings is down

In 325d4f0, Account Settings (https://accountsettings.roblox.com/) was down:

HTTP code: 404
Response time: 67 ms

Resolved: Account Settings is back up in 11bd704 after 1 hour, 6 minutes.


text:
Docs should say what's the smallest model users will see a benefit for

I was working on a minimal example to showcase the benefits of fp8 an H100 without forcing users to download a chunky model like here https://github.com/pytorch-labs/float8_experimental/issues/279
I guess it's expected that fp8 will be slower for tiny models because of overhead in which case we should say in docs what's the minimal model size people should try
Training time in FP16: 7.10 seconds
Training time in FP8: 9.80 seconds

import torch
import torch.nn as nn
import copy
from torch.cuda.amp import autocast
from float8_experimental.float8_linear_utils import swap_linear_with_float8_linear
from float8_experimental.float8_dynamic_linear import Float8DynamicLinear
torch.set_float32_matmul_precision('high')

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer1 = nn.Linear(32, 32)
        self.layer2 = nn.Linear(32, 32)

    def forward(self, x):
        x = torch.relu(self.layer1(x))
        x = self.layer2(x)
        return x

def train(model, data_loader):
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters())
    model.train()

    for data, target in data_loader:
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

def benchmark_training(model, data_loader, iterations=100, warmup_iterations=10):
    # Warm-up phase: Run a few iterations to get the GPU to a steady state
    model = torch.compile(model)
    for _ in range(warmup_iterations):
        train(model, data_loader)

    # Timing phase
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    torch.cuda.synchronize()  # Wait for all operations on the CUDA device to complete
    start_event.record()

    for _ in range(iterations):
        train(model, data_loader)

    end_event.record()
    torch.cuda.synchronize()  # Wait for the events to be recorded
    elapsed_time = start_event.elapsed_time(end_event) / 1000.0  # Convert milliseconds to seconds
    return elapsed_time

data_loader = [(torch.randn(32, 32, device="cuda"), torch.randn(32, 1, device="cuda")) for _ in range(110)]

# Initial model setup
base_model = Model().cuda()

# Training in fp16
model_fp16 = copy.deepcopy(base_model)
fp16_time = benchmark_training(model_fp16, data_loader)

# Training in fp8
model_fp8 = copy.deepcopy(base_model)
swap_linear_with_float8_linear(model_fp8, Float8DynamicLinear)
fp8_time = benchmark_training(model_fp8, data_loader)

print(f"Training time in FP16: {fp16_time:.2f} seconds")
print(f"Training time in FP8: {fp8_time:.2f} seconds")

Great idea, let's do it

https://github.com/pytorch/ao/issues/572

authors: msaroufim vkuzo


text:
Add a "no_conversion" flow to torch-tensorrt

Adds a "no_conversion" option to torch-tensorrt which when enabled will replace the standard conversion and engine insertion with an embedded function call for each convertible segment. This allows inspection of the partition without running conversion and the possibility to convert each engine individually in subsequent runs.
Future work:
Allow this flow to run without a GPU to enable TRT convertibility/partitioning linting flows on host machines
Partition without running shape propagation when in the no-convert flow
Fixes # (#1361)
Please delete options that are not relevant and/or add your own.

New feature (non-breaking change which adds functionality)

[ ] My code follows the style guidelines of this project (You can use the linters)

[ ] I have performed a self-review of my own code

[ ] I have commented my code, particularly in hard-to-understand areas and hacks

[ ] I have made corresponding changes to the documentation

[ ] I have added tests to verify my fix or my feature

[ ] New and existing unit tests pass locally with my changes

[ ] I have added the relevant labels to my PR in so that relevant reviewers are notified

@bowang007 Could something like this solve our graph stitching issues? Say if the way partitioning works is it creates a bunch of methods but a method is either 100% PyTorch or 100% TRT. May make things like collections way easier too.

authors: mfeliz-cruise narendasan

repo: pytorch/TensorRT


text:
[Question] MBU in automated CI?

Hi folks, thanks for the great work.
With https://github.com/pytorch/ao/pull/135 merged, vLLM could see benefit from torch.compile backend given compiler-native integration with PagedAttention kernels.
Is there an easy way to see what the latest/nightly MBU is for torch compile on say, H100 / Llama3 70B?
Also interested in cold start compile time
cc @msaroufim

@anijain2305 do we have any benchmark numbers for the cold start compile time?

Related https://github.com/pytorch/pytorch/issues/125958

authors: cadedaniel msaroufim supriyar

repo: pytorch/ao


text:
torchaudio.transforms.Loudness is soooo slow on GPU

🐛 Describe the bug
torchaudio.transforms.Loudness() is painfully slow on a GPU (A100) with, say, 200000 samples.
On CPU it flies.
If this is expected, at least it should be documented and a warning should be given the first time it is run on a GPU.
Versions
Collecting environment information...
PyTorch version: 1.13.1+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31
Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 515.65.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] mypy==0.991
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.4
[pip3] pytorch-lightning==1.9.0
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchmetrics==0.11.0
[pip3] torchview==0.2.5
[pip3] torchvision==0.13.1
[conda] Could not collect

Hi @turian — thanks for creating the issue. Would you mind posting a code snippet that would allow us to reproduce what you're seeing?

import torchaudio
import torch

from tqdm.auto import tqdm

l = torchaudio.transforms.Loudness(sample_rate=44100)
l = l.to("cpu")
x = torch.rand(1, 44100 * 10, device="cpu")
for i in tqdm(list(range(10))):
  l(x)

l = torchaudio.transforms.Loudness(sample_rate=44100)
l = l.to("cuda")
x = torch.rand(1, 44100 * 10, device="cuda")
for i in tqdm(list(range(10))):
  l(x)

The first loop takes under a second.
The second one takes 45 seconds per audio.

@turian thanks! Was able to reproduce on my side. It looks like the execution time is dominated by the for loop within lfilter. Coincidentally, there is a related issue open at https://github.com/pytorch/audio/issues/1408 with a solution landing imminently — perhaps you can follow that issue for updates.

Running the given script with #3018 gives the following;
100%|██████████| 10/10 [00:00<00:00, 42.81it/s]
100%|██████████| 10/10 [00:02<00:00,  3.50it/s]

while on the main branch, it indeed takes forever.
100%|██████████| 10/10 [00:00<00:00, 49.14it/s]

cc @yoyololicon Thanks for your PR, it is very timely.

@hwangjeff @mthrok shall we close?

authors: hwangjeff mthrok turian

repo: pytorch/audio


text:
Mutating a forked element changes element value in all branches

🐛 Describe the bug
When using Forker all child pipelines receive the same object. In case this object is mutable, e.g. a dictionary, modification of the object in one branch, modifies it in all. This leads to unexpected behavior. This is related to #1032, but probably affecting a lot more pipelines as it does not involve nesting.
In the following example, only the dict1 dictionary is (explicitly) modified, but when printing the values of both branches have changed.
import torchdata.datapipes as dp

def to_dict(item):
    return {
        "value": item,
        "metadata": f"item value: {item}",
    }

def add5(value):
    return value + 5

it = dp.iter.IterableWrapper(range(5))

dict1, dict2 = it.map(to_dict).fork(2)

dict1 = dict1.map(add5, input_col="value", output_col="new_value")

for d1, d2 in dict1.zip(dict2):
    print(f"{d1 is d2}, {d1}, {d2}")

output
True, {'value': 0, 'metadata': 'item value: 0', 'new_value': 5}, {'value': 0, 'metadata': 'item value: 0', 'new_value': 5}
True, {'value': 1, 'metadata': 'item value: 1', 'new_value': 6}, {'value': 1, 'metadata': 'item value: 1', 'new_value': 6}
True, {'value': 2, 'metadata': 'item value: 2', 'new_value': 7}, {'value': 2, 'metadata': 'item value: 2', 'new_value': 7}
True, {'value': 3, 'metadata': 'item value: 3', 'new_value': 8}, {'value': 3, 'metadata': 'item value: 3', 'new_value': 8}
True, {'value': 4, 'metadata': 'item value: 4', 'new_value': 9}, {'value': 4, 'metadata': 'item value: 4', 'new_value': 9}

I would expect that the dict2 is not modified and that the d1 is d2  test fails.
Versions
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35
Python version: 3.10.9 | packaged by conda-forge | (main, Feb  2 2023, 20:20:04) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-1030-aws-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   46 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          8
On-line CPU(s) list:             0-7
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
CPU family:                      6
Model:                           106
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1
Stepping:                        6
BogoMIPS:                        5799.91
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd ida arat avx512vbmi pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear flush_l1d arch_capabilities
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       192 KiB (4 instances)
L1i cache:                       128 KiB (4 instances)
L2 cache:                        5 MiB (4 instances)
L3 cache:                        54 MiB (1 instance)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Versions of relevant libraries:
[pip3] mypy==1.0.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.5
[pip3] pytorch-lightning==1.9.2
[pip3] torch==1.13.1
[pip3] torchdata==0.5.1+59663c1
[pip3] torchmetrics==0.11.1
[pip3] torchvision==0.14.1a0+b69fce3
[conda] mkl                       2022.2.1         h84fe81f_16997    conda-forge
[conda] numpy                     1.23.5          py310h53a5b5f_0    conda-forge
[conda] pytorch                   1.13.1          cpu_py310hd11e9c7_1    conda-forge
[conda] pytorch-lightning         1.9.2              pyhd8ed1ab_0    conda-forge
[conda] torchdata                 0.5.1           py310h6e501d2_2    conda-forge
[conda] torchmetrics              0.11.1             pyhd8ed1ab_0    conda-forge
[conda] torchvision               0.14.1          cpu_py310hdfb2906_0    conda-forge

I personally think it's the expected behavior of fork operation. We want to minimize the copy of objects within the pipeline. The existing solution would be creating different DataPipes rather than using fork.
Since there are multiple users asking this feature, we should decide which option is better:

Add a deepcopy argument to make sure fork doing copy on the provided object.
Add a new DataPipe (tentatively names it as replicate) dedicatedly to do copy.

For basic mutable objects, in particular dicts, a shallow copy would already eliminate a lot of pitfalls.
To avoid excessive copying one could only do it for mutable objects.
from collections.abc import MutableMapping

if isinstance(item, MutableMapping):
    item = copy.copy(item)

IMHO at least the behavior should be documented, such that everyone is aware of it.

For basic mutable objects, in particular dicts, a shallow copy would already eliminate a lot of pitfalls.

It depends. If the inner object is something like integer, then yes. Otherwise, it still needs deepcopy.

IMHO at least the behavior should be documented, such that everyone is aware of it.

Totally makes sense. The inline-doc does refer to copy which gives users confusion. Would you love to open a PR to fix it?

Note that, this issue is mostly similar #1032

IMHO at least the behavior should be documented, such that everyone is aware of it.

Totally makes sense. The inline-doc does refer to copy which gives users confusion. Would you love to open a PR to fix it?

I can try to phrase a cautionary note and open an PR for it. Is there a general advice on how to proceed in those cases though? Then I could mention it directly. At the moment I am doing
fork1, fork2 = dp.fork(2)
fork1 = fork1.map(copy.copy)
fork2 = fork2.map(copy.copy)

which solves the problem, but does not read very well.
How about adding a copy_fn to Forker? It can default to None and if needed the user can supply copy, deepcopy or even a custom copy function.

How about adding a copy_fn to Forker? It can default to None and if needed the user can supply copy, deepcopy or even a custom copy function.

If it's acceptable to use a copy_fn, you can do something like:
dp = ...
fork1 = pickle.loads(pickle.dumps(dp))
fork2 = dp

Another option is to do a function to return DataPipe:
def get_dp():
    it = dp.iter.IterableWrapper(range(5))
    dict_dp = it.map(to_dict)
    return dict_dp

fork1 = get_dp()
fork2 = get_dp()

I was thinking that fork can have an additional argument copy_element=False or (deep_copy=False). If it is set to True, it can copy or deep copy the element before yielding it. Thoughts?

I created a PR for pytorch pytorch/pytorch#96030 with a potential interface and a note highlighting the behaviour. Please let me know what you think.

Coming from a cpp background, I'm strongly opposed to making premature copies just to be safe. In my opinion, the best course of action is to highlight this behavior in the documentation and let the user decide if and how he wants to copy objects. Otherwise we needless incur unnecessary and potentially unwanted overhead. Adding another map that does the copying into the pipeline is trivial and also more in-line with torchdata's modular approach IMHO.

@sehoffmann
I don't disagree with you about making premature copies and another map would do the work. But, generally speaking, users expect more syntax sugar when doing python functional programming. So, we end up with fork no-copy by default and detailed comment if copy strategy is provided.
Let us know if you think the doc is not sufficient https://pytorch.org/data/main/generated/torchdata.datapipes.iter.Forker.html?highlight=fork#torchdata.datapipes.iter.Forker

Closing this issue for now as https://github.com/pytorch/pytorch/pull/96030 has been merged. Pls feel free to re-open it if needed.

authors: NivekT ejguan falckt sehoffmann

repo: pytorch/data


text:
Error when trying to train with pipeline parallelism

Hi guys,
I was trying to train a transformer model with pipeline parallelism. Is this supposed to work already?
The command i tried (following the translation example):
fairseq-train     data-bin/iwslt14.tokenized.de-en     --arch transformer_iwslt_de_en_pipeline_parallel --share-decoder-input-output-embed     --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0     --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000     --dropout 0.3 --weight-decay 0.0001     --criterion label_smoothed_cross_entropy --label-smoothing 0.1     --max-tokens 4096     --eval-bleu     --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}'     --eval-bleu-detok moses     --eval-bleu-remove-bpe     --eval-bleu-print-samples     --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --pipeline-model-parallel --pipeline-encoder-balance '[8]' --pipeline-encoder-devices '[0]' --pipeline-decoder-balance '[1,6,1]' --pipeline-decoder-devices '[0,1,0]' --pipeline-chunks 1 --distributed-world-size 2
error:
2020-10-23 17:17:08 | INFO | fairseq.tasks.translation | [de] dictionary: 8848 types
2020-10-23 17:17:08 | INFO | fairseq.tasks.translation | [en] dictionary: 6632 types
2020-10-23 17:17:08 | INFO | fairseq.data.data_utils | loaded 7283 examples from: data-bin/iwslt14.tokenized.de-en/valid.de-en.de
2020-10-23 17:17:08 | INFO | fairseq.data.data_utils | loaded 7283 examples from: data-bin/iwslt14.tokenized.de-en/valid.de-en.en
2020-10-23 17:17:08 | INFO | fairseq.tasks.translation | data-bin/iwslt14.tokenized.de-en valid de-en 7283 examples
Traceback (most recent call last):
  File "/secondary/thies/.virtualenvs/pytorch-23102020/bin/fairseq-train", line 33, in <module>
    sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-train')())
  File "/tertiary/thies/fairseq/fairseq_cli/train.py", line 352, in cli_main
    distributed_utils.call_main(cfg, main)
  File "/tertiary/thies/fairseq/fairseq/distributed_utils.py", line 301, in call_main
    cfg.distributed_training.distributed_world_size,
  File "/secondary/thies/.virtualenvs/pytorch-23102020/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 247, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/secondary/thies/.virtualenvs/pytorch-23102020/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 205, in start_processes
    while not context.join():
  File "/secondary/thies/.virtualenvs/pytorch-23102020/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 166, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/secondary/thies/.virtualenvs/pytorch-23102020/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/tertiary/thies/fairseq/fairseq/distributed_utils.py", line 283, in distributed_main
    main(cfg, **kwargs)
  File "/tertiary/thies/fairseq/fairseq_cli/train.py", line 74, in main
    model = task.build_model(cfg.model)
  File "/tertiary/thies/fairseq/fairseq/tasks/translation.py", line 327, in build_model
    model = super().build_model(args)
  File "/tertiary/thies/fairseq/fairseq/tasks/fairseq_task.py", line 548, in build_model
    model = models.build_model(args, self)
  File "/tertiary/thies/fairseq/fairseq/models/__init__.py", line 56, in build_model
    return ARCH_MODEL_REGISTRY[cfg.arch].build_model(cfg, task)
  File "/tertiary/thies/fairseq/fairseq/model_parallel/models/pipeline_parallel_transformer/model.py", line 277, in build_model
    checkpoint=args.pipeline_checkpoint,
  File "/tertiary/thies/fairseq/fairseq/model_parallel/models/pipeline_parallel_transformer/model.py", line 57, in __init__
    + [encoder.final_layer_norm]
  File "/secondary/thies/.virtualenvs/pytorch-23102020/lib/python3.6/site-packages/torch/nn/modules/module.py", line 796, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'TransformerEncoder' object has no attribute 'embedding_layer'

For training, the a single Pipe() module needs to be created for the Transformer encoder-decoder model. So, you need to set --pipeline-balance and --pipeline-devices in the training command, instead of --pipeline-encoder-balance, --pipeline-encoder-devices, --pipeline-decoder-balance, --pipeline-decoder-devices.
For inference/generation, two Pipe() modules are created, one for the encoder and one for the decoder, since the encoder and decoder are called separately during generation. So, in that case, you need to set --pipeline-encoder-balance, --pipeline-encoder-devices, --pipeline-decoder-balance, --pipeline-decoder-devices instead.

Awesome, works now.
Thank you very much.

repo: pytorch/fairseq


text:
RecursionError: maximum recursion depth exceeded in comparison

I use ort like this:
...
model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
model = ORTModule(model)
model = nn.parallel.DistributedDataParallel(model, find_unused_parameters=True, device_ids=[device])
...

But found error:
Traceback (most recent call last):
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/users/min.du/hdlt/feature_j5fsd_configs/HDLT/hdlt/engine/ddp_trainer.py", line 156, in _main_func
    main_func(local_rank, *args)
  File "/home/users/min.du/hdlt/feature_j5fsd_configs/HDLT/tools/train.py", line 163, in train_entrance
    trainer.fit()
  File "/home/users/min.du/hdlt/feature_j5fsd_configs/HDLT/tools/trainer_wrapper.py", line 225, in fit
    self._trainer.fit()
  File "/home/users/min.du/hdlt/feature_j5fsd_configs/HDLT/hdlt/engine/trainer.py", line 298, in fit
    profiler=self.profiler,
  File "/home/users/min.du/hdlt/feature_j5fsd_configs/HDLT/hdlt/engine/processors/processor.py", line 265, in __call__
    model_outs = model(*_as_list(batch_i))
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 705, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/ortmodule.py", line 41, in _forward
    return self._execution_manager(self._is_training()).forward(*inputs, **kwargs)
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/_training_manager.py", line 67, in forward
    build_gradient_graph = self._export_model(*inputs, **kwargs)
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/_graph_execution_manager.py", line 206, in _export_model
    schema = _io._extract_schema({'args': copy.copy(inputs), 'kwargs': copy.copy(kwargs)})
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/_io.py", line 300, in _extract_schema
    data[key] = _extract_schema(data[key])
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/_io.py", line 291, in _extract_schema
    data[idx] = _extract_schema(data[idx])
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/_io.py", line 291, in _extract_schema
    data[idx] = _extract_schema(data[idx])
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/_io.py", line 291, in _extract_schema
    data[idx] = _extract_schema(data[idx])
  [Previous line repeated 949 more times]
  File "/home/users/min.du/venvs/pytorch1.8/lib/python3.6/site-packages/onnxruntime/training/ortmodule/_io.py", line 287, in _extract_schema
    if isinstance(data, abc.Sequence):
  File "/home/users/min.du/venvs/pytorch1.8/lib64/python3.6/abc.py", line 184, in __instancecheck__
    if subclass in cls._abc_cache:
  File "/home/users/min.du/venvs/pytorch1.8/lib64/python3.6/_weakrefset.py", line 75, in __contains__
    return wr in self.data
RecursionError: maximum recursion depth exceeded in comparison

Any suggestion?

Any chance your input has strings in it? and such string would have 949 chars

Can you retest? https://github.com/microsoft/onnxruntime/pull/8098 may have fixed your issue (merged the same day you created the issue)

@DuinoDu - after the re-test, if the issue persists, can u pls provide re-producible steps/code scripts, and possibly along with the model? thx

Hi, @DuinoDu without waiting, I cloned pytorch repo, and leverage the UTs in the file for the same:
~/pytorch/torch/testing/_internal/distributed/distributed_test.py
However, I can't reproduce your error with it. (Although there's other error encountered, but it's unrelated. And the fix will be in future release.) Therefore it will be nice if you can provide more details, i.e., small reproducible case if possible, if you still see the issue with the fix (https://github.com/microsoft/onnxruntime/pull/8098).
Thanks.

@DuinoDu FYI - by reverting the fix, I can repro the same exception.
FAILED orttraining_test_ortmodule_api.py::test_input_with_string_exception - RecursionError: maximum recursion depth exceeded in comparison

Hi @DuinoDu, we are closing this issue now, as we believe we have resolved it. Please re-open or create a new issue if you need more assistance. Thank you!

@DuinoDu - please feel free to re-open it if you still have the same issue.
Thanks.

authors: DuinoDu natke thiagocrepaldi ytaous

repo: pytorch/ort