text:
[ENH] remove select_columns from conditional_join

PR Description
Please describe the changes proposed in the pull request:

remove deprecated select_columns function from other functions/tests
use align instead of merge for left/right join
add support for timedelta dtype

This PR builds on #1288 .

PR Checklist

Please ensure that you have done the following:

[x] PR in from a fork off your branch. Do not PR from <your_username>:dev, but rather from <your_username>:<feature-branch_name>.

[x] If you're not on the contributors list, add yourself to AUTHORS.md.

[x] Add a line to CHANGELOG.md under the latest version header (i.e. the one that is "on deck") describing the contribution.

Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

Automatic checks
There will be automatic checks run on the PR. These include:

Building a preview of the docs on Netlify
Automatically linting the code
Making sure the code is documented
Making sure that all tests are passed
Making sure that code coverage doesn't go down.

Relevant Reviewers

Please tag maintainers to review.

@ericmjl

🚀 Deployed on https://deploy-preview-1297--pyjanitor.netlify.app

GitBot Summary of Changes
The pull request includes several changes across multiple files, primarily focusing on enhancing the conditional_join function and updating the select function usage:

In CHANGELOG.md, a new enhancement is added to the list, stating that conditional_join now supports timedelta dtype.

In conditional_join.py, the conditional_join function is updated to support timedelta dtype. The function now supports efficient joins on inequality operators. The function's documentation is updated to reflect these changes. Several examples are added to illustrate the new functionality.

In deconcatenate_column.py, impute.py, move.py, pivot.py, utils.py, io.py, and test_clean_names.py, the usage of select_columns is replaced with select.

In test_conditional_join.py, new test cases are added to verify the correct behavior of the updated conditional_join function.

In xlsx_cells function in io.py, the usage of select_columns is replaced with select.

In test_clean_names.py, the usage of select_columns is replaced with select in the test cases.

In test_conditional_join.py, new test cases are added to verify the correct behavior of the updated conditional_join function, especially with the new support for timedelta dtype.

cc: @samukweku, please check for correctness!

@samukweku I think we're good to merge, right? I've already approved the changes, and all tests pass so far.

@ericmjl yes it is ok to merge. Thanks

authors: ericmjl samukweku

repo: pyjanitor-devs/pyjanitor


text:
deprecate sheetname from xlsx_table

PR Description
Please describe the changes proposed in the pull request:

Deprecate sheet_name from xlsx_table (not required as table names are unique in a workbook)
less number of parameters (filepath/workbook and table, if passed)
do away with the namedtuple approach - stick to for loop (simple enough)

This PR resolves #1338 .

PR Checklist

Please ensure that you have done the following:

[ ] PR in from a fork off your branch. Do not PR from <your_username>:dev, but rather from <your_username>:<feature-branch_name>.

[ ] If you're not on the contributors list, add yourself to AUTHORS.md.

[ ] Add a line to CHANGELOG.md under the latest version header (i.e. the one that is "on deck") describing the contribution.

Do use some discretion here; if there are multiple PRs that are related, keep them in a single line.

Automatic checks
There will be automatic checks run on the PR. These include:

Building a preview of the docs on Netlify
Automatically linting the code
Making sure the code is documented
Making sure that all tests are passed
Making sure that code coverage doesn't go down.

Relevant Reviewers

Please tag maintainers to review.

@ericmjl

🚀 Deployed on https://deploy-preview-1339--pyjanitor.netlify.app

Internal Server Error

Internal Server Error

Merged, as it LGTM!

source: gharchive/pull-request

license: MIT

license_type: permissive

license_source: github-api


text:
move Docker image to ubuntu:18.04

I believe the information below (evidence of builds of 18.04) for both Dockerfile and Dockerfile.min should be all that is required. I've have had to add apt -y install sox (see issue https://github.com/pykaldi/pykaldi/issues/212) although this was required for both ubuntu 16.04 and 18.04..
Evidence of completed 18.04 Dockerfile Build
Processing dependencies for pykaldi==0.1.2
Searching for numpy==1.18.2
Best match: numpy 1.18.2
Adding numpy 1.18.2 to easy-install.pth file
Installing f2py script to /usr/local/bin
Installing f2py3 script to /usr/local/bin
Installing f2py3.6 script to /usr/local/bin

Using /usr/local/lib/python3.6/dist-packages
Finished processing dependencies for pykaldi==0.1.2
Removing intermediate container d2b948194f65
 ---> eb70cb336a44
Successfully built eb70cb336a44
Successfully tagged pykaldi/pykaldi:latest

Evidence of completed 18.04 Dockerfile.min Build
Processing /pykaldi/tools/clif
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.6/dist-packages (from pyclif==0.3) (46.1.3)
Requirement already satisfied: pyparsing>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from pyclif==0.3) (2.4.6)
Requirement already satisfied: protobuf>=3.2 in /usr/local/lib/python3.6/dist-packages/protobuf-3.11.4-py3.6.egg (from pyclif==0.3) (3.11.4)
Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.2->pyclif==0.3) (1.14.0)
Building wheels for collected packages: pyclif
  Building wheel for pyclif (setup.py): started
  Building wheel for pyclif (setup.py): finished with status 'done'
  Created wheel for pyclif: filename=pyclif-0.3-cp36-cp36m-linux_x86_64.whl size=945432 sha256=be88624c5ebaa2a915675dea4c4652c3454a859d186a568ef594e2d61f7a2937
  Stored in directory: /tmp/pip-ephem-wheel-cache-fhr3nawu/wheels/c9/c8/fe/c9af8b4939ce79c213ee1db2d0f8a33473c3b1afea04fa1270
Successfully built pyclif
Installing collected packages: pyclif
Successfully installed pyclif-0.3
Done installing CLIF.
Removing intermediate container c4b526c6b341
 ---> 304d15cc07df
Successfully built 304d15cc07df
Successfully tagged pykaldi/pykaldimin:latest

Workaround for Dockerfile.min Build Failure on "cub" requirement.
Both builds with Dockerfile.min for ubuntu 16.04 and 18.04 failed with the error below:
ln -s openfst-1.6.7 openfst
Configuring ...
Checking compiler g++ ...
Checking OpenFst library in /kaldi/tools/openfst ...
Checking cub library in  ...
***configure failed: Could not find file /cub/cub.cuh:
  you may not have installed cub.  Go to ../tools/ and type
  e.g. 'make cub'; cub is a new requirement. ***

I worked around this by modifying the line ./configure --shared to ./configure --use-cuda=no --shared and the build was complete. I did not see a similar issue with a build from the Dockerfile image.

Thanks. I'll also add the cub install and the sox for completeness.

authors: colinbjohnson vrmpx

repo: pykaldi/pykaldi


text:
The difference in inference time between tch and onnx

Hi,
In my rust code, there is the difference in inference time between tch model and onnx session.
tch model is much faster than onnx session. I wonder why.
Actually, there was little difference between the two in python, but there is a lot in rust.
I have no idea, I really want to solve this problem.
thank you,

If you are using a non-CPU execution provider, ort could be gracefully falling back to run on CPU - see here to debug.

Thank you!
Actually, I'm trying to use ort for inferencing onnx model in Jetson nano orin.
But it doesn't seem to support a non-CPU execution provider. the CPU is only used when my code runs.
Is there any way to bind the GPU in my device, Jetson nano orin? Please let me know.

Microsoft doesn't provide prebuilt binaries for Jetson with CUDA, so you'll have to build ONNX Runtime from source. See here: https://onnxruntime.ai/docs/build/eps.html#nvidia-jetson-tx1tx2nanoxavier

source: gharchive/issue

repo: pykeio/ort


text:
Use alpha tags for syncing with PyLECO

Can we add tags for "pre-releases" / "alpha"-versions, such that PyLECO can state, to which alpha-version it is compliant?

That would be fine with me.

authors: bilderbuchi bmoneke

repo: pymeasure/leco-protocol


text:
windows 10 pipenv.vendor.pexpect has no attribute 'spawn'

On Windows10, Python3.6.5, pipenv 11.10.3, I get this error when I try to start a shell.
It worked fine for a while. I shared the volume to a linux instance (I know) and worked with it there. I think that's when the problems started with my Windows box.
So yeah, I probably shot myself in the foot there, but I don't see a way to start fresh again.
Can I just wipe it out and start fresh somehow?
In any case, here's the traceback:
$ pipenv shell
Traceback (most recent call last):
  File "c:\program files\python36\lib\site-packages\pipenv\core.py", line 2192, in do_shell
    c = pexpect.spawn(
AttributeError: module 'pipenv.vendor.pexpect' has no attribute 'spawn'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files\python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python36\Scripts\pipenv.exe\__main__.py", line 9, in <module>
  File "c:\program files\python36\lib\site-packages\pipenv\vendor\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "c:\program files\python36\lib\site-packages\pipenv\vendor\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "c:\program files\python36\lib\site-packages\pipenv\vendor\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\program files\python36\lib\site-packages\pipenv\vendor\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\program files\python36\lib\site-packages\pipenv\vendor\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "c:\program files\python36\lib\site-packages\pipenv\cli.py", line 607, in shell
    three=three, python=python, fancy=fancy, shell_args=shell_args
  File "c:\program files\python36\lib\site-packages\pipenv\core.py", line 2206, in do_shell
    pew.workon_cmd([workon_name])
  File "c:\program files\python36\lib\site-packages\pipenv\patched\pew\pew.py", line 381, in     workon_cmd
    shell(env, cwd=project_dir)
  File "c:\program files\python36\lib\site-packages\pipenv\patched\pew\pew.py", line 208, in shell
    shell_name = Path(shell).stem
  File "c:\program files\python36\lib\pathlib.py", line 999, in __new__
    self = cls._from_parts(args, init=False)
  File "c:\program files\python36\lib\pathlib.py", line 656, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "c:\program files\python36\lib\pathlib.py", line 640, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Here's the result of pipenv.help minus the pipfile.lock contents because there's a lot.
$ python -m pipenv.help
<details><summary>$ python -m pipenv.help output</summary>

Pipenv version: `'11.10.3'`

Pipenv location: `'C:\\Program Files\\Python36\\lib\\site-packages\\pipenv'`

Python location: `'C:\\Program Files\\Python36\\python.exe'`

Other Python installations in `PATH`:

  - `2.7`: `C:\Python27\python.exe`
  - `3.6`: `C:\Program Files\Python36\python.exe`

  - `3.6.5`: `C:\Program Files\Python36\python.exe`
  - `2.7.13`: `C:\Python27\python.exe`
  - `3.6.5`: `C:\Windows\py.exe`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.16299',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'win32'}
```

System environment variables:

  - `ALLUSERSPROFILE`
  - `APPDATA`
  - `COMMONPROGRAMFILES`
  - `COMPUTERNAME`
  - `COMSPEC`
  - `COMMONPROGRAMFILES(X86)`
  - `COMMONPROGRAMW6432`
  - `EDITOR`
  - `EXEPATH`
  - `HOME`
  - `HOMEDRIVE`
  - `HOMEPATH`
  - `LANG`
  - `LOCALAPPDATA`
  - `LOCALDFS`
  - `LOGONSERVER`
  - `MSYSTEM`
  - `NUMBER_OF_PROCESSORS`
  - `OS`
  - `PATH`
  - `PATHEXT`
  - `PLINK_PROTOCOL`
  - `PROCESSOR_ARCHITECTURE`
  - `PROCESSOR_IDENTIFIER`
  - `PROCESSOR_LEVEL`
  - `PROCESSOR_REVISION`
  - `PROGRAMFILES`
  - `PS1`
  - `PSMODULEPATH`
  - `PUBLIC`
  - `PWD`
  - `PROGRAMDATA`
  - `PROGRAMFILES(X86)`
  - `PROGRAMW6432`
  - `SESSIONNAME`
  - `SHLVL`
  - `SYSTEMDRIVE`
  - `SYSTEMROOT`
  - `TEMP`
  - `TERM`
  - `TERM_PROGRAM`
  - `TERM_PROGRAM_VERSION`
  - `TMP`
  - `UATDATA`
  - `USERDNSDOMAIN`
  - `USERDOMAIN`
  - `USERDOMAIN_ROAMINGPROFILE`
  - `USERNAME`
  - `USERPROFILE`
  - `VBOX_MSI_INSTALL_PATH`
  - `VSCODE_CWD`
  - `VSCODE_IPC_HOOK`
  - `VSCODE_NLS_CONFIG`
  - `VSCODE_NODE_CACHED_DATA_DIR_8980`
  - `VSCODE_PID`
  - `WINDIR`
  - `_`
  - `PYTHONDONTWRITEBYTECODE`
  - `PIP_PYTHON_PATH`

Expected result
Expected the pipenv shell to start.
Actual result
Traceback above.
Steps to replicate
Not sure.

Step 1) uninstall pipenv
Step 2) reinstall it
This should fix the problem, we will have a new release out shortly as well, but let us know if this doesn't help

That worked, thanks. Now I'll try to remember to not use multiple OS's in the same repo!

authors: techalchemy tiarno

repo: pypa/pipenv


text:
Pipenv doesn't respect the PIPENV_PYTHON to select the correct python version.

Hi,
See my following testing on using the env var PIPENV_PYTHON:
Run the following commands from my project folder:
$ pipenv --version
pipenv, version 2018.11.26

$ env | grep PIPENV_PYTHON
PIPENV_PYTHON=/home/werner/.pyenv/shims/python

$ /home/werner/.pyenv/shims/python --version
Python 3.4.9

$ pipenv --three
Creating a virtualenv for this project…
Pipfile: /home/werner/aaa/Pipfile
Using /home/werner/.pyenv/versions/3.7.5/bin/python3 (3.7.5) to create virtualenv…
⠧ Creating virtual environment...Already using interpreter /home/werner/.pyenv/versions/3.7.5/bin/python3
Using base prefix '/home/werner/.pyenv/versions/3.7.5'
New python executable in /home/werner/aaa/.venv/bin/python3
Also creating executable in /home/werner/aaa/.venv/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /home/werner/.pyenv/versions/3.7.5/bin/python3

✔ Successfully created virtual environment!
Virtualenv location: /home/werner/aaa/.venv
Creating a Pipfile for this project…

As you can see, I set the PIPENV_PYTHON the env var, and it's pointing to python 3.4.9, why still the pipenv will use another python version, for my case, it's 3.7.5?
Regards

I think you're looking for export PIPENV_DEFAULT_PYTHON_VERSION=3.7

No, I don't want to use method, though I don't try it either.  But IMO, this method is not portable, I want to let the pipenv use the python version set by pyenv locally/globally.
Regards

Any update? I have the same issue

I'll take a wild guess that --three points to the system Python version and not the one pyenv sets up.
In my case, I have set export PIPENV_PYTHON=$HOME/.pyenv/shims/python and by using pipenv install -d it always uses the version pyenv sets.

PIPENV_PYTHON is not meant to point at a full python path, it is only ever meant to point at a numeric python version (e.g. 3.7. Whatever you set in PIPENV_PYTHON will be appended to the name of the virtualenv created by pipenv, e.g. /path/to/your/.virtualenvs/yourfolder-<somehash>-$PIPENV_PYTHON -- this is meant for use in environments where you intend to have multiple versions of the same virtual environment for a single project, so you might want, for instance:
for pyversion in 2.7 3.6 3.7 3.8;
	do PIPENV_PYTHON=$pyversion pipenv install --dev && \
		PIPENV_PYTHON=$pyversion pipenv run pytest;
done

If you would like to specify a python version, you will have to pass that in directly via --python (e.g. pipenv install --python=/path/to/python). As a side note, pyenv's shim python path is just a shell script that wraps some pyenv internals, so there is no guarantee that will actually resolve to python when pipenv encounters it.  Pipenv bypasses pyenv shims by default and works more successfully if you specify nothing (as it will determine that you are using pyenv and resolve the shim for you -- at least in the next release it will).
I would strongly recommend using only the version number to tell pipenv which python version you are looking for, e.g. 3.8 or 2.7 or whatever version you like, as this will ensure that you don't wind up with mangled environment paths that become orphaned if you misremember them or fail to set the environment up again later.
Hope this helps to clear up how the environment variable works -- if you are still struggling, make sure to use the latest prerelease of pipenv (pip install --upgrade --pre pipenv) as it will provide more of the functionality you are looking for.
Thanks!

@techalchemy
If that's the case, then you need to fix your documentation because it still says otherwise

authors: AlJohri JimiC Matt52 hongyi-zhao smac89 techalchemy


text:
Command line completion takes almost a second

Issue description
On the command line, when I type pipenv  and then press Tab, it takes almost a second for completions to show. This is very slow in comparison to other completions I use.
Expected result
I expect completions to be shown in about one tenth of a second.
Actual result
It takes about 0.75 seconds.
Steps to replicate
Run this a couple of times in Zsh (or the equivalent in another shell):
typeset -F SECONDS=0; env COMMANDLINE="pipenv" _PIPENV_COMPLETE=complete-zsh pipenv > /dev/null; echo $SECONDS
typeset -F SECONDS=0; env COMMANDLINE="pipenv -" _PIPENV_COMPLETE=complete-zsh pipenv > /dev/null; echo $SECONDS
typeset -F SECONDS=0; env COMMANDLINE="pipenv install --" _PIPENV_COMPLETE=complete-zsh pipenv > /dev/null; echo $SECONDS

$ pipenv --support
Pipenv version: '2020.6.2'
Pipenv location: '/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv'
Python location: '/usr/local/Cellar/pipenv/2020.6.2/libexec/bin/python3.8'
Python installations found:

3.8.2: /Users/marlon/.pyenv/versions/3.8.2/bin/python3
3.7.3: /usr/bin/python3
2.7.16: /usr/bin/python2
2.7.16: /usr/bin/python2.7

PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.8.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '19.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT '
                     '2020; root:xnu-6153.121.2~2/RELEASE_X86_64',
 'python_full_version': '3.8.3',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

PATH
MANPATH
LESS_TERMCAP_mb
TERM_PROGRAM
LESS_TERMCAP_md
PYENV_ROOT
LESS_TERMCAP_me
HELPDIR
TERM
SHELL
HOMEBREW_REPOSITORY
TMPDIR
TERM_PROGRAM_VERSION
ZPFX
LESS_TERMCAP_ue
TERM_SESSION_ID
PMSPEC
USER
LS_COLORS
SSH_AUTH_SOCK
PAGER
LESS_TERMCAP_us
AUTOSWITCH_VERSION
_
WORDCHARS
PWD
P9K_SSH
EDITOR
P9K_TTY
LANG
XPC_FLAGS
ZLE_REMOVE_SUFFIX_CHARS
AUTOSWITCH_FILE
XPC_SERVICE_NAME
HOME
SHLVL
HOMEBREW_PREFIX
ZSH_CACHE_DIR
LESS
LOGNAME
VISUAL
LC_CTYPE
LESS_TERMCAP_so
FZF_DEFAULT_COMMAND
INFOPATH
HOMEBREW_CELLAR
DISPLAY
LESS_TERMCAP_se
__CF_USER_TEXT_ENCODING
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:
Debug–specific environment variables:

PATH: /usr/local/Cellar/pipenv/2020.6.2/libexec/tools:/Users/marlon/.pyenv/bin:/Users/marlon/.pyenv/shims:/usr/local/bin:/usr/local/sbin:/Users/marlon/Applications/apache-tomcat-8.5.55/bin:/usr/local/opt/ncurses/bin:/Users/marlon/.zinit/polaris/sbin:/Users/marlon/.zinit/polaris/bin:/Users/marlon/.zinit/polaris/usr/sbin:/Users/marlon/.zinit/polaris/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:.
SHELL: /usr/local/bin/zsh
EDITOR: nano
LANG: en_US.UTF-8
PWD: /var/folders/24/37_vm48s2tz9bhmj1rzffvl40000gn/T/tmp.J1AQqvjP

Pipenv completion is backed by click-completion, which uses a dynamic means to decide the word to complete. With that said, Pipenv will have to be launched for every time you hit TAB to complete.
That is totally an issue of technical choice.

Another issue is that the suggested way of adding the autocompletion (in the readme) is very, very slow and adds at least half a second to your shell startup time, since it has to wait for Python and pipenv to initialize on every shell startup.
There's a much better way to do the autocompletion setup.
Step 1. Write the autocompletion code to a file or your shell init script.
Step 2. Load the autocompletion code into your shell init script.
Example:
I decided to write it directly into the shell init scripts because I don't want to deal with figuring out a place to put the external file. This is what that looks like. I use both ZSH and Bash on my system:
echo '# Add pipenv autocompletion' >> ~/.zshrc
_PIPENV_COMPLETE=zsh_source pipenv >> ~/.zshrc

echo '# Add pipenv autocompletion' >> ~/.bashrc
_PIPENV_COMPLETE=bash_source pipenv >> ~/.bashrc

Run this ONCE. Never again. The code it adds to your shell init scripts is generic and simply says "when user presses tab to autocomplete a pipenv command, grab the result via pipenv". If pipenv ever refactors the autocompletion to use other environment variables, the autocompletion would break and you would have to refresh this code. But that will most likely never happen.

authors: Bananaman frostming marlonrichert


text:
Trigger deprecation warning when find_packages() finds reserved name

Summary of changes
Trigger SetuptoolsDeprecationWarning whenever find_packages() finds
a top-level package whose name is found in reserved package name list,
e.g. "tests".  Installing these packages is probably always wrong,
yet it is still a frequent mistake in packages using setuptools.  This
warning should increase the chance of the mistake being caught before
releasing.
Closes #3243
Pull Request Checklist

[x] Changes have tests
[x] News fragment added in changelog.d/.
(See documentation for details)

I have to admit that I went with the "minimal change" approach. If you prefer it done some other way, please let me know.

I'm sorry but due to health issues I'll have to limit my computer activity for some time (apparently up to 4 weeks), so I won't be able to look into it any time soon.

Sorry to hear that...
I wish you a good recovery.

Thank you very much.

@mgorny I hope you're back to health. I thought I'd check in on this PR and see if you wanted to continue with it?

authors: abravalheri jaraco mgorny

repo: pypa/setuptools

license: mit

license_source: bigquery


text:
Refactor dropdowns to follow WAI recommendations

Closes #3194, closes #2127.
@di, we now just need to add the JS, as per "approach 1" here: https://www.w3.org/WAI/tutorials/menus/flyout/
Note that our class is not called has-submenu as in the example, it's called dropdown__trigger

@nlhkabu The indentation is kinda wonky in the HTML here, maybe you're using tabs and not spaces? Could you fix it?

Fixed @di - thanks for catching it. I think it was because I copy pasted from the WAI page :(

Hi @nlhkabu ! I took a look at this one in Chrome (Version 65.0.3325.124 (Official Build) beta (64-bit)) and it seems like the bottom of one of the menus is getting chopped off when it interacts with the bottom of the page. (The dropdown menu on the top menu item looks fine.)
The same cutoff of the bottom of the menu happens in Firefox also.

@lgh2 I'm not sure if this PR is supposed to fix that clipping issue, we might need to break that out from #3194 into a separate issue for a separate PR.

@nlhkabu I'm not sure if I really understand what needs done here. The tutorial is suggesting that we make the target content appear on a click rather than a hover, which means that we'd have to a) remove the current hover CSS and b) add some JS to handle the click event instead.
However this would not degrade well if folks don't have JS enabled, as they would have no way to open the menus.
Instead we can bind to a focusin/focusout event, which would make the menu appear/disappear when the user tabbed to the button. I think this would suffice, and it would reduce our userbase of "users that can't use dropdowns" to just users that don't use mice, and don't use JS.
I'll push a commit which does this, and you can tell me if that is what you're looking for.

Hi @di.  Reading over the tutorial - I don't think we need to remove the hover CSS, rather, we can have both the CSS and JS (with click) working in parallel.
A couple of problems I found (I'm not sure if they're on my side of yours)

I can't seem to open the top (logged in user menu) dropdown with the keyboard.
The option buttons are working fine, but I can't actually tab through to the sub items

@lgh2 I just pushed a fix for the cut-off problem.  It would be great if you could help us to continue to test this PR :) The criteria is:

Use only your keyboard to navigate around the page
Can you open each dropdown only using your keyboard?
Can you access each of the sub menu items in the dropdown using only your keyboard?
Now using your only mouse - does each dropdown still work as expected?

I can't seem to open the top (logged in user menu) dropdown with the keyboard.

Ah yeah, I think my selector might not be catching this.

The option buttons are working fine, but I can't actually tab through to the sub items

OK, finally figured this out. Reading https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_button_role was a tremendous help.
We need to refactor how we structure these dropdown menus a bit. Only button elements (or things with role="button") are focusable, so we need to make anything we want to tab to a button.
I made the necessary refactoring for one such dropdown in 7777eeb, the main things are:

Put the dropdown__trigger class on the parent nav element, with two children:

a button element that the user would click/hover on
an element with the dropdown__content class

Don't need the extra li elements wrapping the trigger button
Individual links in the dropdowns need to wrap buttons as well

This actually looks pretty good to me (aside from the buttons in the dropdown having default styling):

@nlhkabu Does that all make sense? I think if you take the same approach with the logged-in user dropdown, it should just work. Can you do the same refactoring elsewhere, style the dropdown buttons, and ensure I didn't totally break anything? 🙂

Heads up @nlhkabu I figured out how to do this myself so I think I can wrap this up on my own!

@nlhkabu I think this is pretty much done! The only thing I couldn't figure out was how to properly set the :focus style for the user indicator menu (it's just getting what dropdown__link has right now:

(You'll need to tab to it to get it to have :focus.)

ah great @di!  You beat me to it!!! Let me pull down and tweak and review :man_dancing:

hmmm - it looks like nesting a <button> inside an <a> and vice-versa is not valid HTML.

Investigating alternatives...

@nlhkabu I think https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_button_role indicates that we can just give role=button to an a tag? Perhaps that would work?
FWIW, the reason that links are wrapping buttons and vice versa is because Firefox does support tabbing to links. Chrome supports tabbing to links and buttons, hence all the tabindex="=-1" on the links.

I don't think adding role=button is going to work, as per https://formidable.com/blog/2014/05/08/anchors-buttons-and-accessibility/

For people who really want to stick with , the aria-role="button" attribute looks like an attractive option. If you add the attribute, the element is pretty much a button, right?

Unfortunately, using an ARIA role will get you part of the way there, but it isn’t a complete solution. While screen readers will interpret the link as a button, the element still responds to keyboard input as if it were a link

It appears that the 'not tabbing to links' issue is actually a mac system setting: https://stackoverflow.com/questions/11704828/how-to-allow-keyboard-focus-of-links-in-firefox
I think it is safe to say that any user who wants to use their keyboard extensively would have updated this setting.
I therefore think we should revert this PR to use links where appropriate :)

@di - I've pushed a commit (with styling) as per my previous comment.
The top dropdown is almost working perfectly for me.  The only thing is that when I tab all the way through the list (and then out of the dropdown) the dropdown remains open.
For the 'options' buttons, I am getting a JS error:

Hi @nlhkabu !
I tested the following things from your comment up thread:
Use only your keyboard to navigate around the page
I can navigate around the page using the tab and arrow keys, this works ok on both /manage/projects/ and, with exceptions noted below, on /manage/project/nachos-txt/releases/.
Can you open each dropdown only using your keyboard?
Using Shift + Enter allows me to open the dropdown menu on the upper left hand side (under "welcome back" but not the dropdown menu under options beside the releases on the release management page at /manage/project/nachos-txt/releases/#content. I tried using some different key combinations (Control + Enter, Shift + Enter, Control + Tab), but none of them dropped down the menu.
Can you access each of the sub menu items in the dropdown using only your keyboard?
I can access all the dropdown items under the welcome back, <username> menu, and I can tab through them. I can't drop down the options menu on the manage releases page.
Now using your only mouse - does each dropdown still work as expected?
Using my mouse, everything works as expected.

Thanks @lgh2 this is consistent with what I am seeing on my side :)

@nlhkabu I'm seeing merge conflicts here in case you want to resolve those :)

@brainwane I think this is on my plate right now, I'll be working on this post-launch.

@nlhkabu I squashed all the previous commits here to make it easier for me to resolve the merge conflicts, there was a lot. Sorry if that causes any trouble!

For the 'options' buttons, I am getting a JS error:

I think I've resolved this issue (at least I can't reproduce it now).

The top dropdown is almost working perfectly for me. The only thing is that when I tab all the way through the list (and then out of the dropdown) the dropdown remains open.

I think this should be the expected behavior? E.g. on Github, you can tab out of the menu and it stays open, and the only way to close it is to go back to the parent and "click" it again with the keyboard.
The only thing remaining I think is that sometime since we last worked on this the focus styles became transparent (but only when using the keyboard):

I think this should be the expected behavior? E.g. on Github, you can tab out of the menu and it stays open, and the only way to close it is to go back to the parent and "click" it again with the keyboard.

Works for me!

The only thing remaining I think is that sometime since we last worked on this the focus styles became transparent (but only when using the keyboard):

Updated :)

:tada: thanks for your work on this @di - we got there in the end :man_dancing: :dancer:

authors: brainwane di lgh2 nlhkabu

repo: pypa/warehouse


text:
Settings: Initial settings of ftrack url

Problem
on completely clean database, Centos 7:

run ./tools/run_settings.sh
fill in ftrack url

On save, I've got:

and in console:
>>> loading environments ...
  - for Avalon ...
  - global OpenPype ...
  - for modules ...
*** OpenPype [ 3.3.0-nightly.5 ] -----------------------------------------------
>>> Using OpenPype from [  /home/annatar/3.0  ]
... OpenPype variant:             [  production  ]
... Running OpenPype from:        [  /home/annatar/3.0  ]
... Using mongodb:                [  localhost  ]
... Using Deadline webservice at: [  http://localhost:8082  ]
... Logging to MongoDB:           [  localhost  ]
...   - port:                     [  27017  ]
...   - database:                 [  openpype  ]
...   - collection:               [  logs  ]
...   - user:                     [  <N/A>  ]
*** WRN: >>> { DictImmutableKeysEntity }: [  project_settings/unreal/project_setup Unknown keys in default values: "install_unreal_python_engine"  ]
*** WRN: >>> { ftrack }: [  Couldn't create ftrack session.  ]
==============================
Required "api_key" not specified. Pass as argument or set in environment variable FTRACK_API_KEY.
==============================
Traceback (most recent call last):
  File "/home/annatar/3.0/openpype/modules/ftrack/ftrack_module.py", line 145, in on_system_settings_save
    session = self.create_ftrack_session()
  File "/home/annatar/3.0/openpype/modules/ftrack/ftrack_module.py", line 353, in create_ftrack_session
    return ftrack_api.Session(**session_kwargs)
  File "/home/annatar/3.0/.venv/lib/python3.7/site-packages/ftrack_api/session.py", line 184, in __init__
    'Required "api_key" not specified. Pass as argument or set in '
TypeERROR: Required "api_key" not specified. Pass as argument or set in environment variable FTRACK_API_KEY.

It looks like it is trying to use FTRACK_API_KEY immediately even if it hasn't had a chance to be set. But the URL is set correctly:

[cuID:PYPE-1284]

I believe this was because ftrack was turned on by default, even before we had chance to set it up. @iLLiCiTiT I don't think this is an issue anymore, what's your take on it?

I don't think this is an issue anymore, what's your take on it?

It still happens if you enable ftrack and set url for the first time. But I don't think it's an issue, or is it? It just tells that ftrack module can't update ftrack with settings values because url was changed (or user does not have filled ftrack api key).

I agree. the message is quite clear as well I think. You just need to restart and all is good. I'll close this.

authors: antirotor iLLiCiTiT mkolar

repo: pypeclub/OpenPype


text:
Maya looks: publish paths as an option for look assigner method

Feature
reference: #1559
to have looks more flexible, lets publish path along with the ids and let the artist choose when applying look what method of look assignment it should use.
This will bring it closer to usd/alembics and allow to apply look to data coming from different dccs.
[cuID:2f3nc24]

The paths (or at least the long names) are actually already stored in the look .json files currently. For example:
{
  "attributes": [],
  "relationships": {
    "shading_engine_MTLSG": {
      "uuid": "627bbd3a632436a203ff0ef8:a6684026b4a1",
      "members": [{
        "name": "|model_01_:_GRP|model_01_:geo_GRP|model_01_:mesh_a_GES|model_01_:mesh_a_GESShape",
        "uuid": "627bbd3a632436a203ff0ef8:9b7a7cba5128"
      }, {
        "name": "|model_01_:_GRP|model_01_:geo_GRP|model_01_:mesh_b_GES|model_01_:mesh_b_GESShape",
        "uuid": "627bbd3a632436a203ff0ef8:f410f33e62ac"
      }]
    },
  }
}

It's currently a json.dump of the "relationships" collected here in CollectLook.

Yes, I think the easiest way would be to drop namespaces on path, treat the root as wildcard (or drop it off completely) and then try to match the path on selected hierarchy.

Is this closed in favor of another issue? Since I believe the implementation for path based look assignments isn't implemented?

Not sure why the ynbot closer this

@iLLiCiTiT @mkolar why does ynbot keep closing this incorrectly?

@iLLiCiTiT @mkolar why does ynbot keep closing this incorrectly

The issue was tagged with false ClickUo ID pointing to different issue which had been already closed. This issue should be now kept since it had been reassigned to new ClickUp task. Thanks @BigRoy for pointing at it!

authors: BigRoy antirotor jakubjezek001 tokejepsen


text:
PEP458: Update TUF repository metadata on project index change

EDIT 2024/06/21: update description; drop discussion points (see https://github.com/pypi/warehouse/pull/15815#issuecomment-2182420372, https://github.com/pypi/warehouse/pull/15815#issuecomment-2182426602)
Part 3 in a series of PRs to integrate Repository Service for TUF (RSTUF) with Warehouse for PEP 458 adoption. Previous PR was https://github.com/pypi/warehouse/pull/15484
This PR adds an async task to the tuf subpackage, in order to capture project changes in TUF repository metadata via the RSTUF API. Relevant Warehouse views -- file upload and project management (remove, yank) -- are updated to trigger the task.
This PR also updates the so far unused render_simple_detail function for this purpose.

Discussion
There are a few open question, for which I'd appreciate input from folks, who are more familiar with Warehouse.
...

Any chance to get some feedback from Warehouse maintainers? Also happy to arrange a call to discuss in person. cc @ewdurbin
On a side-note, we released v1.0.0 of the crypto interface used by RSTUF earlier today, and it includes a "VaultSigner", tailored for signing TUF metadata with a HashiCorp Vault key. :)

Completeness
The PR should trigger TUF metadata updates in all relevant views, but someone needs to confirm. Basically, we need to update TUF metadata whenever a project change would result in a different project simple detail file, e.g. on distribution file upload, file removal, or yank. I wonder, if we could register a db hook for those changes, e.g. after_commit, which would also give us better certainty that the db change isn't rolled back.

It may be worth considering the machinery that handles our CDN cache purges (see warehouse/cache/origin) though it is admittedly opaque and a little convoluted to store "purge keys" on the request for handling after flush. This is woefully undocumented, but has been functionally reliable for a long time.

Async task handling
Metadata update tasks in RSTUF are async. This PR busy polls the RSTUF task status API until the corresponding metadata update task has finished. I'm sure, Warehouse has better ways of dealing with async tasks.

Yes, look for usage of the @tasks.task decorator throughout the codebase.

Just rebased on latest main, resolving conflicts and including some minor refactorings in the original commits for consistency with https://github.com/pypi/warehouse/pull/16098. More significant changes/additions are added on top. The commit history is now a bit evolutionary. Happy to rewrite/squash, if that's a requirement.
cc @ofek, @trishankatdatadog, @dstufft

Completeness
The PR should trigger TUF metadata updates in all relevant views, but someone needs to confirm. Basically, we need to update TUF metadata whenever a project change would result in a different project simple detail file, e.g. on distribution file upload, file removal, or yank. I wonder, if we could register a db hook for those changes, e.g. after_commit, which would also give us better certainty that the db change isn't rolled back.

It may be worth considering the machinery that handles our CDN cache purges (see warehouse/cache/origin) though it is admittedly opaque and a little convoluted to store "purge keys" on the request for handling after flush. This is woefully undocumented, but has been functionally reliable for a long time.

Thanks for the pointer, @ewdurbin! I decided to leave the explicit invocations for now. We can improve this in a follow-up PR.

Async task handling
Metadata update tasks in RSTUF are async. This PR busy polls the RSTUF task status API until the corresponding metadata update task has finished. I'm sure, Warehouse has better ways of dealing with async tasks.

Yes, look for usage of the @tasks.task decorator throughout the codebase.

Thanks, this worked nicely.

Error handling
This means setting up proper retry mechanism for temporary errors, and figuring out user feedback and recovery for permanent errors.

Now that the RSTUF API is called in an async task errors shouldn't affect the normal operation of Warehouse. I suggest to take the PR as is and improve error handling (ignore, retry or notify) in a next iteration.

Completeness
The PR should trigger TUF metadata updates in all relevant views, but someone needs to confirm. Basically, we need to update TUF metadata whenever a project change would result in a different project simple detail file, e.g. on distribution file upload, file removal, or yank. I wonder, if we could register a db hook for those changes, e.g. after_commit, which would also give us better certainty that the db change isn't rolled back.

It may be worth considering the machinery that handles our CDN cache purges (see warehouse/cache/origin) though it is admittedly opaque and a little convoluted to store "purge keys" on the request for handling after flush. This is woefully undocumented, but has been functionally reliable for a long time.

Thanks for the pointer, @ewdurbin! I decided to leave the explicit invocations for now. We can improve this in a follow-up PR.

Async task handling
Metadata update tasks in RSTUF are async. This PR busy polls the RSTUF task status API until the corresponding metadata update task has finished. I'm sure, Warehouse has better ways of dealing with async tasks.

Yes, look for usage of the @tasks.task decorator throughout the codebase.

Thanks, this worked nicely.

(sorry for the noise, I accidentally hit the "close with comment" button)

Just rebased on latest main, resolving conflicts and including some minor refactorings in the original commits for consistency with #16098. More significant changes/additions are added on top. The commit history is now a bit evolutionary. Happy to rewrite/squash, if that's a requirement.
cc @ofek, @trishankatdatadog, @dstufft

Sorry, was on PTO. How can we best help here? For example, would you like us (esp. Donald) to review?

Thanks, @trishankatdatadog! A review by anyone would be amazing. Donald seems to be a great candidate because he  knows Warehouse and TUF extremely well.
As said above, I think, this PR can be merged as is. However, I'd appreciate help with improvements in follow-up PRs (see my comments above about completeness and error handling).

Thanks for the review, @trishankatdatadog!

My high-level Q is: does this work with the JSON Simple API (particularly for PEP 740)?

Great catch! This PR only covers the html simple index, but could be easily extended to also include the json simple index.
AFAICS we'd only have to make sure that

project json simple index is persistent (for some time) like the html simple index is, and
post two TUF targets instead of one.

PEP 691 that TUF uses these target paths:

simple/PROJECT/vnd.pypi.simple.v1.html
simple/PROJECT/vnd.pypi.simple.v1.json

AFAICS we'd only have to make sure that

Got it, thanks. @woodruffw is this correct: did we miss anything?

Thanks for the feedback, @miketheman!
@kairoaraujo, @trishankatdatadog, and friends, please feel free to take over as discussed the other day!

Note: I pushed some comment fixes, but I am still not finished (Work in Progress)

Hey folks,
Ready for another review! 🎉

authors: ewdurbin kairoaraujo lukpueh trishankatdatadog

repo: pypi/warehouse


text:
Is it possible to add the completion?

Hi,
It would be nice if you could autocomplete when you're writing into the interpreter.
For instance:
import sys
sys.p [...]

when you hint TAB, you can see path, prefix and everything it begins by a p from the sys namespace. Cheers,
Damien

Thanks for the suggestion.  That would be a great, feature but I've no idea how much work it might be.  Perhaps we can import an existing module from e.g. ipython to hook up its completion logic.

I am trying to do autocompletion in Pypyjs using Jedi. I am able to import Jedi but when using the auto-complete module, it's giving me Range Error: Maximum call stack size exceeded in the JS console.
Tapan

authors: garaud rfk tapan0007

repo: pypyjs/pypyjs


text:
Fix retry decorator max substitution

Resolves #207

Codecov Report

Merging #208 (3272a3c) into master (dc95053) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master      #208   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           73        73
  Lines         2202      2202
  Branches       308       308
=========================================
  Hits          2202      2202

Impacted Files
Coverage Δ

pypyr/dsl.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc95053...3272a3c. Read the comment docs.

Yes, a good point, I was thought about it, but I have chosen the easiest one...
Personally, I like the first option with max as an extra argument to exec_iteration. It helps to keep the scope of the retry_decorator tidy.

It helps to keep the scope of the retry_decorator tidy.

good point! 👍 Big thank you as always for your excellent work!

authors: Reskov codecov-io yaythomas

repo: pypyr/pypyr


text:
Missing git tag for 0.3.0 release

It would be nice to keep PyPI releases and git tags in sync :)

Oops, sorry!
(And missed the notification of your issue…)
Done: https://github.com/pyrates/biscuits/releases/tag/0.3.0

authors: felixonmars yohanboniface

repo: pyrates/biscuits


text:
Dead Window Ghosts

A window opened over a sea of G_DO_NOT_DRAW leaves behind its afterimage once closed.
Looks like I really do need to clear all screens and windows to G_BLANK before drawing appearances.
If I do this, the differ should still take care of keeping actual drawtimes low.
This would probably be as good an excuse as any to implement Window superclass update methods that then call _update from the subclass.

Finito with ae68184806

repo: pyridine/Roguebud


text:
Implementation of HOAN for mesh classification.

Also, changed self.att_weights initialization to allow attention on messages between skeletons with different features.

The implementation does not match the HOAN implementation from the original paper. So it this PR will be closed.

authors: ajbrent mhajij

repo: pyt-team/TopoModelX


text:
calendar.search-method with timestamp filters yielding too much

this  code not correct return events.
start= 2023-12-01 00:00:00.793241+03:00
end= 2023-12-01 23:59:58.793241+03:00

start = datetime.now(ZoneInfo("Europe/Moscow")).replace(hour=0, minute=0, second=0)
end = start.replace(hour=23, minute=59, second=58)
print(start, end)
for e in c.search(comp_class=Event, start=start, end=end,  sort_keys=("DTSTART",)):
    if e.icalendar_component.get("X-TELEMOST-CONFERENCE"):
        print(Conference(e.icalendar_component, "Europe/Moscow"))

Result -
<Conference uid:'1o1h3lz1v41v9n4agilx3yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v37k5aqz0qnj5yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v42eaqesvjq18yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T15:03:00+03:00' dtend:'2023-12-01T15:06:00+03:00'>
<Conference uid:'1o1h3lz1v32qqkecq84p7yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T00:00:00+03:00' dtend:'2023-12-02T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v37jb80pqkwz9yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T00:00:00+03:00' dtend:'2023-12-02T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v37jdou17koz4yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T00:00:00+03:00' dtend:'2023-12-02T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v37k6id2q50t0yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v399c37e8i7s0yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T00:00:00+03:00' dtend:'2023-12-02T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v41v8u0b3prp6yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T00:00:00+03:00' dtend:'2023-12-02T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v32qsp1riff59yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>
<Conference uid:'1o1h3lz1va12d54vfi7j4yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T09:30:00+03:00' dtend:'2023-12-01T10:00:00+03:00'>
<Conference uid:'1o1h3lz1v39az2eqax4g7yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T05:57:00+03:00' dtend:'2023-12-01T06:00:00+03:00'>
<Conference uid:'1o1h3lz1v42lol7oyhak1yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T14:53:00+03:00' dtend:'2023-12-01T15:23:00+03:00'>
<Conference uid:'1o1h3lz1v42l92t9n91k8yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T14:57:00+03:00' dtend:'2023-12-01T15:27:00+03:00'>
<Conference uid:'1o1h3lz1v421rx9uahkx9yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T13:17:00+03:00' dtend:'2023-12-01T13:18:00+03:00'>
<Conference uid:'1o1h3lz1ux74blqchq460yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T00:00:00+03:00' dtend:'2023-12-02T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v399ejhoh4em9yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-02T00:00:00+03:00' dtend:'2023-12-03T00:00:00+03:00'>
<Conference uid:'1o1h3lz1v43nqe7uk4jx6yandex.ru' timezone:Europe/Moscow dtstart:'2023-12-01T17:48:00+03:00' dtend:'2023-12-01T18:18:00+03:00' >
Problem - not sorted and wrong searching start and stop

Yandex calendar works using the CalDAV protocol.  Your example did not change the resultю. Yes, you can create an account in Yandex Calendar.
Go to the Yandex ID website https://id.yandex.ru/
Log in or register
Go to Security
At the very bottom are the Application passwords
Click on the CalDAV Calendar and create a password
To integrate, you will need a Yandex account login and a previously created password

my code
import logging, caldav
from caldav import Event
from datetime import datetime, UTC
logging.basicConfig(level=logging.DEBUG)
login, token =  login, token
with caldav.DAVClient(url=f'https://{login}:{token}@caldav.yandex.ru') as client:
    principal = client.principal()
    c = principal.calendar(cal_id=cal_id)
    start = datetime.now(ZoneInfo("Europe/Moscow")).replace(hour=0, minute=0, second=0)
    end = start.replace(hour=23, minute=59, second=59)
    print(start, end)
    for e in c.search(comp_class=Event, start=start, end=end,  sort_keys=("DTSTART",)):
        if e.icalendar_component.get("X-TELEMOST-CONFERENCE"):
            print(Conference(e.icalendar_component, "Europe/Moscow"))

I will try to register my own yandex calendar and play with it, but I cannot tell when I will get time for that.

I should also make test code testing the search algorithm with various timezones and see if that may be an issue.  Again, I cannot tell when I will get time for that.

What I learned is that freebusy is not supported in Yandex Calendar. Also, this calendar is constantly changing some event attributes, which is why the token changes even if you don’t change the events at all.

Hi, @tobixen! This code is working!
import logging, caldav
from caldav import Event
from datetime import datetime
from zoneinfo import ZoneInfo
logging.basicConfig(level=logging.DEBUG)
login, token = 'artemismagilov2', 'ifkfyyzfxbtuhewl'
with caldav.DAVClient(url=f'https://{login}:{token}@caldav.yandex.ru') as client:
    principal = client.principal()
    c = principal.calendar()
    dt = datetime.now(ZoneInfo("Europe/Moscow"))
    start = datetime(dt.year, dt.month, dt.day, tzinfo=ZoneInfo("Europe/Moscow"))
    end = start.replace(hour=23, minute=59, second=59)
    print(start, end)
    for e in c.search(comp_class=Event, start=start, end=end, sort_keys=('dtstart',)):
        if e.icalendar_component.get("X-TELEMOST-CONFERENCE"):
            print(Conference(e.icalendar_component, "Europe/Moscow"))

Unable to register account with yandex, it wants to confirm my telephone number, but the SMS doesn't get through :-(

authors: ArtemIsmagilov tobixen

repo: python-caldav/caldav


text:
Add virtual environment (venv) tag

Closes #1701

I'm not sure if this is enough information. Maybe it would be a good idea to have this on the site, where we also go over poetry and/or pipenv?

I'd agree with Xithrius that I'm not sure this tag explains enough at the moment, and may be better off as a page on the site.
For example, there may be editor specific advice people should follow. If someone using Pycharm tried to follow the instructions in this tag they might end up more confused than before.
It could be worth having a tag that explains what a venv is and then just links to other pages for how to use them, maybe including one on our site if we write one.

I agree with @wookie184 , but, why not make a tag that does show how to use venvs and create ones, with a comment on top suggesting for some editors that they have less confusing ways. But, if it's a lot to explain, and there are some examples to show which are important, I would suggest making some page on the pydis website(or the appropriate place) that covers almost everything needed to know about venvs. I wish I could be more of a help, but I have no idea what venv is, how do I use\create one, or what is it for, which is what makes me more excited about this. Hope I'll get to learn it from the amazing coders here!

Completely agree that it's worth mentioning other IDEs that integrate venvs already, else it could prove more confusing. I'm happy to move this to site content, though I think there should be a summary tag to briefly describe their purpose, linking to the site page, which explains how to set them up -- both for PyCharm, VS Code, etc. and manually.

where we also go over poetry and/or pipenv

I imagine packaging should probably be a separate section, or at least subsection, though definitely worth including. It would require users to know about structuring their project first, and have a basic understanding of modules.

This is how the tag currently looks:

would be interested to hear thoughts from others on what they think so far.

I've reworded a few things, just to make it a little more concise. Feel free to make any more suggestions :)
Information to activate from cmd has been added too.

authors: NIRDERIi Xithrius Zedeldi wookie184

repo: python-discord/bot


text:
Initial h-asgi configuration

This is a basic setup for an h-asgi deployment. Me and @MarkKoz got as far as we could without access to any staging environment.
We found 2 major issues with trying to deploy h-asgi in this way. First being a possible issue around conflicting iptables rules with ufw and podman. Where a published port cannot be port forwarded easily. See: https://stackoverflow.com/a/51741599
This should not be an issue if we use only reverse proxies, like nginx. But is something to keep in mind as a possible issue if we ever need a podman container be exposed directly.
Second issue was how to properly implement podman containers in ansible. We did find a lot around how to set up with quotas and memory limits. But it could lead to issues with OOM nuking the download process for podman repeatedly, to the point it corrupts podman to segfault at any command. We can use the Startup*= limits to work around this issue and only download images in a ExecStartPre stage.
Finally, @jchristgit the template you linked has two cgroup definitions. As i have understood you cant stack these, but id like your word on your reason why you had written it like that before removing it.

This PR relates to issue #26

There is no Startup* equivalent for memory limits. I'm not sure if that means the memory limit applies to all stages or only to ExecStart.

There is no Startup* equivalent for memory limits. I'm not sure if that means the memory limit applies to all stages or only to ExecStart.

Currently im looking into separating the downloading of the podman container, as it is not technically needed in the systemd file.

There is no Startup* equivalent for memory limits. I'm not sure if that means the memory limit applies to all stages or only to ExecStart.

Currently im looking into separating the downloading of the podman container, as it is not technically needed in the systemd file.

In that case, probably good to use --pull=never for podman run.

authors: MarkKoz imsofi

repo: python-discord/infra


text:
Image Separater

Script Title - Image Separater
Brief - The script will allow a user to separate different type of images from a directory based on their feature vectors.
Instructions

Create a new folder for your script and file/folder name should be appropriate.
Create a README.md in your folder for program Instructions
add requirements.txt if needed

Programming Language

[x] Python

Happy Coding

Please assign this issue to me.

Can you explain a bit more, Like I have dog and cat images in folder it will separate out both?

Yes, it'll separate out both dog and cat images if we choose cluster parameter as 2. similarly it can also separate humans, animals, tools, etc. But we have define in how many different categories we have to separate manually.

authors: pawangeek vipinkatara

repo: python-geeks/Automation-scripts


text:
Create security_headers.py

Pull Request Template
script name -
Brief about script
Issue no.(must) -
Self Check(Tick After Making pull Request)

[ ] This issue was assigned to me.
[x] One Change in one Pull Request
[ ] My file is in proper folder (Name of folder should be in lowercase with no space in between) (E.g. meet_schedular)
[ ] I am following clean code and Documentation and my code is well linted with flake8.
[ ] I have added README.md and requirements.txt (Include version numbers too e.g. pandas==0.0.1) with my script
[ ] I have used REPO README TEAMPLATE (Necessary)
[ ] Just including required dependencies in requirements.txt (Don't include Python version too)

If issue was not assigned to you Please don't make a PR. It will marked as invalid.

pls check my pr as it is showing flake8 error and greeting but i have pass it in my code but it is showing error for other codes

Hey @avyayjain Linters are fixed now, but please add a README in the folder, related to how this script is working, and change the folder name to "security_header"

Hey @avyayjain see the folder name you're adding change it to add underscore

authors: avyayjain pawangeek


text:
feature: floating image

Hi. Is there any possibility to wrap text around image? Like this:

The short answer is, not presently.
The best short name for this feature might be "floating image", where the picture is placed in an "anchored" graphical frame. It's related in some ways to the existing add_picture() capability, except that the container has these floating properties rather than being inline, where the picture acts in most respects as a (large-ish) character glyph.
I'll leave this issue open as a feature request for this. It's likely to be some time in coming as there are several large features ahead of it in the backlog. Let me know if you'd like to give it a crack yourself.

It would be great if floating images' support could be added.
I'm trying to programmatically write text over image (a pretty standard form, doesn't have to be pretty) and your lib is as close as I got. For now I have to resort to generating two halves of a document and printing them on the same, aligned sheet of paper to achieve the two layers' illusion.

Hi!
I manage to do this inserting the picture on a cell docx table and writing the text on adjacent cells.
e.g.
Create table
table = doc.add_table(rows=1, cols=2)
Style from a docx template document, a table with no borders and no liens
table.style = "branco"
insert picture on the first cell
pic_cells = table.rows[0].cells
paragraph = pic_cells[0].paragraphs[0]
run = paragraph.add_run()
run.add_picture('C:\Users\daniel.oliveira\Desktop\Documentos Daniel\money1.jpg', width=Mm(45), height=Mm(25))
inset text on the second cell
tx_cells = table.rows[0].cells
tx_cells[0].width = -1
tx_cells[1].text = "Test"
doc.save('demo.docx')
you can replicate the image above creating an three columns table and following the same logic

any updates?

Yeah, any updates?

++ Need this feature.
I've readthat some1 acctualy did make the code of it, but that it wasnt implemented.
Solving this with with tables isn't exactly the same aws in when you have more text than the height of the picture it doesnt wraparond it.
Also, you cant add picture this way into header and write on top of it.. something along the lines of "picture.position = behind_text"

Need this feature on an urgent basis! If anybody has a PR for this Please share the link.

An implementation of floating picture (wrapping style = behind text) here https://github.com/dothinking/pdf2docx/issues/54#issuecomment-715925252, should work for "square" mode with a bit of modification.

Any updates? This is very useful! One reason I need to use python-docx is because I do not want to change the image type one by one by hand!

An implementation of floating picture (wrapping style = behind text) here dothinking/pdf2docx#54 (comment), should work for "square" mode with a bit of modification.

changing wp:wrapNone/ to <wp:wrapSquare wrapText = "bothSides"/> will do the work for square text wrap.
the original comment: here dothinking/pdf2docx#54 (comment)

This feature would be much appreciated.

import os
from docx.oxml import parse_xml, register_element_cls
from docx.oxml.ns import nsdecls
from docx.oxml.shape import CT_Picture
from docx.oxml.xmlchemy import BaseOxmlElement, OneAndOnlyOne
from docx import Document
from docx.shared import Inches, Pt

class CT_Anchor(BaseOxmlElement):
    extent = OneAndOnlyOne('wp:extent')
    docPr = OneAndOnlyOne('wp:docPr')
    graphic = OneAndOnlyOne('a:graphic')

    @classmethod
    def new(cls, cx, cy, shape_id, pic, pos_x, pos_y):
        anchor = parse_xml(cls._anchor_xml(pos_x, pos_y))
        anchor.extent.cx = cx
        anchor.extent.cy = cy
        anchor.docPr.id = shape_id
        anchor.docPr.name = 'Picture %d' % shape_id
        anchor.graphic.graphicData.uri = (
            'http://schemas.openxmlformats.org/drawingml/2006/picture'
        )
        anchor.graphic.graphicData._insert_pic(pic)
        return anchor

    @classmethod
    def new_pic_anchor(cls, shape_id, rId, filename, cx, cy, pos_x, pos_y):
        pic_id = 0
        pic = CT_Picture.new(pic_id, filename, rId, cx, cy)
        anchor = cls.new(cx, cy, shape_id, pic, pos_x, pos_y)
        anchor.graphic.graphicData._insert_pic(pic)
        return anchor

    @classmethod
    def _anchor_xml(cls, pos_x, pos_y):
        return (
            '<wp:anchor distT="0" distB="0" distL="0" distR="0" simplePos="0" relativeHeight="0" \n'
            '           behindDoc="1" locked="0" layoutInCell="1" allowOverlap="1" \n'
            '           %s>\n'
            '  <wp:simplePos x="0" y="0"/>\n'
            '  <wp:positionH relativeFrom="page">\n'
            '    <wp:posOffset>%d</wp:posOffset>\n'
            '  </wp:positionH>\n'
            '  <wp:positionV relativeFrom="page">\n'
            '    <wp:posOffset>%d</wp:posOffset>\n'
            '  </wp:positionV>\n'
            '  <wp:extent cx="914400" cy="914400"/>\n'
            '  <wp:wrapNone/>\n'
            '  <wp:docPr id="666" name="unnamed"/>\n'
            '  <wp:cNvGraphicFramePr>\n'
            '    <a:graphicFrameLocks noChangeAspect="1"/>\n'
            '  </wp:cNvGraphicFramePr>\n'
            '  <a:graphic>\n'
            '    <a:graphicData uri="URI not set"/>\n'
            '  </a:graphic>\n'
            '</wp:anchor>' % ( nsdecls('wp', 'a', 'pic', 'r'), int(pos_x), int(pos_y) )
        )

def new_pic_anchor(part, image_descriptor, width, height, pos_x, pos_y):
    rId, image = part.get_or_add_image(image_descriptor)
    cx, cy = image.scaled_dimensions(width, height)
    shape_id, filename = part.next_id, image.filename
    return CT_Anchor.new_pic_anchor(shape_id, rId, filename, cx, cy, pos_x, pos_y)

# refer to docx.text.run.add_picture
def add_float_picture(p, image_path_or_stream, width=None, height=None, pos_x=0, pos_y=0):
    """Add float picture at fixed position `pos_x` and `pos_y` to the top-left point of page.
    """
    run = p.add_run()
    anchor = new_pic_anchor(run.part, image_path_or_stream, width, height, pos_x, pos_y)
    run._r.add_drawing(anchor)

if __name__ == '__main__':
    register_element_cls('wp:anchor', CT_Anchor)
    document = Document("document.docx")

    p = document.add_paragraph()
    add_float_picture(p, 'image_test.png', width=Inches(2.25), pos_x=Pt(400), pos_y=Pt(30))

    p.add_run('Hello World'*50)

    document.save('document.docx')
    os.system("start document.docx")

This worked for me

@Kill0geR this worked perfectly in my case. I oly had to change:
<wp:wrapNone/>
to
<wp:wrapSquare wrapText = "bothSides"/>
in order to set text wrapping to square.
I hope that I will be made easier to do this in the future.

authors: Chrisfufu Daniel-Quistan GaidamakUA Kill0geR MCopperhead cirrus3d dothinking fzyzcjy miksica neube3 pkor sabya14 sachinrevvsales scanny

repo: python-openxml/python-docx


text:
"pip install" should not be executed if no deps

Summary
=======
Variable []
Fixed []

Versions
========

FAIL max: Install failed
  $ /tmp/tmp3ylczuet/bin/python -m pip installERROR: You must give at least one requirement to install (see "pip help install")
  WARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.
  You should consider upgrading via the '/tmp/tmp3ylczuet/bin/python -m pip install --upgrade pip' command.

I can check this out. Can you assign this to me

Here you go!
Steps to reproduce:

Make an empty directory somewhere, I'll call it /tmp/example
Create /tmp/example/setup.py containing these lines, representing a minimal setuptools project that has no dependencies:

from setuptools import setup
setup()

Set up pessimist development environment

cd /path/to/pessimist
make venv
source .venv/bin/activate

4. Run against that directory.

python -m pessimist /tmp/example

The bug is calling `run()` to install packages, when there were no packages that needed to be installed.  This triggers the "Install failed" message.  Those should be some hints to get you started; feel free to ask questions.

Fixed in https://github.com/python-packaging/pessimist/commit/b9cfb1ea1fbfdc5aa419bcae8d9203dfae523305

authors: Antsthebul thatch

repo: python-packaging/pessimist


text:
Decide and implement a versioning scheme

Is your feature request related to a problem? Please describe.
Create a more through versioning scheme with auto incremental version numbers for deployment.

Finally it's done. 😄


text:
Crash when a checkpoint cannot be found.

I'm getting the following error when I restart after installing this plugin
(pysssss:better_combos) Unable to get path for checkpoints v1-5-pruned-emaonly.ckpt
Error handling request
Traceback (most recent call last):
  File "/home/pauls/StableDiffusion/ComfyUI/cpu-only/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pauls/StableDiffusion/ComfyUI/cpu-only/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pauls/StableDiffusion/ComfyUI/cpu-only/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pauls/StableDiffusion/ComfyUI/server.py", line 46, in cache_control
    response: web.Response = await handler(request)
                             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pauls/StableDiffusion/ComfyUI/server.py", line 416, in get_object_info
    out[x] = node_info(x)
             ^^^^^^^^^^^^
  File "/home/pauls/StableDiffusion/ComfyUI/server.py", line 395, in node_info
    info['input'] = obj_class.INPUT_TYPES()
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pauls/StableDiffusion/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts/py/better_combos.py", line 119, in INPUT_TYPES
    populate_items(names, "checkpoints")
  File "/home/pauls/StableDiffusion/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts/py/better_combos.py", line 98, in populate_items
    names.sort(key=lambda i: i["content"].lower())
  File "/home/pauls/StableDiffusion/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts/py/better_combos.py", line 98, in <lambda>
    names.sort(key=lambda i: i["content"].lower())
                             ~^^^^^^^^^^^
TypeError: string indices must be integers, not 'str'

https://github.com/pythongosssss/ComfyUI-Custom-Scripts/blob/27555d4f71bb4e24b87571f89eab2b4a06677bb6/py/better_combos.py#L98
Seems to be that if the code inside the loop cannot find the checkpoint (in my case it was a broken symlink), the entry in names is left as a string rather than a dictionary. The indexing then goes wrong.
Obviously, I'm going to fix the symlink that caused this but seeing as you have that continue code path, it would be good that the code didn't crash if it got taken. My suggestion would be not mutating the original names list, but constructing a new list. However, you'd need to return the new list to the caller, which changes the function signature. Alternatively, delete the item from the list.

@WeeBull I encountered the same problem as you, because my path uses mklink to link, how should I modify it?

@pythongosssss please help~

oh！ mklink is ok！ Just the file link symbol won't work！ 3q！

repo: pythongosssss/ComfyUI-Custom-Scripts


text:
🛑 2FA Api is down

In 7ac8db1, 2FA Api (https://twostepverification.roblox.com/) was down:

HTTP code: 404
Response time: 54 ms

Resolved: 2FA Api is back up in 0888695 after 23 minutes.

repo: pythoniaweb/statusboom


text:
Changing the architecture

How to modify the architecture if my image shape is (160, 60)

First of all, for which tutorial, but basically that's basics, so you need to find it out by yourself

repo: pythonlessons/mltu


text:
Make app looks good on mobile

Clear and concise description of the problem

We are currently able to visit the site even on mobile, but it doesn't look good.
Suggested solution

Make app looks good with left sidebar and generated code in the middle.
Alternative
N/A

Additional context

Is this GitHub bug? Not closing automatically?

Interesting...
but on the right we see:

Successfully merging a pull request may close this issue.
MAY close... :)) so
it's might be a bug)

authors: trsvchn ydcjeff

repo: pytorch-ignite/code-generator


text:
Add 4-bit Adam

From https://github.com/thu-ml/low-bit-optimizers
Results for fine-tuning ViT-H (630M params) with BF16 AMP, batch size 4, 1 epoch, on 4070Ti SUPER:

Adam impl
max memory (GB)
time taken
accuracy

PyTorch
12.98
10m 08s
87.70

bnb 8-bit
8.31
8m 38s
86.22

ao 8-bit
8.32
10m 54s
86.67

lpmm 4-bit
7.72
7m 48s
84.70

ao 4-bit
7.72
9m 17s
85.60

Sanity check:

Model size (FP32) = 0.630 x 4 = 2.52 GB
Optim state in FP32 = 2.52 x 2 = 5.04 GB
Optim state in 8-bit = 5.04 / 4 = 1.26 GB -> 3.78 GB savings
Optim state in 4-bit = 5.04 / 8 = 0.63 GB -> 4.41 GB savings

The branchless binary search example kinda blew my mind so writing a short explanation for others
So let's say we have a tensor and we'd like to cast it to 4 bit and want to search over a codebook that's not uniform with nf4 (normalized float4) being the most popular example. qmap will in the case of a 4 bit optimizer contain 2^ = 16 elements and if you 0 index the largest element is 15
nf4_qmap = [-1.0, -0.6961928009986877, -0.5250730514526367,
-0.39491748809814453, -0.28444138169288635, -0.18477343022823334,
-0.09105003625154495, 0.0, 0.07958029955625534, 0.16093020141124725,
0.24611230194568634, 0.33791524171829224, 0.44070982933044434,
0.5626170039176941, 0.7229568362236023, 1.0]

In this case when quantizing a new parameter instead of going through the entire codebook linearly you can binary search over it. The codebook or qmap has 16 elements and so you'll need 4 binary search steps
The key intuition is listed in this blog https://blog.demofox.org/2017/06/20/simd-gpu-friendly-branchless-binary-search/
With the full algorithm being
input = input.view(-1)
codes = torch.where(input >= qmap[8], 8, 0)
codes += torch.where(input >= qmap[codes + 4], 4, 0)
codes += torch.where(input >= qmap[codes + 2], 2, 0)
codes += torch.where(input >= qmap[codes + 1], 1, 0)

The first step input = input.view(-1) is because as we're casting the elements of the tensor we can treat it as a vector
The second step which is more interesting does codes = torch.where(input >= qmap[8], 8, 0) and what this will do is check if an input value is larger than the midpoint of the qmap (remember it has 16 elements) if it is indeed larger then we know the index we're searching for is at least at 8 so we set codes to 8. Keep in mind that this is also a vectorized operation otherwise we would have called this code
The third step we continue the binary search starting from the previously found index 8 and we can to add it 4 so we're now doing binary search at index 12 in this case let's assume input is not larger than qmap[4+8=12] we add 0 to the index so our codes index is still 8
We go to the third step and and now we search at index 8+0+2 and we keep going
When we're doing a binary search over codebooks that are typically small because we usually quantize to 8 bit (2^8) or 4 bit (2^4) then we can manually unroll the binary search algorithm into 8 or 4 steps respectively.

Adding on about the binary search. I haven't verified myself, but I think apart from time complexity improvement, it is also an improvement in memory usage, which can help with kernel fusion as less shared memory is needed. The naive approach codes = (qmap.view(1, -1) - input.view(-1, 1)).abs().argmin(-1) will required (input_size, codebook_size) memory - or if we tile the input, it will be (tile_size, codebook_size) of shared memory, while the binary search approach only needs (2-5x tile_size) of memory (including rounding logic).

authors: gau-nernst msaroufim

repo: pytorch/ao


text:
Parallel test with pytest-xdist

Currently this PR creates 2 workers, each with 2 GPUs

pytest test --verbose -s --dist load --tx popen//env:CUDA_VISIBLE_DEVICES=0,1 --tx popen//env:CUDA_VISIBLE_DEVICES=2,3

I think only FSDP tests require 2 GPUs
https://github.com/pytorch/ao/blob/4f53882c6dfaf42daad8ca01595f44fe6d51e69c/test/dtypes/test_nf4.py#L484-L494
https://github.com/pytorch/ao/blob/4f53882c6dfaf42daad8ca01595f44fe6d51e69c/test/prototype/test_low_bit_optim.py#L161-L168
Thus, CI runtime can be further improved by

Mark these tests as "require 2 GPUs"
Run the remaining tests with 4 workers, each with 1 GPU
Run the 2-GPU tests in a separate step. Can either be 2 workers with 2 GPUs each, or just normal, non-parallel pytest since there are not a lot of 2-GPU tests right now.

UPDATE: parallel tests with 4 1-GPU workers + sequential multi-GPU tests is implemented now. Some problems I encountered along the way (for future reference and improvement). pytest-xdist provides multiple methods to distribute work across workers. Refer to here for the full list. I have tried the following

--dist load (send each test to next available worker):

TestSaveLoadMeta use the same filename test.pth which is problematic when multiple tests write/read the same file. I made a unique name for each test to solve it for now, though the better way is to use pytest tmp_path fixture (not compatible with unittest.TestCase).
test_quant_llm_quantize_ebits_2_mbits_2 gets a strange torch.compile warning in PyTorch 2.3 only torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:436] [0/48] failed to eagerly compile backwards for dynamic, suppressing in case backwards not needed. Looks harmless though.

--dist loadscope (send a TestClass to next available worker):

TestSaveLoadMeta gets a strange error - RuntimeError: Error: accessing tensor output of CUDAGraphs that has been overwritten by a subsequent forward run. Stack trace: [Could not find stack trace]. To prevent overwriting, clone the tensor outside of torch.compile() or call torch.compiler.cudagraph_mark_step_begin() before each model invocation.. No traceback. Since tests within a class are run by the same worker, there shouldn't be filename conflict test.py. Perhaps some strange interaction between how pytest-xdist sends test to the subprocess worker and CUDA graphs + torch.compile?

--dist loadfile (send each file to next available worker): this should be the least error-prone. However, there is little speedup (24min vs 35min) since test_integration.py file is huge -> load imbalance (probably a good idea to split this into smaller files anyway).

The current approach of this PR is using --dist load. To summarize

Approach
CUDA nightly CI time

pytest sequential (baseline)
35min

pytest-xdist with two 2-GPU workers (this PR, original)
21min

pytest-xdist with four 1-GPU workers + sequential multi-GPU tests (this PR, latest)
19min

I also notice that it takes 3min to pull the Docker image and another 3-4min to download and install packages. So it seems like two 2-GPU workers have pretty good scaling (7 + 14 vs 7 + 28), while four 1-GPU workers is not much of an improvement (7 + 12 vs 7 + 28). I guess the 2-GPU FSDP tests can be run in parallel too for the latest configuration.

What I had tried messing around with before was using https://docs.pytest.org/en/stable/how-to/mark.html
which is useful to mark the 2 gpu and 1 gpu tests independently, and in the github action file you can just have 2 pytest commands to run them one after the other

Yup that's what I have in mind too. Just want to make sure only those 2 tests use 2 GPUs.

We can merge this but let's keep an eye out on whether this makes debugging test failures more tricky to debug in teh future

@msaroufim I keep seeing this test failing
test/integration/test_integration.py::TestSaveLoadMeta::test_save_load_int8woqtensors

Not exactly clear what's causing it. Should we revert or try to fix it?

Revert and ask questions later


text:
Commit d26e112d8b2171ceb78f4c2566245c977b2ff7d1 appears to have broken the build

🐛 Bug
The changes in commit d26e112d8b2171ceb78f4c2566245c977b2ff7d1 to move away from the deprecated torch type at::DeprecatedTypeProperties seems to have broken the build: https://travis-ci.org/pytorch/audio/builds/578519920
                 from torchaudio/torch_sox.cpp:1:
torchaudio/torch_sox.cpp: In lambda function:
torchaudio/torch_sox.cpp:40:42: error: expected primary-expression before ���>’ token
     auto* data = tensor.data_ptr<scalar_t>();
                                          ^
/home/travis/miniconda3/envs/testenv/lib/python3.7/site-packages/torch/include/ATen/Dispatch.h:12:12: note: in definition of macro ‘AT_PRIVATE_CASE_TYPE’
     return __VA_ARGS__();                          \
            ^
torchaudio/torch_sox.cpp:39:3: note: in expansion of macro ‘AT_DISPATCH_ALL_TYPES’
   AT_DISPATCH_ALL_TYPES(tensor.scalar_type(), "write_audio_buffer", [&] {
   ^
torchaudio/torch_sox.cpp:40:44: error: expected primary-expression before ‘)’ token
     auto* data = tensor.data_ptr<scalar_t>();
                                            ^

Environment

What commands did you used to install torchaudio (conda/pip/build from source)?

Fails on travis and when building/installing with:
pip install git+git://github.com/pytorch/audio.git

Thanks for pointing this out. Reverting in #264.

Thanks for reporting! Fixed with #265. Please feel free to-reopen or open a new PR if the issue persists.

repo: pytorch/audio


text:
Lora single device recipe failing for qlora

The single device lora recipe with qlora configs is failing with below error on new machine with fresh install:
Command:
tune lora_finetune_single_device --config recipes/configs/qlora_finetune_single_device.yaml

Error:
  File "/home/gchauhan/torchtune/recipes/lora_finetune_single_device.py", line 455, in <module>
    sys.exit(recipe_main())
             ^^^^^^^^^^^^^
  File "/home/gchauhan/torchtune/torchtune/config/_parse.py", line 54, in wrapper
    sys.exit(recipe_main(conf))
             ^^^^^^^^^^^^^^^^^
  File "/home/gchauhan/torchtune/recipes/lora_finetune_single_device.py", line 449, in recipe_main
    recipe.setup(cfg=cfg)
  File "/home/gchauhan/torchtune/recipes/lora_finetune_single_device.py", line 152, in setup
    self._model = self._setup_model(
                  ^^^^^^^^^^^^^^^^^^
  File "/home/gchauhan/torchtune/recipes/lora_finetune_single_device.py", line 248, in _setup_model
    utils.validate_expected_param_dtype(model, dtype=self._dtype)
  File "/home/gchauhan/torchtune/torchtune/utils/precision.py", line 161, in validate_expected_param_dtype
    raise ValueError(f"Parameter {name} has dtype {param.dtype}")
ValueError: Parameter layers.0._checkpoint_wrapped_module.attn.q_proj.weight has dtype torch.float8_e5m2fnuz

Environment:
Created by the default install
python -m torch.utils.collect_env

Collecting environment information...
PyTorch version: 2.2.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: CentOS Stream 9 (x86_64)
GCC version: (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.34

Python version: 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.19.0-0_fbk11_zion_11538_g61e82533119f-x86_64-with-glibc2.34
Is CUDA available: True
CUDA runtime version: 12.0.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA H100
GPU 1: NVIDIA H100

Nvidia driver version: 525.105.17
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   52 bits physical, 57 bits virtual
Byte Order:                      Little Endian
CPU(s):                          92
On-line CPU(s) list:             0-91
Vendor ID:                       AuthenticAMD
Model name:                      AMD EPYC 9654 96-Core Processor
CPU family:                      25
Model:                           17
Thread(s) per core:              1
Core(s) per socket:              92
Socket(s):                       1
Stepping:                        1
BogoMIPS:                        4792.77

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.2.1
[pip3] torchao-nightly==2024.3.23
[pip3] torchtune==0.0.1
[pip3] triton==2.2.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] torch                     2.2.1                    pypi_0    pypi
[conda] torchao-nightly           2024.3.23                pypi_0    pypi
[conda] torchtune                 0.0.1                    pypi_0    pypi
[conda] triton                    2.2.0                    pypi_0    pypi

With PyTorch Nightlies
Getting different error with the latest PyTorch nightlies: https://gist.github.com/chauhang/9b74b6bbe19c13b2b12c6a8ad066745c

I think this is related to the torachao nightly change. @rohan-varma mind taking a look?
cc: @ebsmothers

authors: chauhang kartikayk

repo: pytorch/torchtune


text:
Remove most of the PTQ receipe

Summary:
We want to decouple PTQ and torchtune, to avoid headaches of torchao API changes breaking torchtune. Also curernt PTQ APIs in torchtune is not the most up to date ones, by removing them we could deprecate these API in torchao as well.
8da4w ptq is kept because it's used in QAT, but it can be removed after QAT is refactored as well.
User should use torchao directly after torchtune finetuning: https://github.com/pytorch/ao/blob/main/torchao/quantization/README.md to quantize their model
and do eval/benchmark in torchao as well: https://github.com/pytorch/ao/tree/main/torchao/_models/llama
Test Plan:
CI
Reviewers:
Subscribers:
Tasks:
Tags:
Context
What is the purpose of this PR? Is it to

[ ] add a new feature
[ ] fix a bug
[ ] update tests and/or documentation
[x] other (please add here)

Deprecate old APIs
Changelog
What are the changes made in this PR?
Test plan
Please make sure to do each of the following if applicable to your PR. (If you're not sure about any one of these just ask and we will happily help.)

[ ] run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
[ ] add unit tests for any new functionality
[ ] update docstrings for any new or updated methods or classes
[ ] run unit tests via pytest tests
[ ] run recipe tests via pytest tests -m integration_test
[ ] manually run any new or modified recipes with sufficient proof of correctness

[ ] include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

Can you also update our live docs to account for this? See e.g. here and here

should I remove quantization walk through and just point to torchao there?

Does this also mean that we no longer support generation with quantized models? E.g. what about this portion of our generation recipe? Is it possible to just load in the AO quantized checkpoint directly or no?

yeah we'll remove this part as well, but currently it's kept for the last PTQ quantizer Int8DynActInt4WeightQuantizer, that will be removed after the QAT refactor. cc @andrewor14

we expect people to run accuracy eval and perf benchmark in torchao as well (see updated doc as well)

Is it possible to just load in the AO quantized checkpoint directly or no?

I think this could be yes, but I haven't tested in torchtune, here are some tests in torchao: https://github.com/pytorch/ao/blob/e5df48e1033611d6433dc3fd1de7923e0f555ebc/test/quantization/test_quant_api.py#L643-L677

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 27.23%. Comparing base (7eb89e2) to head (1f6d12c).
Report is 4 commits behind head on main.

:exclamation:  There is a different number of reports uploaded between BASE (7eb89e2) and HEAD (1f6d12c). Click for more details.
HEAD has 4 uploads less than BASE

Flag
BASE (7eb89e2)
HEAD (1f6d12c)

5
1

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1219       +/-   ##
===========================================
- Coverage   67.81%   27.23%   -40.59%
===========================================
  Files         219      221        +2
  Lines        9908     9999       +91
===========================================
- Hits         6719     2723     -3996
- Misses       3189     7276     +4087

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
cmake_linux_gpu fails with libcudnn.so.8: cannot open shared object file: No such file or directory

🐛 Describe the bug
The cmake_linux_gpu job is currently failing with the following error message:
OSError: libcudnn.so.8: cannot open shared object file: No such file or directory

Looks like a dependency issue on CUDA. The issue first appeared couple of days ago and persists across nightlies.
Versions
latest main
cc @seemethere

Seems no longer to be the case: https://app.circleci.com/pipelines/github/pytorch/vision/16195/workflows/16ec8058-6884-4aab-90ff-6c6a8a111017/jobs/1312141

authors: datumbox pmeier

repo: pytorch/vision


text:
Misconfiguration of MobileNetV3-Large backbone for DeepLabV3

🐛 Describe the bug
Currently, the MobileNet-V3 (Large) backbone option for DeepLabV3 (added in #3276) is configured with an output stride of 16 (Link); for more background information on the technical jargon, please take a look at #7955. The authors of DeepLabV3 use an output stride of 16 when the rates are (6, 12, 18) and double these rates when the output stride is 8 (Link). The link between the output stride, atrous rates and input spatial resolution holds regardless of the backbone used. Now, the Torchvision port of DeepLabV3 hardcodes the atrous rates to 12, 24, and 36 (Link) instead, and the ResNet-50 and ResNet-101 backbones are configured correctly with an output stride of 8. To verify this, I've written a short script:
import torch
import torchvision

input = torch.rand(1, 3, 512, 512)
# A pre-forward hook function to attach to one of the ASPP module's convolutional layers
# The input is the backbone-encoded feature maps, so we're just printing its shape
hook = lambda _, i: print(i[0].shape)

# Try out deeplabv3_resnet50() or deeplabv3_resnet101() :)
model = torchvision.models.segmentation.deeplabv3_mobilenet_v3_large()
model.classifier[0].convs[0].register_forward_pre_hook(hook)
model.eval()

with torch.no_grad():
    out = model(input)["out"]  # Prints (N, C, 32, 32) instead of (N, C, 64, 64)!

This is a deviation from the guidelines set in the paper more so than it is a bug. If there's a reason why such a deviation exists, it isn't well-documented or made clear. In the case that this isn't a documentation issue, there are a few possible resolutions: (1) modify the MobileNet backbone to have an output stride of 8 or (2) make the atrous rates configurable via the DeepLabHead constructor and set the rates to (6, 12, 18) (Link). The latter is much easier to implement, and could even be extended to allow user-configurable atrous rates, which is loosely motivated in #7955; see the fourth footnote and its reference.
Versions
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.4 (arm64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:38:29) [Clang 13.0.1 ] (64-bit runtime)
Python platform: macOS-13.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.25.0
[pip3] pytorch-lightning==2.0.4
[pip3] pytorch-nlp==0.5.0
[pip3] torch==2.0.1
[pip3] torchmetrics==1.0.0
[pip3] torchvision==0.15.2
[conda] numpy                     1.25.0                   pypi_0    pypi
[conda] pytorch-lightning         2.0.4                    pypi_0    pypi
[conda] pytorch-nlp               0.5.0                    pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torchmetrics              1.0.0                    pypi_0    pypi
[conda] torchvision               0.15.2                   pypi_0    pypi

Hi, I've tried to address this issue in this PR: https://github.com/pytorch/vision/pull/8019

authors: lambda-shuttle nvs-abhilash


text:
Compatible issues with JupyterLab 1.0.0

I'm using JupyterLab 1.0.0 and installed pyviz_comms via the following command.
jupyter labextension install @pyviz/jupyterlab_pyviz@next

How I cannot launch a notebook and encounter the following error. There was no issue before installing pyviz_comms.
Launch Error
Cannot read property 'addFactory' of undefined

I'll try to push out an update today.

I've released a new version of the extension on npm.

authors: dclong philippjfr


text:
Move GetProperties into BaseWrapper

Move CaptureAsImage into BaseWrapper
Spread writable_props
test for UIA GetProperties

Current coverage is 96.52%

Merging #153 into UIA will not affect coverage as of a3536b2

@@              UIA    #153   diff @@
=====================================
  Files          31      31
  Stmts       17317   17317
  Branches        0       0
  Methods         0       0
=====================================
  Hit         16716   16716
  Partial         0       0
  Missed        601     601

Review entire Coverage Diff as of a3536b2

Powered by Codecov. Updated on successful CI builds.

authors: codecov-io moden-py

repo: pywinauto/pywinauto


text:
Pharmacokinetic models

Is your feature request related to a problem? Please describe.
Add more support for pharmacokinetic models (like existing dose-response models). This topic area is a bit unfamiliar to me, but I can add the models from Bonate's book as I read through it. This issue will be a place for me to collect those models. I will also need to look around for external software or examples to compare to.
These would replace the current estimating_equations/dose_response.py file with the more general estimating_equations/pharmacokinetics.py file. Since all imports are done in estimating_equations/__init__.py this won't change anything on the user side.
Describe the solution you'd like
Add corresponding estimating equations
Describe alternatives you've considered
None
Additional context
Bonate PL. "Pharmacokinetic-Pharmacodynamic Modeling and Simulation" 2nd edition.

E_max pharmacokinetic model from pg 101 of Bonate.
E = {E_max C} / {E_50 + C}
where E is the observed effect, E_max is max effect, EC_50 is the concentration with 50% of maximal effect, and C is concentration. Score function is given in Equations 14 and 15 on pg 102.
\psi_emax = (Y - {E_max C} / {EC_50 + C}) (C / {EC_50 + C})
\psi_ec50 = (Y - {E_max C} / {EC_50 + C}) ( {-E_max C} / {EC_50 + C}^2)
Figure 1 of Chapter 4 might be good to replicate?

Example with data from Table 22 on page 185

This also links back to #6

More model options (and software to check against)
https://cran.r-project.org/web/packages/DoseFinding/DoseFinding.pdf

Some notes on E-max models
https://blog.djnavarro.net/posts/2024-01-09_emax-models/

A side-note here: The DoseFinding documentation claims that " the toxicology literature this model is also called four-parameter log-logistic (4pLL) model". I think this is untrue. When comparing to Ritz et al., their 4P log-logistic model is of a different form. Further, comparing DoseFinding's sigEmax and drc's drm with fct=LL.4() give different results. Specifically, the lower limit and upper limit are close but do not match.
I spent like a week on this issue. What I've learned is that the language is a mess. Always look at the math

New structure in v2.3 with E-max and 4P-log-logistic models. There are some other models I may consider adding in the future. But for near future, don't have any specific to add

repo: pzivich/Delicatessen


text:
🛑 nitter.tiekoetter.com is down

In 2b989a6, nitter.tiekoetter.com (https://nitter.tiekoetter.com) was down:

HTTP code: 502
Response time: 871 ms

Resolved: nitter.tiekoetter.com is back up in d3b80bb after 28 minutes.


text:
🛑 nitter.projectsegfau.lt is down

In 627c545, nitter.projectsegfau.lt (https://nitter.projectsegfau.lt) was down:

HTTP code: 503
Response time: 381 ms

Resolved: nitter.projectsegfau.lt is back up in b2f868c after 24 minutes.


text:
🛑 nitter.domain.glass is down

In c7d8587, nitter.domain.glass (https://nitter.domain.glass) was down:

HTTP code: 503
Response time: 414 ms

Resolved: nitter.domain.glass is back up in 3ed3077 after 2 hours, 20 minutes.


text:
🛑 nitter.adminforge.de is down

In 0ca0130, nitter.adminforge.de (https://nitter.adminforge.de) was down:

HTTP code: 502
Response time: 576 ms

Resolved: nitter.adminforge.de is back up in 69b795a after 7 minutes.


text:
🛑 t.floss.media is down

In 4dc1604, t.floss.media (https://t.floss.media) was down:

HTTP code: 0
Response time: 0 ms

Resolved: t.floss.media is back up in 41d9ace after 15 minutes.


text:
🛑 t.floss.media is down

In e633aeb, t.floss.media (https://t.floss.media) was down:

HTTP code: 0
Response time: 0 ms

Resolved: t.floss.media is back up in 5400ed7 after 37 minutes.


text:
🛑 nitter.in.projectsegfau.lt is down

In eb7e7e2, nitter.in.projectsegfau.lt (https://nitter.in.projectsegfau.lt) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.in.projectsegfau.lt is back up in b5c737b.


text:
Add a labeling feature

Replicate NFD's node labeling feature for the kernel version, but use the node's .status.nodeInfo.kernelVersion field instead.
Users should be able to choose if they want to use NFD or that lightweight node labeler:

if NFD, then set DaemonSet selectors based on the NFD label;
if OOTO, then start an additional controller that merely labels nodes whenever their .status.nodeInfo.kernelVersion changes.

Implemented in #3.

repo: qbarrand/oot-operator


text:
Support for Eldom Smart boilers

Like this one.

This would be great! Tried to adjust the current version by changing flat by smart and referring to the 0.2.0 package but I get an error and I don't have the knowledge to solve it.

Hello, and what about the heaters like that: https://eldominvest.com/en/category/20.html
I will buy you more than 1 beer :)
Cheeers,
Juraj

@juras72, I'd love to support those as well! 😄 The only problem is that I cannot get a hold of the APIs for those devices. 😕 I tried reaching out to Naturela (the developers of those APIs), but they weren't super keen on giving me some docs.

@jeroenb21, I just released 1.0.0 and I added support for smart boilers.
Can you try out the integration and let me know if it works well for you?

@juras72, I'll reopen a new issue requesting support for the devices that you've shown.

This issue is resolved with 1.0.0.

@jeroenb21, I just released 1.0.0 and I added support for smart boilers.
Can you try out the integration and let me know if it works well for you?

Hi Daniel,
I installed the newer version, it says 2.0.0 ?
I can login now and see the boiler. But there is not a lot I can do with it. Only some consumption numbers.

Hi, @jeroenb21!
Oh, that's interesting. You should also see the water heater entity itself. 😕
I'm testing with the following smart boiler: https://eldominvest.com/bg/product/502.html
I'm getting this functionality:

The functionalities are:

Water heater entity functionalities
Powerful mode enabling
Sensor data (that you already see)

Can you try updating the integration again? I just pushed a minor version to fix two smart boiler API calls that I had mistaken. And let me know if it works well this time.

Hi @danielgospodinow,
I edited the above comment but I see it doesnt really show. But after a hour it was working like it should.
Thank you !

Hi Daniel,
I got this message when adjusting the tempreture. Seems its looking at the wrong url.

@jeroenb21, yes, that's correct. I had a bug a while back that's fixed now. Can you try and update to the latest version? It's 2.2.3. The issue should no longer be there.

repo: qbaware/homeassistant-eldom


text:
Provide more icons for species and specimen

provides more icon options for species and specimen
allows users to change the default icons when creating or editing an experiment
stores selected icon (labels) in the experiment table

Should these icons also be shown on the project summary? And if so from which experiment are they chosen?

that is the big question 🤷♂

repo: qbicsoftware/data-manager-app