text:
2FA/multifactor auth: staging/testing rollout

What's the problem this feature will solve?
To finish #996 (see #5567), we need to test MFA with real users on real packages; asking them to spin up dev environments is too hard and won't help multi-maintainer projects reason well about what MFA policies they want to set up.
Describe the solution you'd like
My tentative suggestion is:

We first roll out MFA on test.pypi.org and publicize it to project maintainers for a designated 2-week testing period, and we allot extra time for dealing with support requests during that time
We then roll it out to pypi.org and announce it on the PyPI announcement email list, PSF blog, etc.

Additional context

If things go awry during initial testing, is there any chance we will need to wipe tokens from users' accounts?
Are there particular categories of user we need to make sure we get in the beta test? Ideas that come to mind for me: Windows, mobile, users of particular TOTP implementations and U2F implementations, people on very slow connections, people who habitually block a lot of cookies/JS, maintainers who maintain 20+ projects, people with very old and weird PyPI accounts (e.g., we do not have verified email for them, their passwords do not adhere to current policy), users with upwards of 10 MFA methods they want to add to their accounts, multi-maintainer projects, multi-owner projects, organizations where users share an auth token within a group.
Is test.pypi.org the right place for this? Should we spin up some other instance for this particular kind of testing, since maintainers do use Test PyPI for real uploads (of packages that need testing)?

Cc @ewdurbin .

If we want to roll out to TestPyPI first, we'll probably want to feature-flag this feature, which would be a bit of work.
It might be preferable to do a true "beta" on PyPI instead, with a limited group of users. This could just be a boolean field on the User model that enables the feature, which we set in the admin UI. And we could just skip this check and enable it for all TestPyPI users, and eventually remove it once we've rolled everything out.
This could allow us to do a timeline like:

1 week period on TestPyPI for everyone
2 week beta on PyPI for select users
Open to all users

If things go awry during initial testing, is there any chance we will need to wipe tokens from users' accounts?

Possibly? I think we should be explicit in our "beta" announcement that this might happen, so users aren't surprised.

Are there particular categories of user we need to make sure we get in the beta test?

I think all the groups you listed would be ideal.

people who habitually block a lot of cookies/JS

This is a good point -- it looks pretty likely that our dependency on qrious means that this won't work for users blocking JavaScript. We probably need to test this and add a fallback, or maybe revisit this dependency. (cc @woodruffw)

Is test.pypi.org the right place for this?

I think it is. We don't really claim that TestPyPI should be depended on for much.

If we want to roll out to TestPyPI first, we'll probably want to feature-flag this feature, which would be a bit of work.
It might be preferable to do a true "beta" on PyPI instead, with a limited group of users. This could just be a boolean field on the User model that enables the feature, which we set in the admin UI. And we could just skip this check and enable it for all TestPyPI users, and eventually remove it once we've rolled everything out.
This could allow us to do a timeline like:

1 week period on TestPyPI for everyone
2 week beta on PyPI for select users
Open to all users

+1 to this approach.

This is a good point -- it looks pretty likely that our dependency on qrious means that this won't work for users blocking JavaScript. We probably need to test this and add a fallback, or maybe revisit this dependency.

It should be relatively easy to add a fallback here: we can display the otpauth:// URL directly, and the user can copy-and-paste that into their TOTP application or use their QR generator of choice. Right now that URL is only available via the aria-label of the canvas that we render the QR in, but that wouldn't be hard to change at all. (Sidenote: I'm not a frontend developer at all, so I have no idea how to check whether a user has disabled Javascript. Is it just <noscript>?)

@nlhkabu - does this feel ok to you? If so then I will declare consensus and start planning this, writing sets of "hey come test this" announcement text, etc.

Hi @brainwane - sounds very reasonable to me.
I'd like to run some direct user tests alongside open testing - sessions where I am able to observe users and understand anywhere they get stuck.
I've set up a Google form for registering users for this.  @brainwane would you be able to:

review the form
add a link to it in your announcement

To be clear, I don't think that manual user testing should block the launch of 2FA, unless I find any major technical issues that are not raised by open beta testing.

@nlhkabu One small comment on the form: might want to ask for their PyPI username as well, so we can enable 2FA for them during the private beta.

#5567 is close to being done. I'm going to plan on merging it on or before Friday, May 3rd, which would give us the following timeline:

On or before 5/3 - Merge #5567
5/3 - enable 2FA across all of TestPyPI, send announcements
5/3-5/20 - private beta on PyPI, add users as necessary
5/20 - enable for all PyPI users

OK, draft announcement:
Subject line: PyPI two-factor auth (2FA) trial May 3-20
Dear PyPI users:
To increase the security of PyPI downloads, we're beginning to introduce two-factor authentication (2FA) as an option for project maintainers and owners. If you set up 2FA on your PyPI account, then once you set it up, you must provide a second method of identity verification (other than your username and password) to log in.
Starting this Friday, May 3rd, you'll be able to use 2FA on Test PyPI. And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.
PyPI currently supports a single 2FA method: generating a code through a Time-based One-time Password (TOTP) application. Therefore, to use 2FA on PyPI, you'll need to provision an application (usually a mobile phone app) in order to generate authentication codes; our help documentation will give you suggestions and pointers.
During this testing period, if things go awry, there's a chance we will need to wipe tokens from users' accounts, so if you choose to try it, please be forewarned. We suggest you make sure you have a PyPI-verified email address on your user account before trying the feature, to make potential account recovery smoother.
And please let us know if you run into glitches.
We expect to end this testing period on May 20th, then enable the 2FA feature for all PyPI users, and move on to working on WebAuthn support.
Thanks to the Open Technology Fund for funding this work. More progress reports at our wiki page.
-the PyPI team

(I need to add a note calling out that this doesn’t affect the upload endpoint currently.)

Looks good to me @brainwane - thank you. You can probably change this sentence to be shorter:

If you set up 2FA on your PyPI account, you must provide a second method of identity verification (other than your username and password) to log in.

First:

And if you'd like to try 2FA on official PyPI, please fill out this Google form so we can invite you to the private beta, which we plan to hold 3-20 May.

Should this say that it's really more of a user test? It's not just joining a beta, the plan is to sit with these folks for an hour, right? I eagerly clicked the link, then was disappointed since I didn't have the hour.
Second: Is there a plan to require 2FA eventually, or, if there isn't a plan, is it worth threatening that that might happen eventually? I think it makes sense to have 2FA for software repos like PyPi, but I know requiring it would be difficult. Maybe there's a middle ground to start pushing for.

@mlissner You don't need to be available for that hour to be in the beta, just respond "no". :)

I'm like 10% sure that changed? Anyway, good!

When we flipped the switch for this feature on Test PyPI & canon PyPI, we did not initially update the user model so that existing Test PyPI accounts had the feature on. Ernest (I believe) ran some SQL to update all the Test PyPI accounts yesterday and turn on the flag.
Right now, new Test PyPI accounts (as in, created since that time yesterday) do not have the flag set so they don't see 2FA in their account settings.
I'd like @ewdurbin to run the "turn this on for everybody on Test PyPI" SQL again just to get that taken care of, then figure out what to do next.
I think @di told me that turning it on by default for all Test PyPI accounts would also turn it on for all canon PyPI accounts (does this mean we ought to have a better feature flag system in general, or just for this feature?) (and I presume that constraint would also stop us from doing something to fix this like: verifying the email address triggers turning on 2FA support).

Reran the enabling of 2FA on test.pypi.org

Today's the 20th and it looks like we aren't done with the beta yet, because #5866 is a blocker.
Once we finish that, there are some UI issues that we should fix, but it's ok to roll out 2FA to everyone on pypi.org before fixing them.
And then a note for next time: let's address #5869 (or having "flip bit in user model" in the runbook for the WebAuthN rollout).

@ewdurbin @dstufft @di do we have any way of knowing how many users on PyPI have turned on 2fa? Is that even a useful statistic?
I naively assume that it would be, and that we would want the number to go up over time, and that we would want a rough estimation in particular of how many users have it turned on who own or maintain at least one package.
(I am ok with the answer being "it is a pain to check that" in which case I will probably ask that we check it, like, nowish and then again in a couple months, as we check how well our staging/rollout/publicity are going. I am of course also ok with the answer being "we cannot know that" or "it is a vulnerability to say this publicly" or what have you.)

warehouse=> select ur.role_name, count(*) from users as u join (select distinct user_id, role_name from roles) as ur on u.id = ur.user_id where u.totp_secret is not null group by ur.role_name;
 role_name  | count
------------+-------
 Maintainer |    62
 Owner      |   214
(2 rows)

Update as of this time:
warehouse=> SELECT ur.role_name,
warehouse->        Count(*)
warehouse-> FROM   users AS u
warehouse->        JOIN (SELECT DISTINCT user_id,
warehouse(>                              role_name
warehouse(>              FROM   roles) AS ur
warehouse->          ON u.id = ur.user_id
warehouse-> WHERE  u.totp_secret IS NOT NULL
warehouse-> GROUP  BY ur.role_name;
 role_name  | count
------------+-------
 Maintainer |    88
 Owner      |   326
(2 rows)

@nlhkabu For the WebAuthn rollout, @ewdurbin suggested that we initially add a badge in the Account Settings marking the WebAuthn 2FA method as "beta". I've thus filed #5976 just now.

Update as of today:
 role_name  | count
------------+-------
 Maintainer |   105
 Owner      |   446
(2 rows)

Per our conversation in the Friday meeting we plan to roll out the WebAuthn feature on test.pypi.org and on pypi.org with the "beta" badge (#5976) but without doing a bunch of messy bit-flipping. I've also updated https://wiki.python.org/psf/WarehousePackageMaintainerTesting so it'll be ready to publicize.
Working on copy for blog post(s), mailing list posts, social media, and so on.

OK, today we're rolling WebAuthn out as beta and asking a few people to test it, and in the afternoon we'll post blog posts and I'll notify distutils-sig et alia. Later this week I'll publicize it to some more communities, and then in maybe 10 days we'll remove the "beta" badge and I'll email pypi-announce and python-announce. Once that is done I figure we can close the issue.

@brainwane I've tried that on Test PyPI.
So I have a TOTP set up. I clicked on Add 2FA with security key.
It prompted me to enter a Key name which I did (Yubikey Neo).
After that, clicking Provision key does nothing visually. So I've opened DevTools.
I can see a successful GET request to https://test.pypi.org/manage/account/webauthn-provision/options with some JSON payload in the response. It looks legit, contains my user data and a challenge.
After clicking more times on that button, each of them produces an exception being logged to the JS console.
[8] bind-modal-keys.js:43 Uncaught (in promise) DOMException: A request is already pending.
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ webauthn.js:179
(anonymous) @ webauthn.js:23
r @ runtime.js:55
(anonymous) @ runtime.js:293
t.(anonymous function) @ runtime.js:107
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
d @ raven.js:445
[2] bind-modal-keys.js:43 Uncaught (in promise) DOMException: The operation either timed out or was not allowed. See: https://w3c.github.io/webauthn/#sec-assertion-privacy.
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
Promise.then (async)
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ webauthn.js:179
(anonymous) @ webauthn.js:23
r @ runtime.js:55
(anonymous) @ runtime.js:293
t.(anonymous function) @ runtime.js:107
r @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
(anonymous) @ bind-modal-keys.js:43
d @ raven.js:445

Thanks for the bug report @webknjaz - I've turned that into #6050 so we can address it there.

We've had a few issues (such as #6106) due to a misconfiguration of test.pypi.org, and we're getting more user testing now thanks to outreach by @nlhkabu (such as this email to the Education SIG which I forwarded to Software Carpentry's discussion list). I'm going to wait till we have a few more data points from Nicole's testing, so we can fix issues that crop up. Then I'll call the beta finished and we should make further public announcements.

Thanks @brainwane.  FYI, I have user testing scheduled most days in the coming week.  The final session is scheduled for Tuesday 9th.

@nlhkabu ran user tests, summed up her findings in #6173, and we've now addressed those issues, yay! But we still have a few WebAuthn issues left in the milestone before I'll call the beta finished.
Once we approve #6084 we'll have one more auth feature -- API tokens -- to test via a beta period on test.pypi.org and pypi.org, so I'll keep this issue open to track that, then close it.

@ewdurbin May I ask:

how many users have turned on 2FA at all?
how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?
what proportion & number of logins over the past week were protected by 2FA?
how many API tokens currently exist?
whether it's easy to set this up in Datadog so I can query it myself instead of asking you?

@brainwane I'll query for those numbers before lunch, unfortunately we don't have this in Datadog at the moment, we'd need to implement a scheduled task to populate them... which perhaps is worthy of doing.

how many users have turned on 2FA at all?

warehouse=> select count(distinct user_id) from (SELECT user_id::uuid from user_security_keys UNION SELECT id::uuid from users where totp_secret is not null) as a;
 count
-------
  1688
(1 row)

how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

TOTP
warehouse=> select count(*) from users where totp_secret is not null;
 count
-------
  1632
(1 row)

WebAuthn
warehouse=> select count(distinct user_id) from user_security_keys ;
 count
-------
   132
(1 row)

what proportion & number of logins over the past week were protected by 2FA?

This we may be able to get from Datadog...

how many API tokens currently exist?

warehouse=> select count(*) from macaroons ;
 count
-------
    21
(1 row)

Past Week:
Percentage of Logins using Two-Factor Authentication: 7.71%

Issues we need to resolve before ending WebAuthn beta:

[ ] Chrome issue #6050
[ ] Edge issue #6146
[ ] ask for password, not username, when disabling 2FA #5825

(Possibly others - will discuss in meeting this week.)
Issue(s) we need to resolve before ending API token beta:

[ ] maintainers can't create project tokens #6262

Percentage of Logins (to pypi.org via browser, I think) using Two-Factor Authentication:
May: 2.25%
June: 3.08%
July 1-21: 4.62%
July 21-25: 9.72% (July 25th was our post to Discourse about API tokens)
July 25-28: 15.45%
And, for logins in the past 2 days, it's 23.1%. That is mostly a weekend, so it'll be interesting to see whether the trend changes on weekdays...

Is there any plan for an API to create upload tokens? E.g. I'd like to have a command-line tool prompt me once for my password & 2FA code, then obtain and store a project-scoped token to use for uploads.

We have updated the token username and prefix in #6342.
username: @token => token
password/token: pypi: => pypi-
These changes should alleviate the need for escaping heroics.
The previous format will continue to work for now, but users will be notified to update their configurations to match the new syntax before the beta period is over.

Contractors on the OTF-funded work need to stop/deprioritize work on the security features in order to ensure we complete the accessibility and internationalization work by the end of the month. Therefore, even though some security features are still in beta, I'm closing the milestone.

I'm writing a discuss.python.org/distutils-sig post now to update our community on the end of the OTF-funded work. I'd like for us to have a few more beta blockers closed before we send out a pypi-announce email, to reduce how much time we and other volunteers spend on support issues; it's ok with me if that means the email doesn't go out for a few more weeks.

Update for today:

how many users have turned on 2FA at all?

warehouse=> select count(distinct user_id) from (SELECT user_id::uuid from user_security_keys UNION SELECT id::uuid from users where totp_secret is not null) as a;
 count
-------
  3544
(1 row)

how many users have a TOTP method provisioned, and how many have a WebAuthn key provisioned?

TOTP
warehouse=> select count(*) from users where totp_secret is not null;
 count
-------
  3398
(1 row)

WebAuthn
warehouse=> select count(distinct user_id) from user_security_keys ;
 count
-------
   336
(1 row)

how many API tokens currently exist?

warehouse=> select count(*) from macaroons ;
 count
-------
  1127
(1 row)

authors: brainwane di ewdurbin mlissner nlhkabu takluyver webknjaz woodruffw


text:
Attemting to view account settings asks to reauthenticate even immediately after login

Describe the bug
After logging in, and immediately visiting the account settings page only a few seconds later, a "Confirm password to continue" page is shown.
My account has U2F 2FA enabled, in case that ends up being relevant.
Expected behavior
Immediately after login (and having provided both a password and the U2F), access should likely be enabled across any part of the site, including sensitive areas, until some timeout has elapsed.
To Reproduce
In a private window (or other place where you have not logged in), visit the home page, click log in, fill out your credentials, authenticate via U2F if applicable, then click "Account Settings" in the menu bar after logging in.
Quick video attached:

My Platform
FF80, macOS, U2F via Yubikey
Additional context
#8191 seems likely related (though don't know that the behavior is new after it).

Thanks for the report, I've noticed this as well. It looks like the initial login is not setting the "time to reauthenticate" in the session.

I couldn't create this issue again

Perhaps indeed 2FA/U2F is required to observe the behavior then?

Yes, it definitely is. The record_auth_timestamp function is only called when the user doesn't have 2FA set: https://github.com/pypa/warehouse/blob/ab78391c8e69173aa878c08f88259756de1db7b4/warehouse/accounts/views.py#L200
This needs to happen here as well:
https://github.com/pypa/warehouse/blob/ab78391c8e69173aa878c08f88259756de1db7b4/warehouse/accounts/views.py#L249-L261
Really, since _login_user is used in both, it should happen here instead:
https://github.com/pypa/warehouse/blob/ab78391c8e69173aa878c08f88259756de1db7b4/warehouse/accounts/views.py#L777-L788

source: gharchive/issue

authors: Julian di imsahil007

repo: pypa/warehouse

license_type: permissive

license_source: github-api


text:
Nuke: Render Creator doesn't handle Integer knobs

Describe the bug
Using integer knob in project_anatomy/imageio/nuke/nodes/requiredNodes/0/knobs breaks creating render node
To Reproduce

Go to Settings, project_anatomy/imageio/nuke/nodes/requiredNodes/0/knobs
Set file_type:exr, compression:DWAB, dw_compression_level:60
Launch Nuke, create render node
See error:

Traceback (most recent call last):
  File "C:\Users\robur\OpenPype\openpype\tools\creator\window.py", line 435, in _on_create
    data={"variant": variant}
  File "C:\Users\robur\OpenPype\openpype\pipeline\create\legacy_create.py", line 152, in legacy_create
    instance = plugin.process()
  File "<string>", line 127, in process
  File "C:\Users\robur\OpenPype\openpype\hosts\nuke\api\lib.py", line 931, in create_write_node
    **_data
  File "C:\Users\robur\OpenPype\openpype\hosts\nuke\api\lib.py", line 389, in add_write_node
    w[k].setValue(v)
ValueError: expecting a sequence of 1 floats, 2 found

Expected behavior
Settings should allow Nuke Integer knobs
Desktop (please complete the following information):
Windows 10, Nuke 12.2 (Python 2)
Additional context
Possibly conversion issue?
https://github.com/pypeclub/OpenPype/blob/143f7b8ac4eb9f8dcc00e9f27373d820a87132d5/openpype/hosts/nuke/api/lib.py#L355
[cuID:25u449e]

It also doesn't set boolean values. It just ignores them in settings and doesn't fail.
IMHO this might happen some time ago when Nuke implementation was moved from Avalon.
Maybe @iLLiCiTiT  might know how to fix it?

Is this still an issue? @jakubjezek001 you've looked into the code recently, can you check and validate?

authors: iLLiCiTiT jrsndl

repo: pypeclub/OpenPype

license: MIT


text:
how to debug the fpga?

I wonder if anyone has used iverilog to debug the FPGA?  That plus gtkwave are a very powerful pair of applications to debug.
It would require a bit of setup with some test scripts but might help debug custom FPGA changes.

Here is an example of the output from gtkwave during the startup of the iir module

I haven't! But I'm interested in this topic. What's the advantage of iverilog? Can we use vivado's simulator for this purpose?

The advantage of iverilog is that it is quick to make changes and re-run the simulation.  I have pushed my version of the harness here https://github.com/peteasa/pyrpl_testbench/tree/main/fpga.  This does not replace the vivado simulator, its just very quick to run with little overhead.

authors: nano-aaiv peteasa

repo: pyrpl-fpga/pyrpl


text:
Recurrent event with an exception instance leads to multiple RECURRENCE-ID values in the exception instance

I created an iCalendar object as follow:

Create bi-weekly event
Edit the title of the second instance at 2024-04-25 in Thunderbird

This leads to an iCalendar object with two events, one master and one exception instance:
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
BEGIN:VEVENT
UID:c26921f4-0653-11ef-b756-58ce2a14e2e5
DTSTART;VALUE=DATE:20240411
DTEND;VALUE=DATE:20240412
DTSTAMP:20240429T181103Z
LAST-MODIFIED:20240429T181103Z
RRULE:FREQ=WEEKLY;INTERVAL=2
SEQUENCE:1
SUMMARY:Test 1
X-MOZ-GENERATION:1
END:VEVENT
BEGIN:VEVENT
UID:c26921f4-0653-11ef-b756-58ce2a14e2e5
RECURRENCE-ID;VALUE=DATE:20240425
DTSTART;VALUE=DATE:20240425
DTEND;VALUE=DATE:20240426
CREATED:20240429T181031Z
DTSTAMP:20240429T181103Z
LAST-MODIFIED:20240429T181103Z
SEQUENCE:1
SUMMARY:Test 1 (edited)
X-MOZ-GENERATION:1
END:VEVENT
END:VCALENDAR

When searching for the event via caldav, two instances are returned. The second instance contains the RECURRENCE-ID field twice as shown in the following test case:
def testRecurringDateWithExceptionSearch(self):
    self.skip_on_compatibility_flag("read_only")
    self.skip_on_compatibility_flag("no_recurring")
    c = self._fixCalendar()

    # evr2 is a bi-weekly event starting 2024-04-11
    e = c.save_event(evr2)

    r = c.search(
        start=datetime(2024, 3, 31, 0, 0),
        end=datetime(2024, 5, 4, 0, 0, 0),
        event=True,
        expand=True,
    )

    assert len(r) == 2

    assert 'RRULE' not in r[0].data
    assert 'RRULE' not in r[1].data

    assert isinstance(r[0].icalendar_component['RECURRENCE-ID'], icalendar.vDDDTypes)
    # OK:
    # vDDDTypes(2024-04-11, Parameters({'VALUE': 'DATE'}))

    assert len(r[1].icalendar_component['RECURRENCE-ID']) == 1
    # fails, since there are multiple values:
    # [vDDDTypes(2024-04-25, Parameters({'VALUE': 'DATE'})),
    #  vDDDTypes(2024-04-25, Parameters({'VALUE': 'DATE'}))]

A possible solution would be to add a check for an already existing RECURRENCE-ID value here: https://github.com/python-caldav/caldav/blob/f4d5cd2844df99d15a07b72d4f3adf67635ea03b/caldav/objects.py#L2005-L2007
Another option would be to just overwrite the value.

Here is a minimal working example test case:
import socket
import tempfile
import threading
import time
from datetime import datetime

import icalendar
import pytest
import radicale
import radicale.server
import requests
from caldav import DAVClient
from caldav.objects import Calendar

@pytest.fixture
def radicale_calendar():
    radicale_host = 'localhost'
    radicale_port = 52121
    url = f'http://{radicale_host}:{radicale_port}/'
    username = 'user'
    password = 'some-password'

    serverdir = tempfile.TemporaryDirectory()
    serverdir.__enter__()

    configuration = radicale.config.load()
    configuration.update(
        {
            'server': {'hosts': f'127.0.0.1:{radicale_port}'},
            'storage': {'filesystem_folder': serverdir.name}
        }
    )

    shutdown_socket, shutdown_socket_out = socket.socketpair()
    radicale_thread = threading.Thread(
        target=radicale.server.serve,
        args=(configuration, shutdown_socket_out),
    )
    radicale_thread.start()

    i = 0
    while True:
        try:
            requests.get(url)
            break
        except:
            time.sleep(0.05)
            i += 1
            assert i < 100

    client = DAVClient(url=url, username=username, password=password)
    principal = client.principal()
    calendar = principal.make_calendar(
        name='Yep', cal_id='123'
    )

    yield calendar

    shutdown_socket.close()
    serverdir.__exit__(None, None, None)

recurring_event_with_exception = """BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
BEGIN:VEVENT
UID:c26921f4-0653-11ef-b756-58ce2a14e2e5
DTSTART;VALUE=DATE:20240411
DTEND;VALUE=DATE:20240412
DTSTAMP:20240429T181103Z
LAST-MODIFIED:20240429T181103Z
RRULE:FREQ=WEEKLY;INTERVAL=2
SEQUENCE:1
SUMMARY:Test 1
X-MOZ-GENERATION:1
END:VEVENT
BEGIN:VEVENT
UID:c26921f4-0653-11ef-b756-58ce2a14e2e5
RECURRENCE-ID;VALUE=DATE:20240425
DTSTART;VALUE=DATE:20240425
DTEND;VALUE=DATE:20240426
CREATED:20240429T181031Z
DTSTAMP:20240429T181103Z
LAST-MODIFIED:20240429T181103Z
SEQUENCE:1
SUMMARY:Test (edited)
X-MOZ-GENERATION:1
END:VEVENT
END:VCALENDAR"""

def test_expanding_caldav_search_with_recurrent_event_having_exception(radicale_calendar: Calendar):
    radicale_calendar.save_event(recurring_event_with_exception)

    r = radicale_calendar.search(
        start=datetime(2024, 3, 31, 0, 0),
        end=datetime(2024, 5, 4, 0, 0, 0),
        event=True,
        expand=True,
    )

    assert len(r) == 2

    assert 'RRULE' not in r[0].data
    assert 'RRULE' not in r[1].data

    assert isinstance(r[0].icalendar_component['RECURRENCE-ID'], icalendar.vDDDTypes) == 1
    # vDDDTypes(2024-04-11, Parameters({'VALUE': 'DATE'}))

    assert len(r[1].icalendar_component['RECURRENCE-ID']) == 1
    # [vDDDTypes(2024-04-25, Parameters({'VALUE': 'DATE'})),
    #  vDDDTypes(2024-04-25, Parameters({'VALUE': 'DATE'}))]

Thanks for your report, you've done a good job debugging and pinpointing the issue as well as writing up test code.  Would you care to make it into a pull request? :-)

Sure, I created a draft pull request: https://github.com/python-caldav/caldav/pull/395
There is still an issue with Xandikos on my machine. Do you have an idea maybe what this could be?

I've had different problems with different versions of Xandikos.  Do you get an error message?

authors: dozed tobixen

repo: python-caldav/caldav

license_source: bigquery


text:
CI: check for dependency licenses

Since our project is licensed under the MIT License, we can't be using any dependencies in our project.
This PR adds a step to the CI lint phase that will use pip-licenses to verify that all the installed packages are part of an ALLOWED_LICENSE variable. This variable is currently set to be every license we currently use.
We opted to use an allowlist instead of a denylist to make sure that new licenses are reviewed by hand and added to that variable.

How was it determined which licences are compatible with MIT?

I've looked at choose-a-license.org to see what their requirements are, and they all seemed to be okay with being used with MIT. All of those are very permissive.
Note that I'm not a lawyer and this is not a legal advice, you know the drill

I looked at LGPL and even if it might be compatible with MIT, it does have some extra requirements. See section 4, which states that copies of licence must be included among other things.

@MarkKoz my understanding here is we are only a consumer of the original library and never modify it, meaning it isn't a derived work and aren't subject to those rules. Although I'm honestly not sure, should we ask anyone about this?

@MarkKoz my understanding here is we are only a consumer of the original library and never modify it, meaning it isn't a derived work and aren't subject to those rules. Although I'm honestly not sure, should we ask anyone about this?

It is a combined work. What I've heard is importing a library is considered to be linking to it. So, see the licence's definition of a combined work:

A “Combined Work” is a work produced by combining or linking an Application with the Library. The particular version of the Library with which the Combined Work was made is also called the “Linked Version”.

As this project uses pre-commit, I feel like it would make sense to add this as a pre-commit hook, using the local configuration options, and setting it to only run on edits to pyproject.toml/poetry.lock. This would help ensure that regressions are discovered before they're ever commited.

I must be doing something wrong, as I tried this locally and added fuzzywuzzy as a dependency and this did not flag it at all.

@MarkKoz alright, so we have two packages that uses LGPL. One of them is flake8-import-order which is quite easy to swap for Isort and would be quite an improvement.
The other is chardet, a dependency of aiohttp. It is a bit more problematic, but we could dodge it using the accelator lib instead cChardet, which is under the Mozilla license.
Maybe we could explore that if we want to get rid of them? Another solution would be to simply include a LICENSE-THIRD-PARTY file. I honestly surprised LGPL would require that, I don't see it being mentioned anywhere on the Internet, neither do I see any project which include the license. Maybe people are just not following the license.
@onerandomusername we opted to run it only in the CI to facilitate things, as running custom hooks on both Windows and Linux can be a pain. I don't think it matters when it is detected, as the work will have to be re-done either way. Also I don't know how you managed to not make it work, here is an (accidental) working example: https://github.com/python-discord/bot/pull/1779/commits/384d5b4b6b945e1aac3e1dbf82c1faa81cebbcf4. Updating pip-licenses added a new dep that weren't part of our set of allowed licenses and the CI correctly rejected it.

source: gharchive/pull-request

authors: Akarys42 MarkKoz onerandomusername

repo: python-discord/bot


text:
Support eval (snekbox) file system and attachment display

Support for the new snekbox file system specifications and API changes, adds uploading of file attachments from snekbox.
From issues: https://github.com/python-discord/snekbox/issues/103 and https://github.com/python-discord/snekbox/issues/114
Requires merging of Snekbox PR first: https://github.com/python-discord/snekbox/pull/159
Changes

Supports receiving and rendering most file formats supported by discord itself (images, audio, video, text, etc.)
Some dataclass structure refactoring for snekbox to simplify design and make feature additions easier

Current features:

[x] Upload files returned by snekbox eval, configurable size limit and number of files
[x] All normal !eval usage now will create a main.py target on snekbox to support other file system features such as __file__ and inspect.get_source as well as improved error messages from 3.11 that are currently missing.
[x] An output folder will be created within the /home snekbox current working directory. Any files written within there will be uploaded.

Possible features for future PRs that will be supported with the snekbox changes.

[ ] Support for file attachments to be sent to snekbox
[ ] Support for attaching source files using https://paste.pythondiscord.com/ links
[ ] Support for naming multiple code blocks as different files (can be nice to demonstrate how imports work)

Examples

Per-file size limit of 8MiB (up to 50MiB possible since our bot is in a lv2+ boosted server)

Per-message file limit (currently 10 files)

Refactored #2422 to be compatible and fixed merge conflicts with #2408


text:
minor correction to config for chars filtering rule

200 -> 4200

I didn't see the other PR. Sorry about that.


text:
Compare two Images

Script Title - compare_img
Brief - A python image processing script for pixel-wise comparison between two images.

Program will take two images as input and perform image-processing to output quantitative measures about the differences in the given images.

Instructions

Create a new folder for your script and file/folder name should be appropriate.
Create a README.md in your folder for program Instructions
add requirements.txt if needed

Programming Language

[x] Python

Happy Coding

Thanks @pawangeek for this, will be making my first PR in some time!

repo: python-geeks/Automation-scripts


text:
create a contributing.md

Reference to issue #63
This is just a copy of @dujm file

This is just a copy of @dujm file

Now we can add more best practices in the md. Btw, here is one reference we could use pandas-mentoring/doc/source/development/contributing.rst

We already have that file copied to this repo. :)
We'll see what we can do, but eventually we may want to merge the guide you started with the pandas contributing documentation, so anyone who wants to contribute to pandas can benefit from it. But this is a lot of work, will take time until we see exactly what we want to do. :)

I'm having an issue that I don't fully understand. My branch master now has the correct stucture that I want for my pull request, but my branch contributing-guide does not. When I try to merge my branch master with by branch contributing-guide, the merge does not happen but it does not throw any issues. Should I close this PR and try again creating a new branch from my master branch and then creating a new pull request?

@martinagvilas  I don't know what branch contributing-guide is.  But I think it's really nice that you want to understand the problem so that we can all learn from it. So how about we just leave this PR open? Maybe others can answer your question and then you could try with this branch again.
Meanwhile, exactly as you said, a faster way is to create a new branch and send a new PR (only pickup the commit id for adding contribution.md)
  # 1) Add a new branch from your origin/master branch
   git branch new-branch origin/master
   git checkout new-branch

   # 2) Make a commit of your work (e.g. contribution.md)
   git add contribution.md
   git commit -m 'add contribution.md'

   # 3) If you had multiple commits, but you want to select a specific commit(e.g. commit id: bbb909f) for a pull request
   git cherry-pick bbb909f

   # 4) Push to your new-branch
   git push origin new-branch

@martinagvilas I'd say the problem is that you're merging from your outdated master. Let me explain how I see the workflow to update a branch.
Git is distributed, and from git prespective all branches are created equal, and any branch of any fork is as good as the rest. For convenience and by convention, we all consider the master branch of the https://github.com/python-sprints/pandas-mentoring the "good" one. When a commit reaches that branch, becomes part of our official code, and all the rest of branches and forks are somehow temporary copies to work on.
So, what we want is to get what's in that specific branch, into the branch where you are working. The first step is to download it locally:

git fetch upstream

This will create an upstream/master branch, that is read-only, and an exact copy of the "good" branch, but locally.
Then we want to be in the branch to update, to make it active, if it's not we do:

git checkout contributing-guide

And finally, we tell git to copy all the commits (the ones that are not already there), from the branch upstream/master to contributing-guide with:

git merge upstream/master

Note that in the whole process I never used or changed your local master (which is different from upstream/master). I have the feeling that you're doing the process correctly, except that instead of merging from upstream/master, you're merging from master. Quite reasonable, and those git things are far from intuitive at the beginning, but since your master is probably behind contributing-guide, weird things are happening.
Does this make sense?

Hi Marc, makes perfect sense! Thank you for the explanation.
However, I tried what you said, my master is now updated with upstream/master, but when I try to merge upstream/master inside my contributing-guide branch, git says "Already up to date". But when I inspect the contents of this branch it only shows three files (the readme, the contributing and the learning points markdowns, thus missing the folders and the .gitignore), which I find super weird and do not want to push the changes as I will break this PR even more. Do you have an idea of why this might be and how to solve it?

I think I got it now. Apparently, I was stuck in a rebase. Could you check if now is alright? Sorry for the troubles and thanks for the explanation!

There are still files being deleted, but looks better. If merging says you're already up to date, it means that you have a commit in your branch deleting those files. The merge only adds new commits from the "good" branch, but if you have a commit from for example fixing a conflict that didn't go well and deleted those file, this won't be fixed by the merge.
One solution could be check in the commits of this branch which is the one deleting the file and revert it (git revert <commit-id> if I'm not wrong).
Another one that I use somehow often is in your branch do a git reset --soft upstream/master. This will undo all the commits, and leave all the changes in the staging area. Then with git reset you can unstage the changes you don't want to commit, and just leave the ones that you want. After the commit when you push, first try it normally, but it will fail and then you'll have to push with the -f flag. This is because you rewrote the history and git won't let you overwrite the history in a push unless you're explicit you want it.

@martinagvilas I started sifting throw all of the emails to capture some of the feedback @datapythonista and other shared when folks got stuck. I also took a look at the contribution file when I was uploading it initially and I agree with @dujm. We should leverage as much of that work as possible.
However, my main question is duplication do we want to add a bit more rigor to the existing contributions.rst file (by just adding a best practice section under the git section) or so we want to keep them in both places? Maybe we provide a link from the contributors section to our wiki page. Please let me know what you guys think?

You call, but just two things to consider.
I see the GitHub wiki just a fancy interface to a secondary github repo of markdowns (GitHub literally uses a git repo to store the files). Since everybody using this repo is able to work directly with the markdown files, and it probably adds value that the changes to the pages are done through a PR process, and not individual edits, I'd personally disable the wiki and keep things centralized in the main project repo.
Unrelated, the first actual task for pandas will be to write a better contributing documentation for the project. This will have a very big overlap with the contributing documentation you're writing. I'd move forward for now without worrying much about that, but probably worth to keep in mind when making any decision.

@datapythonista I agree that it makes it easier for everyone to contribute ! I’ll keep mining those nuggets out of the PR messages. Thanks!

Third attempt, but I think I got it right now. I couldn't find the commit that deleted the files, but I force-pushed everything into a clean state now. Let me know if its alright.

authors: datapythonista dujm martinagvilas sparalic

repo: python-sprints/pandas-mentoring


text:
🛑 Billing Api is down

In 5b723aa, Billing Api (https://billing.roblox.com/) was down:

HTTP code: 404
Response time: 58 ms

Resolved: Billing Api is back up in 2e6a243 after 23 minutes.

repo: pythoniaweb/statusboom


text:
memory alignment issue in torch.compile mode

There appears to be an alignment issue when attempting to run with torch.compile.
This is reproduced with nightly build (2.4.0.dev20240511+cu121) and the simple script below.
When printing the shape and stride of the internal vectors, we get:
torch.Size([1040, 1040]) and (1, 1152) indicating that the memory allocator has padded to 128-alignment.
import torch
def train():
    from float8_experimental import config
    config.enable_amax_init = False  # only needed for autocast + compile + FSDP +  float8 delayed
    config.enable_pre_and_post_forward = False  # only needed for autocast + compile + FSDP +  float8 delayed

    from float8_experimental.float8_linear import Float8Linear
    from float8_experimental.float8_linear_utils import (
            swap_linear_with_float8_linear
        )
    model = torch.nn.Sequential(torch.nn.Flatten(), torch.nn.Linear(256 * 256 * 3, 1040),
                torch.nn.Linear(1040,1040)
                )
    swap_linear_with_float8_linear(model,Float8Linear)
    device = torch.device('cuda')
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    criterion = torch.nn.CrossEntropyLoss()
    model = model.to(device)
    model = torch.compile(model, fullgraph=True)
    model.train()
    inputs = torch.randn([16, 3, 256, 256], dtype=torch.float32).cuda()
    label = torch.randint(high=1040, size=[16], dtype=torch.int64).cuda()
    with torch.autocast(device_type='cuda', dtype=torch.bfloat16, enabled=True):
        outputs = model(inputs)
        loss = criterion(outputs, label)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

if __name__ == '__main__':
    train()

https://github.com/pytorch/ao/issues/569

authors: czmrand vkuzo


text:
❓ [Question] Bert lost a lot of accuracy when using fp16

❓ Question
BERT Text Classification model run in fp16 gets huge different result compared to fp32
What you have already tried

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

PyTorch Version (e.g., 1.0): 1.13
CPU Architecture:
OS (e.g., Linux): REHL8
How you installed PyTorch (conda, pip, libtorch, source): pip
Build command you used (if compiling from source):
Are you using local sources or building from archives:
Python version: 3.8.10
CUDA version:11.7
GPU models and configuration: Tesla T4
Any other relevant information:

Torch-TensorRT Version: 1.3
Additional context
Model converted from TouchScript to TensorRT
enabled_precisions= {torch.half} # run with 16-bit precision
trt_model = torch_tensorrt.compile(model, inputs=inputs, enabled_precisions=enabled_precisions,
                                          truncate_long_and_double=True, require_full_compilation=False
                                          )

The logs
WARNING: [Torch-TensorRT] - For input input_ids.1, found user specified input dtype as Long, however when inspecting the graph, the input type expected was inferred to be Float
The compiler is going to use the user setting Long
This conflict may cause an error at runtime due to partial compilation being enabled and therefore
compatibility with PyTorch's data type convention is required.
If you do indeed see errors at runtime either:
- Remove the dtype spec for input_ids.1
- Disable partial compilation by setting require_full_compilation to True
WARNING: [Torch-TensorRT] - For input token_type_ids.1, found user specified input dtype as Long, however when inspecting the graph, the input type expected was inferred to be Float
The compiler is going to use the user setting Long
This conflict may cause an error at runtime due to partial compilation being enabled and therefore
compatibility with PyTorch's data type convention is required.
If you do indeed see errors at runtime either:
- Remove the dtype spec for token_type_ids.1
- Disable partial compilation by setting require_full_compilation to True
WARNING: [Torch-TensorRT] - For input attention_mask.1, found user specified input dtype as Long, however when inspecting the graph, the input type expected was inferred to be Double
The compiler is going to use the user setting Long
This conflict may cause an error at runtime due to partial compilation being enabled and therefore
compatibility with PyTorch's data type convention is required.
If you do indeed see errors at runtime either:
- Remove the dtype spec for attention_mask.1
- Disable partial compilation by setting require_full_compilation to True
WARNING: [Torch-TensorRT] - Data types for input tensors have been modified by inserting aten::to operations which cast INT64 inputs to INT32. To disable this, please recompile using INT32 inputs
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT] - Truncating intermediate graph input type from at::kLong to at::kInt
WARNING: [Torch-TensorRT TorchScript Conversion Context] - CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Int64 to Int32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - There may be undefined behavior using dynamic shape and aten::size without setting allow_shape_tensors
WARNING: [Torch-TensorRT] - Sum converter disregards dtype
WARNING: [Torch-TensorRT] - Sum converter disregards dtype
WARNING: [Torch-TensorRT] - Truncating weight (constant in the graph) from Float64 to Float32
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Using PreviewFeature::kFASTER_DYNAMIC_SHAPES_0805 can help improve performance and resolve potential functional issues.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_0))
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_2))
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_1))
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_0))
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_2))
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_1))
        WARNING: [Torch-TensorRT TorchScript Conversion Context] - Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_0))
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_2))
WARNING: [Torch-TensorRT TorchScript Conversion Context] -  (# 0 (SHAPE input_1))
WARNING: [Torch-TensorRT TorchScript Conversion Context] - TensorRT encountered issues when converting weights between types and that could affect accuracy.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Check verbose logs for the list of affected weights.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - - 131 weights are affected by this issue: Detected subnormal FP16 values.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - - 78 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - - 1 weights are affected by this issue: Detected finite FP32 values which would overflow in FP16 and converted them to the closest finite FP16 value.
WARNING: [Torch-TensorRT] - CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars

Hello - could you try compiling with Torch-TensorRT 1.4 or main, to see if the values have changed, since the TensorRT version and our converters will be updated in versions newer than 1.3. Additionally, try calling model.half() prior to compilation to see if the accuracy results change.
@narendasan - could the accuracy issue be related to the subnormal weights/overflow?

Hello - could you try compiling with Torch-TensorRT 1.4 or main, to see if the values have changed, since the TensorRT version and our converters will be updated in versions newer than 1.3. Additionally, try calling model.half() prior to compilation to see if the accuracy results change.
@narendasan - could the accuracy issue be related to the subnormal weights/overflow?

Hi gs-olive, I have tried to compile with Torch-TensorRT1.4 and get the different result compared to running in FP32.
When trying to call model.half() prior to compilation, raise the RuntimeError: expected scalar type Half but found Float.
The code snippet
          model = torch.jit.load(path)
          model.half()
          model.to(device)

          # The model needs to be in evaluation mode
          model.eval()
          # enabled_precisions= {torch.half} # run with 16-bit precision
          # enabled_precisions = {torch.float32}  # run with 32-bit precision
          enabled_precisions= {torch.half}

          trt_model_fp32 = torch_tensorrt.compile(model, inputs=inputs, enabled_precisions=enabled_precisions,
                                                  truncate_long_and_double=True, require_full_compilation=False, debug=True
                                                  )

got same issue

repo: pytorch/TensorRT


text:
KeyError when wrapping a dict with a SequenceWrapper

🐛 Describe the bug
I get the following errors.
from torchdata.datapipes.map import SequenceWrapper

tuple(SequenceWrapper({'a': 100, 'b': 200, 'c': 300, 'd': 400}))  # produces KeyError: 0
tuple(SequenceWrapper({0: "100", 1: "200"}))  # produces KeyError: 2

I expect them to return (100, 200, 300, 400) and ("100", "200"), respectively, instead.
Versions
torchdata 0.4.1

Can I know what is the use case?
You can do tuple(SequenceWrapper(...). to_iter_datapipe(indices=[...]))

A similar thing happens when you do:
dp = SequenceWrapper({'a': 100, 'b': 200, 'c': 300, 'd': 400})
print(list(dp))

> File "\lib\site-packages\torch\utils\data\datapipes\map\utils.py", line 46, in __getitem__
>    return self.sequence[index]
> KeyError: 0

source_dp = IterableWrapper([(i+1, i) for i in range(10)])
map_dp = source_dp.to_map_datapipe()
print(list(map_dp))

> []

Funnily enough

source_dp = IterableWrapper([(i, i) for i in range(10)])
map_dp = source_dp.to_map_datapipe()
print(list(map_dp))
> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

As present in the doc works fine. This prints the values instead of the keys.
This also works as expected:
source_dp = SequenceWrapper(range(5,11))
print(tuple(source_dp))
> [5, 6, 7, 8, 9, 10]

The problem is that while converting to a list/tuple, python does the following (at least according to my debugger): First __len__() is called and then __getitem__() with indices from 0 until an error occurs (not until len-1 ... why?).
Why the first throws a Key Error while the IndexError thrown by __getitem__() in IterToMapConverter or SequenceWrapper  is "ignored" is beyond me.
As mentioned by @ejguan this can be solved by first converting to a IterDataPipe.
Alternatively we could implement __iter__() in IterToMapConverter or SequenceWrapper but I am not sure if we want to do this as theses are MapDatapipes.

Personally I would expect the data.

Personally I would expect the data.

Yeah, I would think so for the most of users. @SvenDS9  How about this. Let's open a RFC issue to discuss to introduce this new behavior for all MapDataPipe. When iter is called, we always return the result for data if possible.
It might gives the possibility for the whole pipeline iterable as I would expect the starting point of the Map pipeline would be SequenceWrapper. If SequenceWrapper can provide result via __iter__, the subsequent MapDataPipes should be able to rely on for d in self.datapipe pattern to do iteration.
And, I personally thinks this behavior should always be warned because users should do mapdatapipe.to_iterdatapipe() to achieve iteration. MapDataPipe is kind not designed for iteration, it's for random accessing.

The main concern in my mind is Cython expects integer indices as input for iterator however we are changing this behavior like accepting any kind of indices.
https://github.com/python/cpython/blob/main/Objects/iterobject.c#L46-L77

authors: SvenDS9 ejguan philip-bl

repo: pytorch/data


text:
[Doc] Fix examples and migrate to sphinx doctest (iter/util)

Similar to #901 and as discussed in #850 this migrates examples in iter/util directory to sphinx doctest format.
As sphinx is also able to check normal doctest-style examples (which will be enabled once doctest_test_doctest_blocks = "" is deleted from conf.py), it is not necessary to convert all examples into the testcode/testoutput block format.
As some tests need to be skipped conditionally.
Changes

Fixes some faulty examples
Migrates many examples in iter/util to testcode/testoutput block format
conditionally skip testing some examples
use functional form in examples

Using the testcode/testoutput block format the output does look slightly different but imo this it's actually an improvement

I also noticed that many datapipes in util also handle loading so should probably be moved to load.
Imho the division into load, transform and util is fine but is in contrast to the documentation, where we have 10(!) different types.


text:
incremental shuffle

This PR adds a filter that performs inline shuffling. Unlike the default shuffle, the incremental shuffle permits fast startup by having both a minimal buffer size and a maximum buffer size. As soon as more than the minimum buffer size samples have been read, samples are passed on to the next stage.
This filter is commonly used with WebDataset to shuffle training samples inline.

@NicolasHug @abhinavarora

authors: msaroufim tmbdev


text:
training speed about fairseq

❓ Questions and Help
On the machine A, wps=120000, on another machine B, wps=60000, what's wrong about this?
What's your environment?

fairseq Version (e.g., 1.0 or master): 0.10.1（A and B）
PyTorch Version (e.g., 1.0)：1.7.1 （A）1.7.0（B）
OS (e.g., Linux): Linux Ubuntu16.04
How you installed fairseq (pip, source): pip
Build command you used (if compiling from source):
Python version: 3.7.5
CUDA/cuDNN version: 11.0(A) 10.2(B)
GPU models and configuration: transformer, 6gpus
Any other relevant information:

It seems like the only difference is PyTorch 1.7.0 and PyTorch 1.7.1? Perhaps there's some improvement in 1.7.1 that makes training faster?
Without more details about your setup/commands run, etc., it's impossible to say for sure :)

It seems like the only difference is PyTorch 1.7.0 and PyTorch 1.7.1? Perhaps there's some improvement in 1.7.1 that makes training faster?
Without more details about your setup/commands run, etc., it's impossible to say for sure :)

It seems like the only difference is PyTorch 1.7.0 and PyTorch 1.7.1? Perhaps there's some improvement in 1.7.1 that makes training faster?
Without more details about your setup/commands run, etc., it's impossible to say for sure :)
command:
python -m torch.distributed.launch --nproc_per_node=6 --nnodes=1 --node_rank=0 --master_addr=127.0.0.1 --master_port=16123 --use_env train_v0.10.py --task speech_recognition --arch vggtransf_conf_1 --criterion cross_entropy_with_acc --optimizer adam --lr-scheduler noam --log-interval 100 --log-format simple --seed 23424 --user-dir asr_transf_conf --vocab-threshold 3 --lmdb-dir chs_dataset --mean-npy mean.npy --std-npy std.npy --feat-dim 80 --max-frames 16700 --min-frame-num 200 --max-frame-num 1600 --required-sample-size-multiple 8 --chunk-size 1000 --specaug-T 100 --specaug-F 20 --char-vocab train_units.dict --num-workers 0 --train-subset train --valid-subset dev --batch-size 83 --required-batch-size-multiple 8 --distributed-world-size 6 --max-epoch 20 --max-update 350000 --clip-norm 1.0 --update-freq 1 --lr 0.0002 --min-lr 0.000001 --tensorboard-logdir tensorboard --save-interval-updates 5000 --adam-betas "(0.9,0.98)" --weight-decay 0.00005 --warmup-updates 20000 --label-smoothing 0.1 --lr-alpha 0.03 --lr-shrink 0.3 --milestones 320000,340000 --fp16-init-scale 128 --fp16-scale-window 2000 --threshold-loss-scale 1.0 --no-progress-bar --specaug --fp16 --fix-batches-to-gpus --find-unused-parameters --save-dir checkpoints --distributed-no-spawn --ddp-backend c10d --distributed-backend nccl

log:
machine A       2021-01-04 13:53:41 | INFO | train_inner | epoch 001:    300 / 12377139 loss=0.88, ce_loss=1.139, nll_loss=0.96, acc=0.83, wps=110333, ups=1.2, wpb=92139.7, bsz=283.6, num_updates=100300, lr=9.47263e-05, gnorm=0.911, clip=8, loss_scale=2048, train_wall=70, wall=0
machine B.      2021-01-04 21:56:30 | INFO | train_inner | epoch 001:    300 / 12377139 loss=0.881, ce_loss=1.134, nll_loss=0.952, acc=0.832, wps=61846.5, ups=0.68, wpb=90680.1, bsz=266.3, num_updates=100300, lr=9.47263e-05, gnorm=0.933, clip=8, loss_scale=2048, train_wall=144, wall=0

It seems like the only difference is PyTorch 1.7.0 and PyTorch 1.7.1? Perhaps there's some improvement in 1.7.1 that makes training faster?
Without more details about your setup/commands run, etc., it's impossible to say for sure :)
command:
python -m torch.distributed.launch --nproc_per_node=6 --nnodes=1 --node_rank=0 --master_addr=127.0.0.1 --master_port=16123 --use_env train_v0.10.py --task speech_recognition --arch vggtransf_conf_1 --criterion cross_entropy_with_acc --optimizer adam --lr-scheduler noam --log-interval 100 --log-format simple --seed 23424 --user-dir asr_transf_conf --vocab-threshold 3 --lmdb-dir chs_dataset --mean-npy mean.npy --std-npy std.npy --feat-dim 80 --max-frames 16700 --min-frame-num 200 --max-frame-num 1600 --required-sample-size-multiple 8 --chunk-size 1000 --specaug-T 100 --specaug-F 20 --char-vocab train_units.dict --num-workers 0 --train-subset train --valid-subset dev --batch-size 83 --required-batch-size-multiple 8 --distributed-world-size 6 --max-epoch 20 --max-update 350000 --clip-norm 1.0 --update-freq 1 --lr 0.0002 --min-lr 0.000001 --tensorboard-logdir tensorboard --save-interval-updates 5000 --adam-betas "(0.9,0.98)" --weight-decay 0.00005 --warmup-updates 20000 --label-smoothing 0.1 --lr-alpha 0.03 --lr-shrink 0.3 --milestones 320000,340000 --fp16-init-scale 128 --fp16-scale-window 2000 --threshold-loss-scale 1.0 --no-progress-bar --specaug --fp16 --fix-batches-to-gpus --find-unused-parameters --save-dir checkpoints --distributed-no-spawn --ddp-backend c10d --distributed-backend nccl

log:
machine A       2021-01-04 13:53:41 | INFO | train_inner | epoch 001:    300 / 12377139 loss=0.88, ce_loss=1.139, nll_loss=0.96, acc=0.83, wps=110333, ups=1.2, wpb=92139.7, bsz=283.6, num_updates=100300, lr=9.47263e-05, gnorm=0.911, clip=8, loss_scale=2048, train_wall=70, wall=0
machine B.      2021-01-04 21:56:30 | INFO | train_inner | epoch 001:    300 / 12377139 loss=0.881, ce_loss=1.134, nll_loss=0.952, acc=0.832, wps=61846.5, ups=0.68, wpb=90680.1, bsz=266.3, num_updates=100300, lr=9.47263e-05, gnorm=0.933, clip=8, loss_scale=2048, train_wall=144, wall=0

@myleott

@myleott

@myleott

@myleott

repo: pytorch/fairseq

license: mit


text:
pytorch feature_column

Have you planned a method similar to the feature column in tf？

Hi @Acmedao, I'm not sure what you mean. If you could share a use case you have in mind that isn't served by TorchRec / PyTorch, we can help point you to the right direction.

authors: Acmedao sarckk

repo: pytorch/torchrec


text:
Will future support include expert parallel and sequence parallel (such as ring attention)?

Great job! Will future support include expert parallel and sequence parallel (such as ring attention)?

Yes we are actively working on them, so stay tuned :)

authors: Doraemonzzz wanchaol

repo: pytorch/torchtitan


text:
Deep Fusion Modules

Context
What is the purpose of this PR? Is it to

[x] add a new feature
[ ] fix a bug
[ ] update tests and/or documentation
[ ] other (please add here)

This implements modules from the Fusion Models RFC #1283 necessary for building Deep Fusion Models.
Changelog
This introduces the model_fusion folder in modules and the following moduels

fusion_models.DeepFusionModel
fusion_layer.FusionLayer
fusion_embed.FusionEmbedding
fusion_utils.regsiter_fusion_module

Along with these modules there is an init.py, a new section in the modules api docs, and a unit test for each module/function added. No existing code was edited.
Test plan
Please make sure to do each of the following if applicable to your PR. (If you're not sure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.)

[x] run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
[x] add unit tests for any new functionality
[x] update docstrings for any new or updated methods or classes
[x] run unit tests via pytest tests
[x] run recipe tests via pytest tests -m integration_test
[ ] manually run any new or modified recipes with sufficient proof of correctness
[ ] include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

UX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Example of docstring: https://github.com/pytorch/torchtune/blob/6a7951f1cdd0b56a9746ef5935106989415f50e3/torchtune/modules/vision_transformer.py#L285
Example in our docs: https://pytorch.org/torchtune/main/tutorials/qat_finetune.html#applying-qat-to-llama3-models

[ ] I did not change any public API;
[ ] I have added an example to docs or docstrings;

@joecummings my guess is that each of these files are going to include many more classes, especially as we add support for early fusion models (extrapolating from the RFC). If this is true, it might make sense to keep these lightweight

Codecov Report
Attention: Patch coverage is 39.38462% with 197 lines in your changes missing coverage. Please review.

Project coverage is 27.56%. Comparing base (d0b89e2) to head (ae2820e).
Report is 1 commits behind head on main.

Files
Patch %
Lines

...rchtune/modules/model_fusion/test_fusion_models.py
37.77%
56 Missing :warning:

...orchtune/modules/model_fusion/test_fusion_layer.py
46.03%
34 Missing :warning:

torchtune/modules/model_fusion/_fusion_embed.py
23.80%
32 Missing :warning:

torchtune/modules/model_fusion/_fusion_layer.py
29.54%
31 Missing :warning:

...orchtune/modules/model_fusion/test_fusion_embed.py
45.45%
24 Missing :warning:

torchtune/modules/model_fusion/_fusion_models.py
43.47%
13 Missing :warning:

...orchtune/modules/model_fusion/test_fusion_utils.py
42.85%
4 Missing :warning:

torchtune/modules/model_fusion/_fusion_utils.py
57.14%
3 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1338      +/-   ##
==========================================
+ Coverage   27.24%   27.56%   +0.32%
==========================================
  Files         262      270       +8
  Lines       12129    12473     +344
==========================================
+ Hits         3304     3438     +134
- Misses       8825     9035     +210

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter kartikayk pbontrager

repo: pytorch/torchtune


text:
Add packing to wikitext dataset

Context
What is the purpose of this PR? Is it to

[ ] add a new feature
[X] fix a bug
[ ] update tests and/or documentation
[ ] other (please add here)
#1788
Please link to any issues this PR addresses.

Changelog
What are the changes made in this PR?

Add packing to wikitext dataset.

Test plan
Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

[X] run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
[ ] add unit tests for any new functionality
[X] update docstrings for any new or updated methods or classes
[X] run unit tests via pytest tests
[ ] run recipe tests via pytest tests -m integration_test
[ ] manually run any new or modified recipes with sufficient proof of correctness
[ ] include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

UX
If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example

[X] I did not change any public API
[ ] I have added an example to docs or docstrings

@RdoubleA @joecummings here everything is okay, might be merged

Looks like you might need to merge/rebase on main?

Yeah, let me fix merge conflict

authors: joecummings krammnic


text:
Support more flexible ways to instantiate models in torchvision.models, e.g., remove fc layers, support for pretrained=True and num_classes！=1000

🚀 Feature
Support more flexible ways to instantiate models in torchvision.models,
e.g., instantiate ResNet-50 only for feature extraction without FC layer like Keras style:
torchvision.models.resnet50(include_top=False)

load pretrained weights when class number doesn't equals to 1000(imagenet):
torchvision.models.resnet50(pretrained=True, num_classes=10)

Motivation
In many situations, we need more flexiable ways to instantiate models in torchvision.models.
For examples, when finetuning a ResNet-50 classification model on a dataset of 10 classes, we need  torchvision.models.resnet50(pretrained=True, num_classes=10), but it is not supported now. In current implementation, num_classes should be 1000 when pretrained=True. To implement this case, we should support partial copy from pretrained weights(load weights except for FC layer).
Another example: sometimes we need instantiate the models in torchvision.models as backbone, which means the FC layer is no more needed. Implementation in this case is also needed. We can support this case with an additional argument, like include_top=False in Keras.
Pitch
A possible solution is to modify some codes in models constructions. At least two more features should be realized:

Support for loading partial weights: when num_classes!=1000, weights can still be loaded except for last FC layer.
Support for backbone mode(FC layer moved): when an argument (like include_top) is set to False, the last layer(FC layer) will be moved.

We can apply these modifications to many basic models in torchvision.models.

Hi,
Thanks for opening the issue to start the discussion!
I have a few worries about the current approach:

include_top: what is a top? Is it the last FC layer? should it also include the last layer4 block or not? what about models like VGG, which have 3 MLP for the final classifier, should top be the last MLP or the last 3? There is no single answer for that, and instead of providing a solution that only works in a single case, it's IMO preferable to let the users decide themselves what they want to remove / include
loading partial weights: the same question applies, what if users want to change more than the number of classes, but also the internal number of features for the last classifier?

For those reasons, I inclined to say that we shouldn't be providing those high-level model manipulation blocks (which are limited in scope), but instead encourage users to modify the models as their framework / application need.
Thoughts?

Yes, I do understand your worries. But in my opinions:

In many CV tasks, we remove all fc layers (perhaps also pooling layer after last conv layer) from these classification models (resnet-50, vgg-16, etc). The remained full convolutional backbones serve as feature exactors. I think we can add two arguments like the following to control that:

something like include_fc={True, False} to remove all fc layers, softmax classifiers,...

something like 'pooling={None, avg, adaptive}' to add pooling layer to the end or not, and the pooling type

I think, in most cases, loading partial weights for the above situations can satisfies the most cases. At least, support for setting pretrained=True and num_classes!=1000 for the same time is a big puls for flexibility.

There are already some discussions for this:
#184, #190,
discussion about #173

Another issue that is related: #2152
I think that adding those extra arguments like include_fc={True, False} or pooling={None, avg, adaptive} makes it fairly complicated, but not generally more generic.
One could arguably say that they would want to control the number of fc layers (maybe adding more than the initial one?). I think that given the generality of the task (model surgery), I would prefer to keep the implementation simple as it facilitates the users modifying the code themselves.
This is because PyTorch model definition couples the layers (in the __init__) with the execution (in forward), so anything added / removed in the __init__ should also be handled in the forward, increasing complexity.

@fmassa OK. I will fork and do modifications in my own repository.

@ljqcava the way I would recommend doing it is to instead of forking torchvision (and having to install your forked version) to either:
Subclass the model that you want to extend (preferred)
Here is an example:
class MyResNet(torchvision.models.resnet.ResNet):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        del self.fc

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        return x

Copy-paste the whole resnet.py file
This is a solution for maximum flexibility. The benefits from copying the file instead of forking torchvision is that you can still use torchvision in your code (and new updates that comes with it), and just import your custom resnet implementation.
For example
import torchvision
import my_resnet

As such, I'm closing the issue but let me know if you have further questions.

authors: fmassa ljqcava

repo: pytorch/vision


text:
run as administrator

hi
how can open app that need to administrator permission?
(run as administrator)

I guess, your python process executing the Pywinauto should be running as admin. Also you might want to have a look at pyuac: https://gist.github.com/Preston-Landers/267391562bc96959eb41

Also it's worth noting that UAC confirmation window can't be automated by OS design. You have to disable UAC manually in OS settings. Then ShellExecuteEx approach would work.

https://github.com/yinkaisheng/Python-UIAutomation-for-Windows/blob/master/uiautomation/uiautomation.py#L2324

authors: HSalehpour20 airelil lanzorg vasily-v-ryabov

repo: pywinauto/pywinauto


text:
New propagation library created

What: propagation library created
How: moved code related only with propagation from tf.utils to propagation.py in libraries
Why: cleaner code

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: superplay1:x: frosati1You have signed the CLA already but the status is still pending? Let us recheck it.

Related: https://github.com/shaimach/c3po/issues/49

authors: CLAassistant frosati1 lazyoracle


text:
🛑 nitter.in.projectsegfau.lt is down

In 8a23a4e, nitter.in.projectsegfau.lt (https://nitter.in.projectsegfau.lt) was down:

HTTP code: 503
Response time: 1026 ms

Resolved: nitter.in.projectsegfau.lt is back up in cd9cdcd after 6 minutes.


text:
🛑 nitter.projectsegfau.lt is down

In 7af42bc, nitter.projectsegfau.lt (https://nitter.projectsegfau.lt) was down:

HTTP code: 503
Response time: 202 ms

Resolved: nitter.projectsegfau.lt is back up in c96de70 after 17 minutes.


text:
🛑 nitter.cutelab.space is down

In cc9af05, nitter.cutelab.space (https://nitter.cutelab.space) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.cutelab.space is back up in 9608ab9 after 18 minutes.


text:
🛑 t.floss.media is down

In eb52b46, t.floss.media (https://t.floss.media) was down:

HTTP code: 0
Response time: 0 ms

Resolved: t.floss.media is back up in 70a8c9d after 16 minutes.


text:
🛑 nitter.tiekoetter.com is down

In aab6f39, nitter.tiekoetter.com (https://nitter.tiekoetter.com) was down:

HTTP code: 502
Response time: 522 ms

Resolved: nitter.tiekoetter.com is back up in bc8a750 after 10 minutes.


text:
🛑 nitter.ir is down

In aecc678, nitter.ir (https://nitter.ir) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.ir is back up in 6a38770 after 51 minutes.


text:
🛑 nitter.privacytools.io is down

In 07f55bd, nitter.privacytools.io (https://nitter.privacytools.io) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.privacytools.io is back up in 99ae4b2 after 50 days, 10 hours, 17 minutes.


text:
🛑 nitter.projectsegfau.lt is down

In c2bd256, nitter.projectsegfau.lt (https://nitter.projectsegfau.lt) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.projectsegfau.lt is back up in 712b9d8 after 4 minutes.


text:
🛑 tweet.whateveritworks.org is down

In 92dd39e, tweet.whateveritworks.org (https://tweet.whateveritworks.org) was down:

HTTP code: 502
Response time: 656 ms

Resolved: tweet.whateveritworks.org is back up in a4f0ba4 after 24 minutes.


text:
🛑 nitter.privacytools.io is down

In daccd6b, nitter.privacytools.io (https://nitter.privacytools.io) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.privacytools.io is back up in 4a6c48d after 14 minutes.


text:
🛑 nitter.net is down

In ce9307d, nitter.net (https://nitter.net) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nitter.net is back up in 4b44cfa.


text:
🛑 nitter.us.projectsegfau.lt is down

In 44a5d24, nitter.us.projectsegfau.lt (https://nitter.us.projectsegfau.lt) was down:

HTTP code: 503
Response time: 388 ms

Resolved: nitter.us.projectsegfau.lt is back up in 854ac8a after 11 minutes.


text:
🛑 nitter.in.projectsegfau.lt is down

In 2e3d055, nitter.in.projectsegfau.lt (https://nitter.in.projectsegfau.lt) was down:

HTTP code: 503
Response time: 788 ms

Resolved: nitter.in.projectsegfau.lt is back up in 7e09283 after 7 hours, 20 minutes.


text:
Product advertising

Let's update setup.sh, start.sh and stop.sh. Big name of "zebrunner standalone" should appears when user will run this scripts.

https://github.com/qaprosoft/qps-infra/pull/198

Verified

authors: dhreben ilya-varchenya

repo: qaprosoft/qps-infra


text:
Confirm cancel or abort actions

During processes in a dialog window, users can close the dialog by hitting the ESC key and sth. even when clicking outside of the dialog.
An enhancement can be to provide a confirm modal, so users can prevent accidental dialog window closings.

[DM-1320]

repo: qbicsoftware/data-manager-app


text:
Q23 Initial Harness Design

Initially, I was going to try RapidHarness but the projects are limited to 30 connections. Altium is adding a harness design feature as well so I thought I would start with this project and transition over if we can get full access.
NOTE: This will only be merged once we are happy with the design and software CAD tools used. The initial commits will be to just capture what we have so manufacturing can start in a timely manner.

TODO

Add a gear position sensor to this schematic (will go to an analog pin on the ECU)
Verify Shutoff circuit against the rules
Update any BSPD connections based off of "inline breakout box" notes the rules.

Another Note: Map steering wheel to PDM power.

NOTE: also remove 5V supply from the engine speed sensor connection

March 16

Added kill switch signal and return to the dash bulk head. Need this replicated on the front loom. I have placed a female two pin on the kill switch itself.
Placed the driver kill switch in the elec panel harness box.
Joe Will need to help with front loom replication

repo: qfsae/pcb


text:
Change name and iqm5q_q0

Following #https://github.com/qiboteam/qibolab/pull/522 change.

Thanks @Jacfomg. Shouldn't we change all runcards?

Nope, only this runcard uses anharmonicity